2024-04-22T13:13:13.609 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T13:13:13.609 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} --name leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi --verbose --archive /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274 --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/orig.config.yaml 2024-04-22T13:13:13.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T13:13:13.775 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} email: null first_in_suite: false job_id: '7668274' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4362 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 222/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKEKiagJ6SyJK7WnVaOyQPdVnJadhtOLKx3cCeDGe+TX5BKCXObOkZrJXEGm6q2OXllWoXhWr8+6qJU1y8+twN0= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMMa0rhmKvXOnJj/3ALkp/eGJPqVOqOsRVprohL6YlpB+B6jnLK/TpExD8oWmaWdjElPJEFhFMQ7fYnXobp/RbI= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ982rpFDuGRk1BrLBKwzrdWtzmdsLFTS8UHhCYykoLd9auwi9FvcaXmpYRUMDcraQacuv0Am4H9qSWjRFgOehk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_12:36:42 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T13:13:13.775 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa; will attempt to use it 2024-04-22T13:13:13.776 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa/tasks 2024-04-22T13:13:13.776 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T13:13:13.777 INFO:teuthology.task.internal:Checking packages... 2024-04-22T13:13:13.861 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-22T13:13:13.861 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T13:13:13.861 INFO:teuthology.packaging:ref: None 2024-04-22T13:13:13.861 INFO:teuthology.packaging:tag: None 2024-04-22T13:13:13.862 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-22T13:13:13.862 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:13:13.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-22T13:13:14.270 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-22T13:13:14.272 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T13:13:14.279 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T13:13:14.279 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T13:13:14.292 INFO:teuthology.task.internal:Saving configuration 2024-04-22T13:13:14.318 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T13:13:14.329 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T13:13:14.352 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 13:06:20.071797', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKEKiagJ6SyJK7WnVaOyQPdVnJadhtOLKx3cCeDGe+TX5BKCXObOkZrJXEGm6q2OXllWoXhWr8+6qJU1y8+twN0='} 2024-04-22T13:13:14.369 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 13:06:20.066131', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMMa0rhmKvXOnJj/3ALkp/eGJPqVOqOsRVprohL6YlpB+B6jnLK/TpExD8oWmaWdjElPJEFhFMQ7fYnXobp/RbI='} 2024-04-22T13:13:14.391 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 13:06:20.068964', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ982rpFDuGRk1BrLBKwzrdWtzmdsLFTS8UHhCYykoLd9auwi9FvcaXmpYRUMDcraQacuv0Am4H9qSWjRFgOehk='} 2024-04-22T13:13:14.391 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T13:13:14.397 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-22T13:13:14.397 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-22T13:13:14.397 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-22T13:13:14.397 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T13:13:14.501 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8293a450d0>, signals=[15]) 2024-04-22T13:13:14.501 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T13:13:14.509 INFO:teuthology.task.internal:Opening connections... 2024-04-22T13:13:14.510 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:13:14.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:13:14.589 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-04-22T13:13:14.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:13:14.668 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-04-22T13:13:14.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:13:14.745 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T13:13:14.752 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-04-22T13:13:14.770 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-04-22T13:13:14.770 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-04-22T13:13:14.827 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T13:13:14.828 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T13:13:14.829 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T13:13:14.829 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-04-22T13:13:14.848 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-04-22T13:13:14.866 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-04-22T13:13:14.866 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-04-22T13:13:14.923 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T13:13:14.924 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T13:13:14.924 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-04-22T13:13:14.948 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-04-22T13:13:14.967 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-04-22T13:13:14.967 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-04-22T13:13:15.024 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-04-22T13:13:15.024 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-04-22T13:13:15.024 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-04-22T13:13:15.024 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T13:13:15.025 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T13:13:15.025 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-04-22T13:13:15.048 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T13:13:15.059 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T13:13:15.073 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T13:13:15.074 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-04-22T13:13:15.076 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-04-22T13:13:15.079 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-04-22T13:13:15.095 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T13:13:15.104 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T13:13:15.104 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-04-22T13:13:15.132 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-04-22T13:13:15.135 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-04-22T13:13:15.237 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T13:13:15.317 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T13:13:15.333 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-22T13:13:15.333 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-22T13:13:15.333 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-22T13:13:15.334 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T13:13:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:15.352 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-04-22T13:13:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-437.el9.x86_64 2024-04-22T13:13:15.410 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-437.el9.x86_64 2024-04-22T13:13:15.410 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-04-22T13:13:20.751 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-04-22T13:13:23.410 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-22T13:13:27.769 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-22T13:13:29.262 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-22T13:13:31.625 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-22T13:13:36.680 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T13:13:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T13:13:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T13:13:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-22T13:13:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-04-22T13:13:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-22T13:13:38.812 DEBUG:teuthology.orchestra.run.smithi087:> echo no | sudo yum reinstall kernel || true 2024-04-22T13:13:39.308 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 01:13:36 PM UTC. 2024-04-22T13:13:39.427 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:39.438 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-22T13:13:39.439 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:39.439 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:39.439 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 255 k 2024-04-22T13:13:39.439 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-04-22T13:13:39.439 INFO:teuthology.orchestra.run.smithi087.stderr:Operation aborted. 2024-04-22T13:13:39.551 INFO:teuthology.orchestra.run.smithi087.stdout:Is this ok [y/N]: 2024-04-22T13:13:39.551 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum reinstall -y kernel || true 2024-04-22T13:13:40.043 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 01:13:36 PM UTC. 2024-04-22T13:13:40.159 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-22T13:13:40.160 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 255 k 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-04-22T13:13:40.161 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-22T13:13:40.906 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-437.el9.x86_64.rpm 627 kB/s | 255 kB 00:00 2024-04-22T13:13:40.907 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-22T13:13:40.907 INFO:teuthology.orchestra.run.smithi087.stdout:Total 342 kB/s | 255 kB 00:00 2024-04-22T13:13:40.922 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-22T13:13:41.106 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-22T13:13:41.106 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-22T13:13:41.111 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-22T13:13:41.111 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-22T13:13:41.241 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-22T13:13:41.316 INFO:teuthology.orchestra.run.smithi087.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T13:13:41.316 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T13:13:41.940 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T13:13:41.940 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T13:13:42.657 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T13:13:42.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:42.658 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalled: 2024-04-22T13:13:42.658 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T13:13:42.658 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:13:42.658 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-22T13:13:42.856 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T13:13:42.948 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T13:13:42.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T13:13:42.948 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T13:13:42.949 DEBUG:teuthology.parallel:result is None 2024-04-22T13:13:42.949 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T13:13:43.086 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T13:13:43.087 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T13:13:43.090 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T13:13:43.094 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T13:13:43.113 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T13:13:43.127 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T13:13:43.150 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T13:13:43.150 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T13:13:43.152 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T13:13:43.155 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T13:13:43.241 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T13:13:43.256 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T13:13:43.257 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T13:13:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:43.274 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T13:13:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:43.292 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T13:13:43.294 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T13:13:43.318 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T13:13:43.322 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.344 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.357 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.365 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.371 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.383 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T13:13:43.385 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T13:13:43.401 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T13:13:43.401 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T13:13:43.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T13:13:43.414 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T13:13:43.453 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T13:13:43.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T13:13:43.466 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T13:13:43.469 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T13:13:43.481 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T13:13:43.510 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:43.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:43.616 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:43.696 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:43.737 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:13:43.737 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T13:13:43.801 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:43.845 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:43.902 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:43.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:44.029 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:13:44.030 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T13:13:44.095 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:44.141 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T13:13:44.199 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:44.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T13:13:44.325 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:13:44.325 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T13:13:44.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-04-22T13:13:44.393 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-04-22T13:13:44.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-04-22T13:13:44.423 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:13:44.434 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:13:44.477 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:13:44.773 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T13:13:44.784 INFO:teuthology.task.internal:Starting timer... 2024-04-22T13:13:44.784 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T13:13:44.803 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T13:13:44.811 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T13:13:44.811 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-04-22T13:13:44.861 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-04-22T13:13:44.863 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-04-22T13:13:44.945 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-04-22T13:13:44.947 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-04-22T13:13:45.002 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-04-22T13:13:45.004 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T13:13:45.004 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-04-22T13:13:45.024 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-04-22T13:13:45.024 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-04-22T13:13:45.047 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-04-22T13:13:45.047 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-04-22T13:13:45.086 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-04-22T13:13:45.086 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2024-04-22T13:13:45.086 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:13:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:45.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:13:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:45.140 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:13:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:13:45.165 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T13:13:45.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-04-22T13:13:45.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-04-22T13:13:45.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-04-22T13:13:45.240 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T13:13:45.250 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T13:13:45.268 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T13:13:45.269 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi120.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T13:18:24.195 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2024-04-22T13:18:24.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-22T13:18:24.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:18:24.273 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-22T13:18:24.309 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-22T13:18:24.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-04-22T13:18:24.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:18:24.386 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-04-22T13:18:24.423 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-04-22T13:18:24.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-22T13:18:24.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T13:18:24.498 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-04-22T13:18:24.585 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-22T13:18:24.586 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T13:18:24.598 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T13:18:24.599 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T13:18:24.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:18:24.602 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T13:18:24.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:18:24.604 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T13:18:24.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:18:24.628 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T13:18:24.629 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T13:18:24.640 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T13:18:24.642 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T13:18:24.653 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T13:18:24.663 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-04-22T13:18:24.664 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-04-22T13:18:24.667 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T13:18:24.673 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-04-22T13:18:24.675 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-04-22T13:18:24.686 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T13:18:24.688 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T13:18:24.690 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-04-22T13:18:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T13:18:24.700 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-04-22T13:18:24.700 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T13:18:24.713 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T13:18:24.726 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T13:18:24.764 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:18:24.766 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.768 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.770 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.779 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.782 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:18:24.783 INFO:teuthology.run_tasks:Running task install... 2024-04-22T13:18:24.791 DEBUG:teuthology.task.install:project ceph 2024-04-22T13:18:24.791 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-22T13:18:24.791 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-22T13:18:24.791 INFO:teuthology.task.install:Using flavor: default 2024-04-22T13:18:24.797 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T13:18:24.797 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T13:18:24.798 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-22T13:18:24.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:18:24.800 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-22T13:18:24.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:18:24.802 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-22T13:18:24.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:18:24.946 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-22T13:18:24.946 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-22T13:18:25.137 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-22T13:18:25.137 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-22T13:18:25.141 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-22T13:18:25.141 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-22T13:18:25.159 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T13:18:25.160 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:18:25.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T13:18:25.185 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T13:18:25.185 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T13:18:25.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T13:18:25.323 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T13:18:25.333 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T13:18:25.333 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:18:25.333 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T13:18:25.358 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T13:18:25.358 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T13:18:25.387 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-04-22T13:18:25.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-04-22T13:18:25.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T13:18:25.477 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T13:18:25.478 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:18:25.478 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T13:18:25.497 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T13:18:25.510 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T13:18:25.510 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T13:18:25.560 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-04-22T13:18:25.561 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-04-22T13:18:25.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T13:18:25.613 INFO:teuthology.orchestra.run.smithi132.stdout:48 files removed 2024-04-22T13:18:25.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T13:18:25.652 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T13:18:25.716 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-04-22T13:18:25.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-04-22T13:18:25.784 INFO:teuthology.orchestra.run.smithi120.stdout:48 files removed 2024-04-22T13:18:25.810 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T13:18:25.943 INFO:teuthology.orchestra.run.smithi087.stdout:48 files removed 2024-04-22T13:18:25.970 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T13:18:26.674 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 108 kB/s | 83 kB 00:00 2024-04-22T13:18:26.808 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 112 kB/s | 83 kB 00:00 2024-04-22T13:18:26.958 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 115 kB/s | 83 kB 00:00 2024-04-22T13:18:27.008 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-04-22T13:18:27.450 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-22T13:18:27.608 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-22T13:18:27.642 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-04-22T13:18:27.975 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-22T13:18:28.242 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-04-22T13:18:28.975 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T13:18:29.636 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-22T13:18:30.217 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-04-22T13:18:32.035 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T13:18:34.585 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 19 MB 00:04 2024-04-22T13:18:36.701 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-04-22T13:18:36.769 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.2 MB 00:01 2024-04-22T13:18:38.428 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-22T13:18:39.569 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.2 MB 00:01 2024-04-22T13:18:40.560 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-22T13:18:40.869 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-22T13:18:41.045 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-04-22T13:18:42.587 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-22T13:18:44.032 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-22T13:18:45.649 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-22T13:18:47.767 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:18:47.768 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:18:47.778 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T13:18:47.779 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T13:18:47.780 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T13:18:47.781 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T13:18:47.782 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T13:18:47.784 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T13:18:47.785 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T13:18:47.844 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================= 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================= 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T13:18:47.860 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-22T13:18:47.861 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T13:18:47.862 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-04-22T13:18:47.863 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T13:18:47.864 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T13:18:47.865 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T13:18:47.866 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T13:18:47.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T13:18:47.868 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T13:18:47.869 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T13:18:47.870 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T13:18:47.871 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================= 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout:Install 148 Packages 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2024-04-22T13:18:47.872 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:18:47.873 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 218 M 2024-04-22T13:18:47.873 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-04-22T13:18:48.843 INFO:teuthology.orchestra.run.smithi120.stdout:(1/150): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 41 kB/s | 6.6 kB 00:00 2024-04-22T13:18:49.109 INFO:teuthology.orchestra.run.smithi120.stdout:(2/150): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 3.3 MB/s | 890 kB 00:00 2024-04-22T13:18:49.129 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T13:18:49.176 INFO:teuthology.orchestra.run.smithi120.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-22T13:18:49.560 INFO:teuthology.orchestra.run.smithi120.stdout:(4/150): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 6.2 MB/s | 2.4 MB 00:00 2024-04-22T13:18:49.627 INFO:teuthology.orchestra.run.smithi120.stdout:(5/150): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 5.7 MB/s | 5.4 MB 00:00 2024-04-22T13:18:49.819 INFO:teuthology.orchestra.run.smithi120.stdout:(6/150): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 5.8 MB/s | 1.5 MB 00:00 2024-04-22T13:18:50.152 INFO:teuthology.orchestra.run.smithi120.stdout:(7/150): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 8.9 MB/s | 4.7 MB 00:00 2024-04-22T13:18:51.211 INFO:teuthology.orchestra.run.smithi120.stdout:(8/150): ceph-common-19.0.0-3156.gd69f13d8.el7. 8.3 MB/s | 21 MB 00:02 2024-04-22T13:18:51.247 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:18:51.248 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:18:51.258 INFO:teuthology.orchestra.run.smithi132.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T13:18:51.259 INFO:teuthology.orchestra.run.smithi132.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T13:18:51.260 INFO:teuthology.orchestra.run.smithi132.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T13:18:51.261 INFO:teuthology.orchestra.run.smithi132.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T13:18:51.262 INFO:teuthology.orchestra.run.smithi132.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T13:18:51.264 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T13:18:51.265 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T13:18:51.324 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-04-22T13:18:51.345 INFO:teuthology.orchestra.run.smithi120.stdout:(9/150): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 8.9 MB/s | 11 MB 00:01 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-22T13:18:51.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T13:18:51.351 INFO:teuthology.orchestra.run.smithi132.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-22T13:18:51.352 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T13:18:51.353 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T13:18:51.354 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T13:18:51.355 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T13:18:51.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T13:18:51.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T13:18:51.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T13:18:51.359 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T13:18:51.360 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T13:18:51.361 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout:Install 148 Packages 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-04-22T13:18:51.362 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:18:51.363 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 218 M 2024-04-22T13:18:51.363 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-04-22T13:18:51.370 INFO:teuthology.orchestra.run.smithi120.stdout:(10/150): ceph-selinux-19.0.0-3156.gd69f13d8.el 159 kB/s | 25 kB 00:00 2024-04-22T13:18:51.445 INFO:teuthology.orchestra.run.smithi120.stdout:(11/150): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.9 MB/s | 225 kB 00:00 2024-04-22T13:18:51.512 INFO:teuthology.orchestra.run.smithi120.stdout:(12/150): cephfs-shell-19.0.0-3156.gd69f13d8.el 406 kB/s | 27 kB 00:00 2024-04-22T13:18:51.580 INFO:teuthology.orchestra.run.smithi120.stdout:(13/150): libcephfs-devel-19.0.0-3156.gd69f13d8 470 kB/s | 32 kB 00:00 2024-04-22T13:18:51.714 INFO:teuthology.orchestra.run.smithi120.stdout:(14/150): libcephfs2-19.0.0-3156.gd69f13d8.el7. 5.5 MB/s | 747 kB 00:00 2024-04-22T13:18:51.872 INFO:teuthology.orchestra.run.smithi120.stdout:(15/150): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 8.2 MB/s | 17 MB 00:02 2024-04-22T13:18:51.898 INFO:teuthology.orchestra.run.smithi120.stdout:(16/150): libcephsqlite-19.0.0-3156.gd69f13d8.e 888 kB/s | 163 kB 00:00 2024-04-22T13:18:51.923 INFO:teuthology.orchestra.run.smithi120.stdout:(17/150): librados-devel-19.0.0-3156.gd69f13d8. 2.5 MB/s | 126 kB 00:00 2024-04-22T13:18:51.998 INFO:teuthology.orchestra.run.smithi120.stdout:(18/150): libradosstriper1-19.0.0-3156.gd69f13d 4.9 MB/s | 500 kB 00:00 2024-04-22T13:18:52.065 INFO:teuthology.orchestra.run.smithi120.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.gd6 669 kB/s | 45 kB 00:00 2024-04-22T13:18:52.132 INFO:teuthology.orchestra.run.smithi120.stdout:(20/150): python3-ceph-common-19.0.0-3156.gd69f 2.0 MB/s | 135 kB 00:00 2024-04-22T13:18:52.208 INFO:teuthology.orchestra.run.smithi120.stdout:(21/150): python3-cephfs-19.0.0-3156.gd69f13d8. 2.1 MB/s | 161 kB 00:00 2024-04-22T13:18:52.291 INFO:teuthology.orchestra.run.smithi120.stdout:(22/150): python3-rados-19.0.0-3156.gd69f13d8.e 3.8 MB/s | 320 kB 00:00 2024-04-22T13:18:52.383 INFO:teuthology.orchestra.run.smithi120.stdout:(23/150): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 298 kB 00:00 2024-04-22T13:18:52.500 INFO:teuthology.orchestra.run.smithi120.stdout:(24/150): librgw2-19.0.0-3156.gd69f13d8.el7.x86 9.3 MB/s | 5.4 MB 00:00 2024-04-22T13:18:52.525 INFO:teuthology.orchestra.run.smithi120.stdout:(25/150): python3-rgw-19.0.0-3156.gd69f13d8.el7 700 kB/s | 99 kB 00:00 2024-04-22T13:18:52.559 INFO:teuthology.orchestra.run.smithi120.stdout:(26/150): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.4 MB/s | 85 kB 00:00 2024-04-22T13:18:52.651 INFO:teuthology.orchestra.run.smithi120.stdout:(27/150): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.8 MB/s | 170 kB 00:00 2024-04-22T13:18:52.727 INFO:teuthology.orchestra.run.smithi120.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 457 kB/s | 34 kB 00:00 2024-04-22T13:18:52.810 INFO:teuthology.orchestra.run.smithi120.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.7 MB/s | 142 kB 00:00 2024-04-22T13:18:52.869 INFO:teuthology.orchestra.run.smithi120.stdout:(30/150): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 9.0 MB/s | 3.1 MB 00:00 2024-04-22T13:18:52.933 INFO:teuthology.orchestra.run.smithi132.stdout:(1/150): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-22T13:18:53.320 INFO:teuthology.orchestra.run.smithi120.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.gd69f1 7.0 MB/s | 3.5 MB 00:00 2024-04-22T13:18:53.411 INFO:teuthology.orchestra.run.smithi120.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-22T13:18:53.495 INFO:teuthology.orchestra.run.smithi120.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 592 kB/s | 49 kB 00:00 2024-04-22T13:18:53.603 INFO:teuthology.orchestra.run.smithi120.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.gd 152 kB/s | 16 kB 00:00 2024-04-22T13:18:53.666 INFO:teuthology.orchestra.run.smithi132.stdout:(2/150): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 890 kB 00:00 2024-04-22T13:18:53.687 INFO:teuthology.orchestra.run.smithi120.stdout:(35/150): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-04-22T13:18:53.721 INFO:teuthology.orchestra.run.smithi120.stdout:(36/150): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 273 kB 00:00 2024-04-22T13:18:53.733 INFO:teuthology.orchestra.run.smithi132.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-22T13:18:53.772 INFO:teuthology.orchestra.run.smithi120.stdout:(37/150): cephadm-19.0.0-3156.gd69f13d8.el7.noa 8.9 MB/s | 762 kB 00:00 2024-04-22T13:18:53.805 INFO:teuthology.orchestra.run.smithi120.stdout:(38/150): cephfs-top-19.0.0-3156.gd69f13d8.el7. 217 kB/s | 18 kB 00:00 2024-04-22T13:18:53.940 INFO:teuthology.orchestra.run.smithi120.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 482 kB/s | 80 kB 00:00 2024-04-22T13:18:54.140 INFO:teuthology.orchestra.run.smithi120.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-04-22T13:18:54.256 INFO:teuthology.orchestra.run.smithi120.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-22T13:18:54.315 INFO:teuthology.orchestra.run.smithi120.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-04-22T13:18:54.358 INFO:teuthology.orchestra.run.smithi132.stdout:(4/150): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 3.8 MB/s | 2.4 MB 00:00 2024-04-22T13:18:54.415 INFO:teuthology.orchestra.run.smithi120.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-22T13:18:54.592 INFO:teuthology.orchestra.run.smithi132.stdout:(5/150): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-22T13:18:54.907 INFO:teuthology.orchestra.run.smithi120.stdout:(44/150): ceph-test-19.0.0-3156.gd69f13d8.el7.x 14 MB/s | 49 MB 00:03 2024-04-22T13:18:54.932 INFO:teuthology.orchestra.run.smithi120.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 261 kB/s | 135 kB 00:00 2024-04-22T13:18:54.966 INFO:teuthology.orchestra.run.smithi120.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 1.7 MB/s | 1.2 MB 00:00 2024-04-22T13:18:55.050 INFO:teuthology.orchestra.run.smithi120.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-22T13:18:55.075 INFO:teuthology.orchestra.run.smithi120.stdout:(48/150): python3-requests-2.25.1-8.el9.noarch. 745 kB/s | 125 kB 00:00 2024-04-22T13:18:55.075 INFO:teuthology.orchestra.run.smithi132.stdout:(6/150): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 9.7 MB/s | 4.7 MB 00:00 2024-04-22T13:18:55.142 INFO:teuthology.orchestra.run.smithi120.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-22T13:18:55.176 INFO:teuthology.orchestra.run.smithi120.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 646 kB/s | 21 kB 00:00 2024-04-22T13:18:55.209 INFO:teuthology.orchestra.run.smithi120.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 798 kB/s | 107 kB 00:00 2024-04-22T13:18:55.234 INFO:teuthology.orchestra.run.smithi120.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-22T13:18:55.260 INFO:teuthology.orchestra.run.smithi120.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-04-22T13:18:55.302 INFO:teuthology.orchestra.run.smithi120.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-04-22T13:18:55.330 INFO:teuthology.orchestra.run.smithi120.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 271 kB/s | 18 kB 00:00 2024-04-22T13:18:55.353 INFO:teuthology.orchestra.run.smithi120.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-04-22T13:18:55.377 INFO:teuthology.orchestra.run.smithi120.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-22T13:18:55.403 INFO:teuthology.orchestra.run.smithi120.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-22T13:18:55.453 INFO:teuthology.orchestra.run.smithi120.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-22T13:18:55.503 INFO:teuthology.orchestra.run.smithi120.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-22T13:18:55.529 INFO:teuthology.orchestra.run.smithi120.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-22T13:18:55.554 INFO:teuthology.orchestra.run.smithi120.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-04-22T13:18:55.588 INFO:teuthology.orchestra.run.smithi120.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-22T13:18:55.622 INFO:teuthology.orchestra.run.smithi120.stdout:(64/150): libtool-2.4.6-45.el9.x86_64.rpm 2.6 MB/s | 582 kB 00:00 2024-04-22T13:18:55.647 INFO:teuthology.orchestra.run.smithi120.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 600 kB/s | 35 kB 00:00 2024-04-22T13:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-22T13:18:55.739 INFO:teuthology.orchestra.run.smithi120.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-22T13:18:55.792 INFO:teuthology.orchestra.run.smithi132.stdout:(7/150): ceph-common-19.0.0-3156.gd69f13d8.el7. 6.7 MB/s | 21 MB 00:03 2024-04-22T13:18:55.807 INFO:teuthology.orchestra.run.smithi120.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 26 MB/s | 4.1 MB 00:00 2024-04-22T13:18:55.882 INFO:teuthology.orchestra.run.smithi120.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-22T13:18:55.916 INFO:teuthology.orchestra.run.smithi120.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-22T13:18:55.941 INFO:teuthology.orchestra.run.smithi120.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 996 kB/s | 133 kB 00:00 2024-04-22T13:18:55.983 INFO:teuthology.orchestra.run.smithi120.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-22T13:18:56.008 INFO:teuthology.orchestra.run.smithi120.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-22T13:18:56.017 INFO:teuthology.orchestra.run.smithi132.stdout:(8/150): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-22T13:18:56.050 INFO:teuthology.orchestra.run.smithi120.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-04-22T13:18:56.125 INFO:teuthology.orchestra.run.smithi120.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-04-22T13:18:56.134 INFO:teuthology.orchestra.run.smithi132.stdout:(9/150): ceph-selinux-19.0.0-3156.gd69f13d8.el7 216 kB/s | 25 kB 00:00 2024-04-22T13:18:56.150 INFO:teuthology.orchestra.run.smithi120.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-22T13:18:56.176 INFO:teuthology.orchestra.run.smithi120.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-04-22T13:18:56.201 INFO:teuthology.orchestra.run.smithi120.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-04-22T13:18:56.227 INFO:teuthology.orchestra.run.smithi120.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-22T13:18:56.243 INFO:teuthology.orchestra.run.smithi120.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 831 kB/s | 35 kB 00:00 2024-04-22T13:18:56.378 INFO:teuthology.orchestra.run.smithi120.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-22T13:18:56.403 INFO:teuthology.orchestra.run.smithi120.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-04-22T13:18:56.428 INFO:teuthology.orchestra.run.smithi120.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-22T13:18:56.453 INFO:teuthology.orchestra.run.smithi120.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-22T13:18:56.479 INFO:teuthology.orchestra.run.smithi120.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 838 kB/s | 42 kB 00:00 2024-04-22T13:18:56.504 INFO:teuthology.orchestra.run.smithi120.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-22T13:18:56.529 INFO:teuthology.orchestra.run.smithi120.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-22T13:18:56.555 INFO:teuthology.orchestra.run.smithi120.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-22T13:18:56.568 INFO:teuthology.orchestra.run.smithi132.stdout:(10/150): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 11 MB/s | 17 MB 00:01 2024-04-22T13:18:56.580 INFO:teuthology.orchestra.run.smithi120.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T13:18:56.605 INFO:teuthology.orchestra.run.smithi120.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-22T13:18:56.635 INFO:teuthology.orchestra.run.smithi132.stdout:(11/150): cephfs-mirror-19.0.0-3156.gd69f13d8.e 3.3 MB/s | 225 kB 00:00 2024-04-22T13:18:56.689 INFO:teuthology.orchestra.run.smithi120.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-22T13:18:56.714 INFO:teuthology.orchestra.run.smithi120.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-04-22T13:18:56.739 INFO:teuthology.orchestra.run.smithi120.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-04-22T13:18:56.752 INFO:teuthology.orchestra.run.smithi132.stdout:(12/150): ceph-radosgw-19.0.0-3156.gd69f13d8.el 11 MB/s | 11 MB 00:00 2024-04-22T13:18:56.765 INFO:teuthology.orchestra.run.smithi120.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-22T13:18:56.777 INFO:teuthology.orchestra.run.smithi132.stdout:(13/150): cephfs-shell-19.0.0-3156.gd69f13d8.el 190 kB/s | 27 kB 00:00 2024-04-22T13:18:56.790 INFO:teuthology.orchestra.run.smithi120.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 711 kB/s | 36 kB 00:00 2024-04-22T13:18:56.803 INFO:teuthology.orchestra.run.smithi132.stdout:(14/150): libcephfs-devel-19.0.0-3156.gd69f13d8 636 kB/s | 32 kB 00:00 2024-04-22T13:18:56.816 INFO:teuthology.orchestra.run.smithi120.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-04-22T13:18:56.841 INFO:teuthology.orchestra.run.smithi120.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-22T13:18:56.844 INFO:teuthology.orchestra.run.smithi132.stdout:(15/150): libcephfs2-19.0.0-3156.gd69f13d8.el7. 11 MB/s | 747 kB 00:00 2024-04-22T13:18:56.862 INFO:teuthology.orchestra.run.smithi132.stdout:(16/150): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.7 MB/s | 163 kB 00:00 2024-04-22T13:18:56.866 INFO:teuthology.orchestra.run.smithi120.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-22T13:18:56.892 INFO:teuthology.orchestra.run.smithi120.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-22T13:18:56.895 INFO:teuthology.orchestra.run.smithi132.stdout:(17/150): librados-devel-19.0.0-3156.gd69f13d8. 2.5 MB/s | 126 kB 00:00 2024-04-22T13:18:56.917 INFO:teuthology.orchestra.run.smithi120.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-22T13:18:56.920 INFO:teuthology.orchestra.run.smithi132.stdout:(18/150): libradosstriper1-19.0.0-3156.gd69f13d 8.4 MB/s | 500 kB 00:00 2024-04-22T13:18:56.942 INFO:teuthology.orchestra.run.smithi120.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 890 kB/s | 45 kB 00:00 2024-04-22T13:18:56.979 INFO:teuthology.orchestra.run.smithi132.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.gd6 773 kB/s | 45 kB 00:00 2024-04-22T13:18:57.046 INFO:teuthology.orchestra.run.smithi132.stdout:(20/150): python3-ceph-common-19.0.0-3156.gd69f 2.0 MB/s | 135 kB 00:00 2024-04-22T13:18:57.113 INFO:teuthology.orchestra.run.smithi132.stdout:(21/150): python3-cephfs-19.0.0-3156.gd69f13d8. 2.4 MB/s | 161 kB 00:00 2024-04-22T13:18:57.168 INFO:teuthology.orchestra.run.smithi120.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-22T13:18:57.218 INFO:teuthology.orchestra.run.smithi120.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-04-22T13:18:57.222 INFO:teuthology.orchestra.run.smithi132.stdout:(22/150): librgw2-19.0.0-3156.gd69f13d8.el7.x86 16 MB/s | 5.4 MB 00:00 2024-04-22T13:18:57.247 INFO:teuthology.orchestra.run.smithi132.stdout:(23/150): python3-rados-19.0.0-3156.gd69f13d8.e 2.3 MB/s | 320 kB 00:00 2024-04-22T13:18:57.272 INFO:teuthology.orchestra.run.smithi132.stdout:(24/150): python3-rbd-19.0.0-3156.gd69f13d8.el7 5.8 MB/s | 298 kB 00:00 2024-04-22T13:18:57.285 INFO:teuthology.orchestra.run.smithi120.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-04-22T13:18:57.298 INFO:teuthology.orchestra.run.smithi132.stdout:(25/150): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.9 MB/s | 99 kB 00:00 2024-04-22T13:18:57.323 INFO:teuthology.orchestra.run.smithi132.stdout:(26/150): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.7 MB/s | 85 kB 00:00 2024-04-22T13:18:57.385 INFO:teuthology.orchestra.run.smithi120.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-22T13:18:57.390 INFO:teuthology.orchestra.run.smithi132.stdout:(27/150): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.5 MB/s | 170 kB 00:00 2024-04-22T13:18:57.410 INFO:teuthology.orchestra.run.smithi120.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-22T13:18:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:(107/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-22T13:18:57.457 INFO:teuthology.orchestra.run.smithi132.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 514 kB/s | 34 kB 00:00 2024-04-22T13:18:57.461 INFO:teuthology.orchestra.run.smithi120.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 515 kB/s | 26 kB 00:00 2024-04-22T13:18:57.486 INFO:teuthology.orchestra.run.smithi120.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 963 kB/s | 49 kB 00:00 2024-04-22T13:18:57.512 INFO:teuthology.orchestra.run.smithi120.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-22T13:18:57.516 INFO:teuthology.orchestra.run.smithi132.stdout:(29/150): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 14 MB/s | 3.1 MB 00:00 2024-04-22T13:18:57.537 INFO:teuthology.orchestra.run.smithi120.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-22T13:18:57.541 INFO:teuthology.orchestra.run.smithi132.stdout:(30/150): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.7 MB/s | 142 kB 00:00 2024-04-22T13:18:57.604 INFO:teuthology.orchestra.run.smithi120.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-04-22T13:18:57.638 INFO:teuthology.orchestra.run.smithi120.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-22T13:18:57.663 INFO:teuthology.orchestra.run.smithi120.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 737 kB/s | 43 kB 00:00 2024-04-22T13:18:57.688 INFO:teuthology.orchestra.run.smithi120.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 637 kB/s | 32 kB 00:00 2024-04-22T13:18:57.705 INFO:teuthology.orchestra.run.smithi120.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 358 kB/s | 15 kB 00:00 2024-04-22T13:18:57.739 INFO:teuthology.orchestra.run.smithi120.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-22T13:18:57.765 INFO:teuthology.orchestra.run.smithi120.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-22T13:18:57.832 INFO:teuthology.orchestra.run.smithi120.stdout:(119/150): libarrow-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-04-22T13:18:57.857 INFO:teuthology.orchestra.run.smithi120.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-22T13:18:57.858 INFO:teuthology.orchestra.run.smithi132.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.gd69f1 10 MB/s | 3.5 MB 00:00 2024-04-22T13:18:57.890 INFO:teuthology.orchestra.run.smithi120.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-22T13:18:57.916 INFO:teuthology.orchestra.run.smithi120.stdout:(122/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-04-22T13:18:57.933 INFO:teuthology.orchestra.run.smithi132.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.gd6 3.3 MB/s | 249 kB 00:00 2024-04-22T13:18:57.975 INFO:teuthology.orchestra.run.smithi120.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-22T13:18:57.992 INFO:teuthology.orchestra.run.smithi132.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 846 kB/s | 49 kB 00:00 2024-04-22T13:18:58.000 INFO:teuthology.orchestra.run.smithi120.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-22T13:18:58.025 INFO:teuthology.orchestra.run.smithi120.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-22T13:18:58.050 INFO:teuthology.orchestra.run.smithi120.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-22T13:18:58.075 INFO:teuthology.orchestra.run.smithi120.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-04-22T13:18:58.084 INFO:teuthology.orchestra.run.smithi132.stdout:(34/150): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-04-22T13:18:58.101 INFO:teuthology.orchestra.run.smithi120.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-22T13:18:58.109 INFO:teuthology.orchestra.run.smithi132.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3156.gd 141 kB/s | 16 kB 00:00 2024-04-22T13:18:58.126 INFO:teuthology.orchestra.run.smithi120.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-22T13:18:58.134 INFO:teuthology.orchestra.run.smithi132.stdout:(36/150): ceph-volume-19.0.0-3156.gd69f13d8.el7 5.3 MB/s | 273 kB 00:00 2024-04-22T13:18:58.151 INFO:teuthology.orchestra.run.smithi120.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 927 kB/s | 46 kB 00:00 2024-04-22T13:18:58.176 INFO:teuthology.orchestra.run.smithi132.stdout:(37/150): cephadm-19.0.0-3156.gd69f13d8.el7.noa 11 MB/s | 762 kB 00:00 2024-04-22T13:18:58.177 INFO:teuthology.orchestra.run.smithi120.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-22T13:18:58.202 INFO:teuthology.orchestra.run.smithi120.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-22T13:18:58.202 INFO:teuthology.orchestra.run.smithi132.stdout:(38/150): cephfs-top-19.0.0-3156.gd69f13d8.el7. 273 kB/s | 18 kB 00:00 2024-04-22T13:18:58.238 INFO:teuthology.orchestra.run.smithi120.stdout:(133/150): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-04-22T13:18:58.261 INFO:teuthology.orchestra.run.smithi120.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 279 kB/s | 16 kB 00:00 2024-04-22T13:18:58.286 INFO:teuthology.orchestra.run.smithi120.stdout:(135/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-22T13:18:58.320 INFO:teuthology.orchestra.run.smithi120.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-22T13:18:58.335 INFO:teuthology.orchestra.run.smithi132.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 507 kB/s | 80 kB 00:00 2024-04-22T13:18:58.345 INFO:teuthology.orchestra.run.smithi120.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-22T13:18:58.370 INFO:teuthology.orchestra.run.smithi120.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-22T13:18:58.404 INFO:teuthology.orchestra.run.smithi120.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-22T13:18:58.419 INFO:teuthology.orchestra.run.smithi132.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-22T13:18:58.429 INFO:teuthology.orchestra.run.smithi120.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 610 kB/s | 36 kB 00:00 2024-04-22T13:18:58.444 INFO:teuthology.orchestra.run.smithi132.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-22T13:18:58.455 INFO:teuthology.orchestra.run.smithi120.stdout:(141/150): python3-rsa-4.9-2.el9.noarch.rpm 702 kB/s | 59 kB 00:00 2024-04-22T13:18:58.478 INFO:teuthology.orchestra.run.smithi132.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-22T13:18:58.480 INFO:teuthology.orchestra.run.smithi120.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-22T13:18:58.505 INFO:teuthology.orchestra.run.smithi120.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-22T13:18:58.539 INFO:teuthology.orchestra.run.smithi120.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-22T13:18:58.565 INFO:teuthology.orchestra.run.smithi120.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 340 kB/s | 20 kB 00:00 2024-04-22T13:18:58.586 INFO:teuthology.orchestra.run.smithi132.stdout:(43/150): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.2 MB 00:00 2024-04-22T13:18:58.590 INFO:teuthology.orchestra.run.smithi120.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-22T13:18:58.624 INFO:teuthology.orchestra.run.smithi120.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-22T13:18:58.674 INFO:teuthology.orchestra.run.smithi120.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-22T13:18:58.678 INFO:teuthology.orchestra.run.smithi132.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-04-22T13:18:58.762 INFO:teuthology.orchestra.run.smithi132.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-22T13:18:58.787 INFO:teuthology.orchestra.run.smithi132.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 674 kB/s | 135 kB 00:00 2024-04-22T13:18:58.821 INFO:teuthology.orchestra.run.smithi132.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-22T13:18:58.974 INFO:teuthology.orchestra.run.smithi120.stdout:(149/150): librbd1-19.0.0-3156.gd69f13d8.el7.x8 9.0 MB/s | 3.1 MB 00:00 2024-04-22T13:18:59.050 INFO:teuthology.orchestra.run.smithi120.stdout:(150/150): librados2-19.0.0-3156.gd69f13d8.el7. 7.4 MB/s | 3.4 MB 00:00 2024-04-22T13:18:59.052 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-04-22T13:18:59.052 INFO:teuthology.orchestra.run.smithi120.stdout:Total 19 MB/s | 218 MB 00:11 2024-04-22T13:18:59.080 INFO:teuthology.orchestra.run.smithi132.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 689 kB 00:00 2024-04-22T13:18:59.113 INFO:teuthology.orchestra.run.smithi132.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-04-22T13:18:59.139 INFO:teuthology.orchestra.run.smithi132.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-22T13:18:59.155 INFO:teuthology.orchestra.run.smithi132.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 512 kB/s | 21 kB 00:00 2024-04-22T13:18:59.198 INFO:teuthology.orchestra.run.smithi132.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-22T13:18:59.214 INFO:teuthology.orchestra.run.smithi132.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 556 kB/s | 33 kB 00:00 2024-04-22T13:18:59.265 INFO:teuthology.orchestra.run.smithi132.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-22T13:18:59.315 INFO:teuthology.orchestra.run.smithi132.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-22T13:18:59.457 INFO:teuthology.orchestra.run.smithi132.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-04-22T13:18:59.524 INFO:teuthology.orchestra.run.smithi132.stdout:(57/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.2 MB/s | 3.0 MB 00:00 2024-04-22T13:18:59.549 INFO:teuthology.orchestra.run.smithi132.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-04-22T13:18:59.616 INFO:teuthology.orchestra.run.smithi132.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-04-22T13:18:59.650 INFO:teuthology.orchestra.run.smithi132.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-22T13:18:59.676 INFO:teuthology.orchestra.run.smithi132.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-04-22T13:18:59.709 INFO:teuthology.orchestra.run.smithi132.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-04-22T13:18:59.734 INFO:teuthology.orchestra.run.smithi132.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-22T13:18:59.760 INFO:teuthology.orchestra.run.smithi132.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-22T13:18:59.793 INFO:teuthology.orchestra.run.smithi132.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-22T13:19:00.060 INFO:teuthology.orchestra.run.smithi132.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-04-22T13:19:00.094 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-04-22T13:19:00.136 INFO:teuthology.orchestra.run.smithi132.stdout:(67/150): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-22T13:19:00.211 INFO:teuthology.orchestra.run.smithi132.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-22T13:19:00.218 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-04-22T13:19:00.218 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-04-22T13:19:00.253 INFO:teuthology.orchestra.run.smithi132.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-04-22T13:19:00.278 INFO:teuthology.orchestra.run.smithi132.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-22T13:19:00.378 INFO:teuthology.orchestra.run.smithi132.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-22T13:19:00.437 INFO:teuthology.orchestra.run.smithi132.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-22T13:19:00.587 INFO:teuthology.orchestra.run.smithi132.stdout:(73/150): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-22T13:19:00.646 INFO:teuthology.orchestra.run.smithi132.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-22T13:19:00.705 INFO:teuthology.orchestra.run.smithi132.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-22T13:19:00.813 INFO:teuthology.orchestra.run.smithi132.stdout:(76/150): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-22T13:19:00.839 INFO:teuthology.orchestra.run.smithi132.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 192 kB/s | 26 kB 00:00 2024-04-22T13:19:00.864 INFO:teuthology.orchestra.run.smithi132.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-22T13:19:00.897 INFO:teuthology.orchestra.run.smithi132.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-22T13:19:00.923 INFO:teuthology.orchestra.run.smithi132.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 589 kB/s | 35 kB 00:00 2024-04-22T13:19:00.982 INFO:teuthology.orchestra.run.smithi132.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-22T13:19:01.040 INFO:teuthology.orchestra.run.smithi132.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-22T13:19:01.099 INFO:teuthology.orchestra.run.smithi132.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T13:19:01.149 INFO:teuthology.orchestra.run.smithi132.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-22T13:19:01.216 INFO:teuthology.orchestra.run.smithi132.stdout:(85/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-22T13:19:01.250 INFO:teuthology.orchestra.run.smithi132.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 269 kB 00:00 2024-04-22T13:19:01.275 INFO:teuthology.orchestra.run.smithi132.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-22T13:19:01.309 INFO:teuthology.orchestra.run.smithi132.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-04-22T13:19:01.359 INFO:teuthology.orchestra.run.smithi132.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 609 kB/s | 51 kB 00:00 2024-04-22T13:19:01.451 INFO:teuthology.orchestra.run.smithi132.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 378 kB/s | 54 kB 00:00 2024-04-22T13:19:01.518 INFO:teuthology.orchestra.run.smithi132.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 626 kB/s | 42 kB 00:00 2024-04-22T13:19:01.577 INFO:teuthology.orchestra.run.smithi132.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-04-22T13:19:01.635 INFO:teuthology.orchestra.run.smithi132.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T13:19:01.694 INFO:teuthology.orchestra.run.smithi132.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-04-22T13:19:01.753 INFO:teuthology.orchestra.run.smithi132.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 813 kB/s | 47 kB 00:00 2024-04-22T13:19:01.820 INFO:teuthology.orchestra.run.smithi132.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-22T13:19:01.879 INFO:teuthology.orchestra.run.smithi132.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 990 kB/s | 58 kB 00:00 2024-04-22T13:19:01.920 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-04-22T13:19:01.920 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-04-22T13:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:(98/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-22T13:19:02.122 INFO:teuthology.orchestra.run.smithi132.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 387 kB/s | 94 kB 00:00 2024-04-22T13:19:02.146 INFO:teuthology.orchestra.run.smithi132.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T13:19:02.180 INFO:teuthology.orchestra.run.smithi132.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 766 kB/s | 45 kB 00:00 2024-04-22T13:19:02.314 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:21 2024-04-22T13:19:02.730 INFO:teuthology.orchestra.run.smithi132.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-22T13:19:02.905 INFO:teuthology.orchestra.run.smithi132.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-04-22T13:19:02.956 INFO:teuthology.orchestra.run.smithi132.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-04-22T13:19:02.997 INFO:teuthology.orchestra.run.smithi132.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-04-22T13:19:03.190 INFO:teuthology.orchestra.run.smithi132.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-04-22T13:19:03.232 INFO:teuthology.orchestra.run.smithi132.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-04-22T13:19:03.274 INFO:teuthology.orchestra.run.smithi132.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T13:19:03.316 INFO:teuthology.orchestra.run.smithi132.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T13:19:03.358 INFO:teuthology.orchestra.run.smithi132.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-04-22T13:19:03.416 INFO:teuthology.orchestra.run.smithi132.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-04-22T13:19:03.467 INFO:teuthology.orchestra.run.smithi132.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-04-22T13:19:03.508 INFO:teuthology.orchestra.run.smithi132.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-22T13:19:03.550 INFO:teuthology.orchestra.run.smithi132.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 774 kB/s | 32 kB 00:00 2024-04-22T13:19:03.592 INFO:teuthology.orchestra.run.smithi132.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-04-22T13:19:03.642 INFO:teuthology.orchestra.run.smithi132.stdout:(116/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-22T13:19:03.693 INFO:teuthology.orchestra.run.smithi132.stdout:(117/150): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-22T13:19:03.735 INFO:teuthology.orchestra.run.smithi132.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-04-22T13:19:03.768 INFO:teuthology.orchestra.run.smithi132.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 1.0 MB/s | 35 kB 00:00 2024-04-22T13:19:03.827 INFO:teuthology.orchestra.run.smithi132.stdout:(120/150): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-22T13:19:03.903 INFO:teuthology.orchestra.run.smithi132.stdout:(121/150): python3-grpcio-1.46.7-10.el9.x86_64. 27 MB/s | 2.0 MB 00:00 2024-04-22T13:19:03.936 INFO:teuthology.orchestra.run.smithi132.stdout:(122/150): protobuf-compiler-3.14.0-13.el9.x86_ 492 kB/s | 863 kB 00:01 2024-04-22T13:19:03.970 INFO:teuthology.orchestra.run.smithi132.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-22T13:19:04.003 INFO:teuthology.orchestra.run.smithi132.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-22T13:19:04.020 INFO:teuthology.orchestra.run.smithi132.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-22T13:19:04.054 INFO:teuthology.orchestra.run.smithi132.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-04-22T13:19:04.079 INFO:teuthology.orchestra.run.smithi132.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-04-22T13:19:04.104 INFO:teuthology.orchestra.run.smithi132.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-22T13:19:04.147 INFO:teuthology.orchestra.run.smithi132.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-22T13:19:04.180 INFO:teuthology.orchestra.run.smithi132.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-22T13:19:04.205 INFO:teuthology.orchestra.run.smithi132.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-22T13:19:04.231 INFO:teuthology.orchestra.run.smithi132.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-22T13:19:04.264 INFO:teuthology.orchestra.run.smithi132.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-22T13:19:04.290 INFO:teuthology.orchestra.run.smithi132.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-22T13:19:04.323 INFO:teuthology.orchestra.run.smithi132.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-22T13:19:04.349 INFO:teuthology.orchestra.run.smithi132.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-04-22T13:19:04.374 INFO:teuthology.orchestra.run.smithi132.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-22T13:19:04.416 INFO:teuthology.orchestra.run.smithi132.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-22T13:19:04.458 INFO:teuthology.orchestra.run.smithi132.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 860 kB/s | 36 kB 00:00 2024-04-22T13:19:04.491 INFO:teuthology.orchestra.run.smithi132.stdout:(140/150): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-04-22T13:19:04.516 INFO:teuthology.orchestra.run.smithi132.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 875 kB/s | 51 kB 00:00 2024-04-22T13:19:04.559 INFO:teuthology.orchestra.run.smithi132.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-22T13:19:04.617 INFO:teuthology.orchestra.run.smithi132.stdout:(143/150): python3-werkzeug-2.0.3-3.el9.1.noarc 7.2 MB/s | 427 kB 00:00 2024-04-22T13:19:04.643 INFO:teuthology.orchestra.run.smithi132.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-04-22T13:19:04.668 INFO:teuthology.orchestra.run.smithi132.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-22T13:19:04.735 INFO:teuthology.orchestra.run.smithi132.stdout:(146/150): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-22T13:19:04.760 INFO:teuthology.orchestra.run.smithi132.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-22T13:19:05.327 INFO:teuthology.orchestra.run.smithi132.stdout:(148/150): ceph-test-19.0.0-3156.gd69f13d8.el7. 5.3 MB/s | 49 MB 00:09 2024-04-22T13:19:05.394 INFO:teuthology.orchestra.run.smithi132.stdout:(149/150): librbd1-19.0.0-3156.gd69f13d8.el7.x8 5.0 MB/s | 3.1 MB 00:00 2024-04-22T13:19:05.453 INFO:teuthology.orchestra.run.smithi132.stdout:(150/150): librados2-19.0.0-3156.gd69f13d8.el7. 4.7 MB/s | 3.4 MB 00:00 2024-04-22T13:19:05.456 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-22T13:19:05.457 INFO:teuthology.orchestra.run.smithi132.stdout:Total 15 MB/s | 218 MB 00:14 2024-04-22T13:19:06.525 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-04-22T13:19:06.650 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-04-22T13:19:06.650 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-04-22T13:19:07.269 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-22T13:19:07.720 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-04-22T13:19:07.794 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T13:19:07.894 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T13:19:08.047 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T13:19:08.356 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-04-22T13:19:08.357 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-04-22T13:19:08.382 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T13:19:08.503 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:08.633 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:08.680 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:08.771 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:08.813 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:19:08.842 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T13:19:08.868 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T13:19:09.024 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T13:19:09.090 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T13:19:09.119 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:09.205 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:09.228 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:09.295 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:09.313 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:09.336 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:09.378 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:19:09.379 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T13:19:09.390 INFO:teuthology.orchestra.run.smithi087.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T13:19:09.391 INFO:teuthology.orchestra.run.smithi087.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T13:19:09.392 INFO:teuthology.orchestra.run.smithi087.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T13:19:09.392 INFO:teuthology.orchestra.run.smithi087.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T13:19:09.394 INFO:teuthology.orchestra.run.smithi087.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T13:19:09.396 INFO:teuthology.orchestra.run.smithi087.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T13:19:09.397 INFO:teuthology.orchestra.run.smithi087.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T13:19:09.457 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-22T13:19:09.466 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-22T13:19:09.467 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T13:19:09.468 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-22T13:19:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-22T13:19:09.470 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-22T13:19:09.471 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T13:19:09.472 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T13:19:09.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T13:19:09.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T13:19:09.475 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T13:19:09.476 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T13:19:09.477 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T13:19:09.478 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:Install 148 Packages 2024-04-22T13:19:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-04-22T13:19:09.480 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:19:09.480 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 218 M 2024-04-22T13:19:09.480 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-04-22T13:19:09.513 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:09.546 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:09.653 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T13:19:09.752 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T13:19:09.787 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T13:19:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T13:19:09.896 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T13:19:09.930 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T13:19:09.974 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T13:19:10.007 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T13:19:10.035 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T13:19:10.106 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T13:19:10.153 INFO:teuthology.orchestra.run.smithi087.stdout:(1/150): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-22T13:19:10.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T13:19:10.188 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T13:19:10.231 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T13:19:10.310 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T13:19:10.343 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 30/152 2024-04-22T13:19:10.345 INFO:teuthology.orchestra.run.smithi087.stdout:(2/150): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 4.5 MB/s | 890 kB 00:00 2024-04-22T13:19:10.395 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 31/152 2024-04-22T13:19:10.395 INFO:teuthology.orchestra.run.smithi087.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-22T13:19:10.423 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 32/152 2024-04-22T13:19:10.496 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T13:19:10.539 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T13:19:10.614 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T13:19:10.704 INFO:teuthology.orchestra.run.smithi087.stdout:(4/150): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 7.4 MB/s | 5.4 MB 00:00 2024-04-22T13:19:10.752 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T13:19:10.754 INFO:teuthology.orchestra.run.smithi087.stdout:(5/150): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 6.6 MB/s | 2.4 MB 00:00 2024-04-22T13:19:10.817 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T13:19:10.865 INFO:teuthology.orchestra.run.smithi087.stdout:(6/150): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 9.3 MB/s | 1.5 MB 00:00 2024-04-22T13:19:11.274 INFO:teuthology.orchestra.run.smithi087.stdout:(7/150): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 9.1 MB/s | 4.7 MB 00:00 2024-04-22T13:19:11.364 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T13:19:11.454 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T13:19:11.482 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T13:19:11.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T13:19:11.564 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 42/152 2024-04-22T13:19:11.596 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T13:19:11.652 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T13:19:11.726 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T13:19:11.756 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T13:19:11.793 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T13:19:11.840 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T13:19:11.884 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T13:19:11.927 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T13:19:11.969 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T13:19:12.108 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T13:19:12.157 INFO:teuthology.orchestra.run.smithi087.stdout:(8/150): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 13 MB/s | 17 MB 00:01 2024-04-22T13:19:12.161 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T13:19:12.189 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T13:19:12.224 INFO:teuthology.orchestra.run.smithi087.stdout:(9/150): ceph-selinux-19.0.0-3156.gd69f13d8.el7 378 kB/s | 25 kB 00:00 2024-04-22T13:19:12.230 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T13:19:12.326 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T13:19:12.333 INFO:teuthology.orchestra.run.smithi087.stdout:(10/150): ceph-radosgw-19.0.0-3156.gd69f13d8.el 10 MB/s | 11 MB 00:01 2024-04-22T13:19:12.366 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T13:19:12.404 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T13:19:12.424 INFO:teuthology.orchestra.run.smithi087.stdout:(11/150): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.4 MB/s | 225 kB 00:00 2024-04-22T13:19:12.429 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T13:19:12.454 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T13:19:12.491 INFO:teuthology.orchestra.run.smithi087.stdout:(12/150): cephfs-shell-19.0.0-3156.gd69f13d8.el 407 kB/s | 27 kB 00:00 2024-04-22T13:19:12.530 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T13:19:12.550 INFO:teuthology.orchestra.run.smithi087.stdout:(13/150): libcephfs-devel-19.0.0-3156.gd69f13d8 538 kB/s | 32 kB 00:00 2024-04-22T13:19:12.658 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T13:19:12.684 INFO:teuthology.orchestra.run.smithi087.stdout:(14/150): libcephfs2-19.0.0-3156.gd69f13d8.el7. 5.5 MB/s | 747 kB 00:00 2024-04-22T13:19:12.721 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T13:19:12.743 INFO:teuthology.orchestra.run.smithi087.stdout:(15/150): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.7 MB/s | 163 kB 00:00 2024-04-22T13:19:12.749 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T13:19:12.783 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T13:19:12.801 INFO:teuthology.orchestra.run.smithi087.stdout:(16/150): librados-devel-19.0.0-3156.gd69f13d8. 2.1 MB/s | 126 kB 00:00 2024-04-22T13:19:12.827 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T13:19:12.865 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T13:19:12.894 INFO:teuthology.orchestra.run.smithi087.stdout:(17/150): libradosstriper1-19.0.0-3156.gd69f13d 5.3 MB/s | 500 kB 00:00 2024-04-22T13:19:12.905 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T13:19:12.944 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T13:19:12.978 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T13:19:13.011 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T13:19:13.053 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T13:19:13.088 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T13:19:13.176 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T13:19:13.276 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T13:19:13.452 INFO:teuthology.orchestra.run.smithi087.stdout:(18/150): librgw2-19.0.0-3156.gd69f13d8.el7.x86 9.6 MB/s | 5.4 MB 00:00 2024-04-22T13:19:13.464 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T13:19:13.510 INFO:teuthology.orchestra.run.smithi087.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.gd6 776 kB/s | 45 kB 00:00 2024-04-22T13:19:13.578 INFO:teuthology.orchestra.run.smithi087.stdout:(20/150): python3-ceph-common-19.0.0-3156.gd69f 2.0 MB/s | 135 kB 00:00 2024-04-22T13:19:13.589 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T13:19:13.636 INFO:teuthology.orchestra.run.smithi087.stdout:(21/150): python3-cephfs-19.0.0-3156.gd69f13d8. 2.7 MB/s | 161 kB 00:00 2024-04-22T13:19:13.688 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T13:19:13.711 INFO:teuthology.orchestra.run.smithi087.stdout:(22/150): python3-rados-19.0.0-3156.gd69f13d8.e 4.2 MB/s | 320 kB 00:00 2024-04-22T13:19:13.744 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T13:19:13.812 INFO:teuthology.orchestra.run.smithi087.stdout:(23/150): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.9 MB/s | 298 kB 00:00 2024-04-22T13:19:13.870 INFO:teuthology.orchestra.run.smithi087.stdout:(24/150): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 99 kB 00:00 2024-04-22T13:19:13.937 INFO:teuthology.orchestra.run.smithi087.stdout:(25/150): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.3 MB/s | 85 kB 00:00 2024-04-22T13:19:14.154 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-04-22T13:19:14.230 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T13:19:14.304 INFO:teuthology.orchestra.run.smithi087.stdout:(26/150): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 8.4 MB/s | 3.1 MB 00:00 2024-04-22T13:19:14.305 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T13:19:14.380 INFO:teuthology.orchestra.run.smithi087.stdout:(27/150): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 170 kB 00:00 2024-04-22T13:19:14.447 INFO:teuthology.orchestra.run.smithi087.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 513 kB/s | 34 kB 00:00 2024-04-22T13:19:14.488 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T13:19:14.499 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T13:19:14.522 INFO:teuthology.orchestra.run.smithi087.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.9 MB/s | 142 kB 00:00 2024-04-22T13:19:14.541 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T13:19:14.574 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T13:19:14.679 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T13:19:14.742 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T13:19:14.830 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T13:19:14.836 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T13:19:14.860 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T13:19:14.956 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:14.981 INFO:teuthology.orchestra.run.smithi087.stdout:(30/150): ceph-test-19.0.0-3156.gd69f13d8.el7.x 18 MB/s | 49 MB 00:02 2024-04-22T13:19:15.037 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T13:19:15.056 INFO:teuthology.orchestra.run.smithi087.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.gd69f1 6.6 MB/s | 3.5 MB 00:00 2024-04-22T13:19:15.066 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T13:19:15.078 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:15.114 INFO:teuthology.orchestra.run.smithi087.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.gd6 4.2 MB/s | 249 kB 00:00 2024-04-22T13:19:15.116 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:15.144 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T13:19:15.190 INFO:teuthology.orchestra.run.smithi087.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 655 kB/s | 49 kB 00:00 2024-04-22T13:19:15.206 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:15.249 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:19:15.257 INFO:teuthology.orchestra.run.smithi087.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.gd 247 kB/s | 16 kB 00:00 2024-04-22T13:19:15.278 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T13:19:15.304 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T13:19:15.332 INFO:teuthology.orchestra.run.smithi087.stdout:(35/150): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.6 MB/s | 273 kB 00:00 2024-04-22T13:19:15.424 INFO:teuthology.orchestra.run.smithi087.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-04-22T13:19:15.458 INFO:teuthology.orchestra.run.smithi087.stdout:(37/150): cephadm-19.0.0-3156.gd69f13d8.el7.noa 6.0 MB/s | 762 kB 00:00 2024-04-22T13:19:15.465 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T13:19:15.483 INFO:teuthology.orchestra.run.smithi087.stdout:(38/150): cephfs-top-19.0.0-3156.gd69f13d8.el7. 313 kB/s | 18 kB 00:00 2024-04-22T13:19:15.526 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T13:19:15.554 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:15.666 INFO:teuthology.orchestra.run.smithi087.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 385 kB/s | 80 kB 00:00 2024-04-22T13:19:15.667 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:15.697 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:15.760 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:15.782 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:15.805 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:15.806 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T13:19:15.808 INFO:teuthology.orchestra.run.smithi087.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-22T13:19:15.925 INFO:teuthology.orchestra.run.smithi087.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-22T13:19:15.926 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T13:19:15.950 INFO:teuthology.orchestra.run.smithi087.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-04-22T13:19:15.999 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:16.026 INFO:teuthology.orchestra.run.smithi087.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-22T13:19:16.040 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:16.093 INFO:teuthology.orchestra.run.smithi087.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-22T13:19:16.151 INFO:teuthology.orchestra.run.smithi087.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-22T13:19:16.160 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T13:19:16.219 INFO:teuthology.orchestra.run.smithi087.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-04-22T13:19:16.254 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T13:19:16.285 INFO:teuthology.orchestra.run.smithi087.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-04-22T13:19:16.290 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T13:19:16.325 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T13:19:16.400 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T13:19:16.410 INFO:teuthology.orchestra.run.smithi087.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-04-22T13:19:16.441 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T13:19:16.478 INFO:teuthology.orchestra.run.smithi087.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-22T13:19:16.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T13:19:16.526 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T13:19:16.544 INFO:teuthology.orchestra.run.smithi087.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 323 kB/s | 21 kB 00:00 2024-04-22T13:19:16.553 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T13:19:16.623 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T13:19:16.645 INFO:teuthology.orchestra.run.smithi087.stdout:(51/150): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-04-22T13:19:16.677 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T13:19:16.678 INFO:teuthology.orchestra.run.smithi087.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-22T13:19:16.706 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T13:19:16.712 INFO:teuthology.orchestra.run.smithi087.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-04-22T13:19:16.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T13:19:16.779 INFO:teuthology.orchestra.run.smithi087.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 273 kB/s | 18 kB 00:00 2024-04-22T13:19:16.828 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T13:19:16.854 INFO:teuthology.orchestra.run.smithi087.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 149 kB/s | 11 kB 00:00 2024-04-22T13:19:16.861 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 30/152 2024-04-22T13:19:16.913 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 31/152 2024-04-22T13:19:16.913 INFO:teuthology.orchestra.run.smithi087.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-22T13:19:16.941 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 32/152 2024-04-22T13:19:16.996 INFO:teuthology.orchestra.run.smithi087.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 544 kB/s | 45 kB 00:00 2024-04-22T13:19:17.013 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T13:19:17.065 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T13:19:17.141 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T13:19:17.181 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T13:19:17.188 INFO:teuthology.orchestra.run.smithi087.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-04-22T13:19:17.247 INFO:teuthology.orchestra.run.smithi087.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-04-22T13:19:17.264 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T13:19:17.276 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T13:19:17.293 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T13:19:17.314 INFO:teuthology.orchestra.run.smithi087.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 582 kB 00:00 2024-04-22T13:19:17.325 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T13:19:17.325 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T13:19:17.331 INFO:teuthology.orchestra.run.smithi087.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-04-22T13:19:17.365 INFO:teuthology.orchestra.run.smithi087.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 939 kB/s | 47 kB 00:00 2024-04-22T13:19:17.423 INFO:teuthology.orchestra.run.smithi087.stdout:(63/150): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-22T13:19:17.449 INFO:teuthology.orchestra.run.smithi087.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-22T13:19:17.482 INFO:teuthology.orchestra.run.smithi087.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 601 kB/s | 35 kB 00:00 2024-04-22T13:19:17.637 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T13:19:17.689 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:17.779 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:17.810 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 98/152 2024-04-22T13:19:17.846 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T13:19:17.862 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T13:19:17.866 INFO:teuthology.orchestra.run.smithi087.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-04-22T13:19:17.955 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T13:19:17.983 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T13:19:18.033 INFO:teuthology.orchestra.run.smithi087.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-22T13:19:18.043 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T13:19:18.090 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 42/152 2024-04-22T13:19:18.123 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T13:19:18.175 INFO:teuthology.orchestra.run.smithi087.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 6.5 MB/s | 4.7 MB 00:00 2024-04-22T13:19:18.177 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T13:19:18.208 INFO:teuthology.orchestra.run.smithi087.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-22T13:19:18.233 INFO:teuthology.orchestra.run.smithi087.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-22T13:19:18.251 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T13:19:18.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T13:19:18.323 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T13:19:18.344 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T13:19:18.390 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T13:19:18.400 INFO:teuthology.orchestra.run.smithi087.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-04-22T13:19:18.427 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T13:19:18.470 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T13:19:18.476 INFO:teuthology.orchestra.run.smithi087.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-04-22T13:19:18.521 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T13:19:18.543 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:18.639 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:18.666 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T13:19:18.669 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 102/152 2024-04-22T13:19:18.709 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T13:19:18.740 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T13:19:18.743 INFO:teuthology.orchestra.run.smithi087.stdout:(73/150): ceph-common-19.0.0-3156.gd69f13d8.el7 2.4 MB/s | 21 MB 00:08 2024-04-22T13:19:18.781 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T13:19:18.818 INFO:teuthology.orchestra.run.smithi087.stdout:(74/150): postgresql-server-13.14-1.el9.x86_64. 9.5 MB/s | 5.8 MB 00:00 2024-04-22T13:19:18.875 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T13:19:18.902 INFO:teuthology.orchestra.run.smithi087.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-22T13:19:18.914 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T13:19:18.954 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T13:19:18.977 INFO:teuthology.orchestra.run.smithi087.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 342 kB/s | 26 kB 00:00 2024-04-22T13:19:18.980 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T13:19:19.005 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T13:19:19.060 INFO:teuthology.orchestra.run.smithi087.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 230 kB/s | 19 kB 00:00 2024-04-22T13:19:19.088 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T13:19:19.144 INFO:teuthology.orchestra.run.smithi087.stdout:(78/150): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 6.0 MB 00:00 2024-04-22T13:19:19.169 INFO:teuthology.orchestra.run.smithi087.stdout:(79/150): python3-devel-3.9.18-3.el9.x86_64.rpm 573 kB/s | 244 kB 00:00 2024-04-22T13:19:19.195 INFO:teuthology.orchestra.run.smithi087.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-22T13:19:19.215 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T13:19:19.220 INFO:teuthology.orchestra.run.smithi087.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-04-22T13:19:19.262 INFO:teuthology.orchestra.run.smithi087.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-22T13:19:19.273 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T13:19:19.287 INFO:teuthology.orchestra.run.smithi087.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-04-22T13:19:19.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T13:19:19.320 INFO:teuthology.orchestra.run.smithi087.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T13:19:19.342 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T13:19:19.346 INFO:teuthology.orchestra.run.smithi087.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 720 kB/s | 42 kB 00:00 2024-04-22T13:19:19.380 INFO:teuthology.orchestra.run.smithi087.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-22T13:19:19.386 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T13:19:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-22T13:19:19.424 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T13:19:19.438 INFO:teuthology.orchestra.run.smithi087.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-22T13:19:19.464 INFO:teuthology.orchestra.run.smithi087.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-04-22T13:19:19.464 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T13:19:19.494 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T13:19:19.497 INFO:teuthology.orchestra.run.smithi087.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 920 kB/s | 54 kB 00:00 2024-04-22T13:19:19.536 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T13:19:19.564 INFO:teuthology.orchestra.run.smithi087.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 625 kB/s | 42 kB 00:00 2024-04-22T13:19:19.587 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T13:19:19.628 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T13:19:19.631 INFO:teuthology.orchestra.run.smithi087.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 661 kB/s | 44 kB 00:00 2024-04-22T13:19:19.663 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T13:19:19.707 INFO:teuthology.orchestra.run.smithi087.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-22T13:19:19.752 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T13:19:19.774 INFO:teuthology.orchestra.run.smithi087.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 539 kB/s | 36 kB 00:00 2024-04-22T13:19:19.802 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T13:19:19.840 INFO:teuthology.orchestra.run.smithi087.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2024-04-22T13:19:19.916 INFO:teuthology.orchestra.run.smithi087.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-22T13:19:19.991 INFO:teuthology.orchestra.run.smithi087.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 772 kB/s | 58 kB 00:00 2024-04-22T13:19:20.058 INFO:teuthology.orchestra.run.smithi087.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-22T13:19:20.068 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T13:19:20.118 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T13:19:20.125 INFO:teuthology.orchestra.run.smithi087.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 957 kB/s | 64 kB 00:00 2024-04-22T13:19:20.192 INFO:teuthology.orchestra.run.smithi087.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 672 kB/s | 45 kB 00:00 2024-04-22T13:19:20.229 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T13:19:20.286 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T13:19:20.409 INFO:teuthology.orchestra.run.smithi087.stdout:(101/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-22T13:19:20.667 INFO:teuthology.orchestra.run.smithi087.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 47 kB/s | 22 kB 00:00 2024-04-22T13:19:20.899 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:20.933 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:21.018 INFO:teuthology.orchestra.run.smithi087.stdout:(103/150): python3-numpy-1.20.1-5.el9.x86_64.rp 2.8 MB/s | 5.1 MB 00:01 2024-04-22T13:19:21.033 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T13:19:21.083 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T13:19:21.116 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T13:19:21.218 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T13:19:21.275 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T13:19:21.364 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T13:19:21.402 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T13:19:21.478 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:21.520 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:21.578 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T13:19:21.613 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:21.613 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T13:19:21.613 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T13:19:21.613 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:21.616 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T13:19:21.646 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:19:21.703 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T13:19:21.894 INFO:teuthology.orchestra.run.smithi087.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 447 kB/s | 548 kB 00:01 2024-04-22T13:19:21.918 INFO:teuthology.orchestra.run.smithi087.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 343 kB/s | 309 kB 00:00 2024-04-22T13:19:22.018 INFO:teuthology.orchestra.run.smithi087.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-04-22T13:19:22.127 INFO:teuthology.orchestra.run.smithi087.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-22T13:19:22.235 INFO:teuthology.orchestra.run.smithi087.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-22T13:19:22.361 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T13:19:22.436 INFO:teuthology.orchestra.run.smithi087.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-04-22T13:19:22.492 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T13:19:22.728 INFO:teuthology.orchestra.run.smithi087.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-04-22T13:19:23.386 INFO:teuthology.orchestra.run.smithi087.stdout:(111/150): protobuf-compiler-3.14.0-13.el9.x86_ 290 kB/s | 863 kB 00:02 2024-04-22T13:19:23.757 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T13:19:23.855 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T13:19:23.883 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T13:19:23.916 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T13:19:24.045 INFO:teuthology.orchestra.run.smithi087.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 636 kB/s | 838 kB 00:01 2024-04-22T13:19:24.178 INFO:teuthology.orchestra.run.smithi087.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-04-22T13:19:24.227 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T13:19:24.271 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:24.295 INFO:teuthology.orchestra.run.smithi087.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 276 kB/s | 32 kB 00:00 2024-04-22T13:19:24.362 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:24.392 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 98/152 2024-04-22T13:19:24.421 INFO:teuthology.orchestra.run.smithi087.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-04-22T13:19:24.437 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T13:19:24.754 INFO:teuthology.orchestra.run.smithi087.stdout:(116/150): python3-asyncssh-2.13.2-5.el9.noarch 401 kB/s | 548 kB 00:01 2024-04-22T13:19:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 479 kB/s | 172 kB 00:00 2024-04-22T13:19:24.924 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T13:19:24.957 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:25.029 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:25.059 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 102/152 2024-04-22T13:19:25.313 INFO:teuthology.orchestra.run.smithi087.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 568 kB/s | 303 kB 00:00 2024-04-22T13:19:25.355 INFO:teuthology.orchestra.run.smithi087.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 597 kB/s | 358 kB 00:00 2024-04-22T13:19:25.464 INFO:teuthology.orchestra.run.smithi087.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 234 kB/s | 35 kB 00:00 2024-04-22T13:19:25.764 INFO:teuthology.orchestra.run.smithi087.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 496 kB/s | 203 kB 00:00 2024-04-22T13:19:26.081 INFO:teuthology.orchestra.run.smithi087.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 456 kB/s | 144 kB 00:00 2024-04-22T13:19:26.215 INFO:teuthology.orchestra.run.smithi087.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-04-22T13:19:26.340 INFO:teuthology.orchestra.run.smithi087.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-04-22T13:19:26.465 INFO:teuthology.orchestra.run.smithi087.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-04-22T13:19:26.582 INFO:teuthology.orchestra.run.smithi087.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-04-22T13:19:26.733 INFO:teuthology.orchestra.run.smithi087.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 131 kB/s | 20 kB 00:00 2024-04-22T13:19:27.283 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:27.322 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:27.866 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:27.909 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:27.984 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T13:19:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T13:19:27.984 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:28.019 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:19:29.053 INFO:teuthology.orchestra.run.smithi087.stdout:(128/150): python3-grpcio-1.46.7-10.el9.x86_64. 581 kB/s | 2.0 MB 00:03 2024-04-22T13:19:29.116 INFO:teuthology.orchestra.run.smithi087.stdout:(129/150): libarrow-9.0.0-10.el9.x86_64.rpm 628 kB/s | 4.4 MB 00:07 2024-04-22T13:19:29.158 INFO:teuthology.orchestra.run.smithi087.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 440 kB/s | 46 kB 00:00 2024-04-22T13:19:29.225 INFO:teuthology.orchestra.run.smithi087.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 419 kB/s | 1.0 MB 00:02 2024-04-22T13:19:29.342 INFO:teuthology.orchestra.run.smithi087.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 349 kB/s | 79 kB 00:00 2024-04-22T13:19:29.367 INFO:teuthology.orchestra.run.smithi087.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 276 kB/s | 58 kB 00:00 2024-04-22T13:19:29.443 INFO:teuthology.orchestra.run.smithi087.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-22T13:19:29.560 INFO:teuthology.orchestra.run.smithi087.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 469 kB/s | 90 kB 00:00 2024-04-22T13:19:29.576 INFO:teuthology.orchestra.run.smithi087.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 212 kB/s | 28 kB 00:00 2024-04-22T13:19:29.660 INFO:teuthology.orchestra.run.smithi087.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-22T13:19:29.702 INFO:teuthology.orchestra.run.smithi087.stdout:(138/150): python3-pecan-1.4.2-3.el9.noarch.rpm 572 kB/s | 272 kB 00:00 2024-04-22T13:19:29.836 INFO:teuthology.orchestra.run.smithi087.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 268 kB/s | 36 kB 00:00 2024-04-22T13:19:29.852 INFO:teuthology.orchestra.run.smithi087.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-22T13:19:29.944 INFO:teuthology.orchestra.run.smithi087.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 512 kB/s | 188 kB 00:00 2024-04-22T13:19:30.053 INFO:teuthology.orchestra.run.smithi087.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 235 kB/s | 51 kB 00:00 2024-04-22T13:19:30.137 INFO:teuthology.orchestra.run.smithi087.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 467 kB/s | 90 kB 00:00 2024-04-22T13:19:30.245 INFO:teuthology.orchestra.run.smithi087.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-22T13:19:30.270 INFO:teuthology.orchestra.run.smithi087.stdout:(145/150): python3-webob-1.8.7-6.el9.noarch.rpm 550 kB/s | 230 kB 00:00 2024-04-22T13:19:30.612 INFO:teuthology.orchestra.run.smithi087.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 522 kB/s | 192 kB 00:00 2024-04-22T13:19:30.729 INFO:teuthology.orchestra.run.smithi087.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 632 kB/s | 427 kB 00:00 2024-04-22T13:19:30.905 INFO:teuthology.orchestra.run.smithi087.stdout:(148/150): librados2-19.0.0-3156.gd69f13d8.el7. 12 MB/s | 3.4 MB 00:00 2024-04-22T13:19:31.071 INFO:teuthology.orchestra.run.smithi087.stdout:(149/150): librbd1-19.0.0-3156.gd69f13d8.el7.x8 9.2 MB/s | 3.1 MB 00:00 2024-04-22T13:19:32.713 INFO:teuthology.orchestra.run.smithi087.stdout:(150/150): thrift-0.15.0-2.el9.x86_64.rpm 663 kB/s | 1.6 MB 00:02 2024-04-22T13:19:32.719 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-22T13:19:32.720 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.4 MB/s | 218 MB 00:23 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-04-22T13:19:33.162 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-04-22T13:19:33.163 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:33.336 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:19:33.351 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:19:33.351 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:33.351 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T13:19:33.351 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:33.760 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-04-22T13:19:33.885 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-04-22T13:19:33.885 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-04-22T13:19:34.833 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:19:34.862 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:19:34.862 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:34.863 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T13:19:34.863 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:19:34.863 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:19:34.863 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:35.084 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:19:35.107 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:35.560 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:19:35.575 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:35.592 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-04-22T13:19:35.592 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-04-22T13:19:35.682 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T13:19:35.739 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T13:19:35.794 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T13:19:35.980 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T13:19:36.028 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T13:19:36.106 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T13:19:36.194 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T13:19:36.335 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T13:19:36.369 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T13:19:36.409 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T13:19:36.438 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T13:19:36.840 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:19:36.870 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:19:36.902 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 122/152 2024-04-22T13:19:36.964 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 123/152 2024-04-22T13:19:36.993 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:19:38.024 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:19:38.070 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:19:39.034 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:19:39.062 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:19:39.165 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:19:39.333 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 127/152 2024-04-22T13:19:39.365 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:19:39.387 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-04-22T13:19:39.438 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:39.440 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:19:39.452 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:19:39.549 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/152 2024-04-22T13:19:39.597 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 131/152 2024-04-22T13:19:39.616 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:19:39.632 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:19:39.632 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:39.632 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T13:19:39.632 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:40.518 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T13:19:40.543 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:19:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:40.611 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:19:40.632 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:19:40.632 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:40.633 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T13:19:40.633 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:40.678 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:19:40.700 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:41.004 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:19:41.026 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:19:41.026 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:41.026 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T13:19:41.026 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:19:41.027 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:19:41.027 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:41.111 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:19:41.137 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:41.361 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:19:41.384 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:41.604 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-04-22T13:19:41.674 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T13:19:41.699 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T13:19:41.742 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T13:19:41.829 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:19:41.852 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:41.967 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T13:19:42.015 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T13:19:42.071 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T13:19:42.077 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T13:19:42.192 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:42.263 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T13:19:42.313 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T13:19:42.330 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 5/152 2024-04-22T13:19:42.360 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:42.382 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T13:19:42.443 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:42.471 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T13:19:42.484 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:19:42.514 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T13:19:42.539 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T13:19:42.608 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T13:19:42.647 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T13:19:42.679 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T13:19:42.689 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T13:19:42.707 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T13:19:42.737 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T13:19:42.765 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:42.868 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:42.891 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 13/152 2024-04-22T13:19:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T13:19:43.125 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:19:43.156 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:19:43.188 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 122/152 2024-04-22T13:19:43.199 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:43.234 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T13:19:43.249 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 123/152 2024-04-22T13:19:43.279 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:19:43.340 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T13:19:43.437 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T13:19:43.490 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T13:19:43.525 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T13:19:43.599 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T13:19:43.633 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T13:19:43.676 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T13:19:43.708 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T13:19:43.736 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T13:19:43.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T13:19:43.867 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T13:19:43.897 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T13:19:43.933 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T13:19:44.009 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T13:19:44.043 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 30/152 2024-04-22T13:19:44.087 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 31/152 2024-04-22T13:19:44.115 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 32/152 2024-04-22T13:19:44.186 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T13:19:44.239 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T13:19:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T13:19:44.318 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:19:44.356 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:19:44.452 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T13:19:44.508 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T13:19:45.042 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T13:19:45.129 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T13:19:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T13:19:45.200 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T13:19:45.238 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 42/152 2024-04-22T13:19:45.271 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T13:19:45.321 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:19:45.326 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T13:19:45.348 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:19:45.399 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T13:19:45.438 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T13:19:45.449 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:19:45.475 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T13:19:45.522 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T13:19:45.557 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T13:19:45.601 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T13:19:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/152 2024-04-22T13:19:45.622 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 127/152 2024-04-22T13:19:45.643 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T13:19:45.650 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:19:45.657 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/152 2024-04-22T13:19:45.674 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:19:45.674 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:45.674 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T13:19:45.674 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:19:45.675 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:19:45.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:45.698 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/152 2024-04-22T13:19:45.726 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:19:45.730 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T13:19:45.738 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:19:45.758 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T13:19:45.779 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T13:19:45.835 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/152 2024-04-22T13:19:45.860 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T13:19:45.883 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 131/152 2024-04-22T13:19:45.894 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T13:19:45.936 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T13:19:45.942 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 142/152 2024-04-22T13:19:45.988 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T13:19:46.017 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 144/152 2024-04-22T13:19:46.032 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T13:19:46.070 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T13:19:46.110 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T13:19:46.135 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T13:19:46.160 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T13:19:46.235 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T13:19:46.363 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T13:19:46.400 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T13:19:46.428 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T13:19:46.463 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T13:19:46.507 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T13:19:46.545 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T13:19:46.576 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T13:19:46.589 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:46.606 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T13:19:46.625 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:46.640 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T13:19:46.644 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:46.673 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T13:19:46.714 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T13:19:46.749 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T13:19:46.809 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T13:19:46.837 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:19:46.837 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:19:46.860 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:46.880 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T13:19:46.904 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:19:46.911 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T13:19:46.927 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:19:46.927 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:46.927 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T13:19:46.927 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:46.958 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T13:19:46.971 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:19:46.994 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:19:46.995 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:46.995 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T13:19:46.995 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:19:46.995 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:19:46.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:47.066 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T13:19:47.130 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T13:19:47.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:19:47.329 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:19:47.420 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 146/152 2024-04-22T13:19:47.522 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T13:19:47.563 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T13:19:47.645 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T13:19:47.685 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T13:19:47.685 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:19:47.777 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:19:47.777 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:19:47.864 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T13:19:47.909 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T13:19:47.943 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T13:19:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T13:19:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T13:19:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T13:19:48.236 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T13:19:48.414 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T13:19:48.442 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T13:19:48.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T13:19:49.177 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T13:19:49.283 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T13:19:50.533 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T13:19:50.621 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T13:19:50.649 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T13:19:50.682 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T13:19:50.997 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T13:19:51.029 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:51.119 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 97/152 2024-04-22T13:19:51.150 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 98/152 2024-04-22T13:19:51.187 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T13:19:51.664 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T13:19:51.698 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:51.769 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 101/152 2024-04-22T13:19:51.791 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 102/152 2024-04-22T13:19:51.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/152 2024-04-22T13:19:51.967 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/152 2024-04-22T13:19:52.008 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/152 2024-04-22T13:19:52.041 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T13:19:52.068 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T13:19:52.258 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 142/152 2024-04-22T13:19:52.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T13:19:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 144/152 2024-04-22T13:19:52.958 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:53.009 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:53.030 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:19:53.821 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 146/152 2024-04-22T13:19:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T13:19:53.946 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T13:19:54.044 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T13:19:54.077 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T13:19:54.077 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:19:54.092 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:54.129 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:54.152 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:19:54.152 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:19:54.157 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/152 2024-04-22T13:19:54.158 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/152 2024-04-22T13:19:54.159 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/152 2024-04-22T13:19:54.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/152 2024-04-22T13:19:54.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/152 2024-04-22T13:19:54.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/152 2024-04-22T13:19:54.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/152 2024-04-22T13:19:54.160 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/152 2024-04-22T13:19:54.171 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/152 2024-04-22T13:19:54.171 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/152 2024-04-22T13:19:54.171 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T13:19:54.172 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T13:19:54.173 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T13:19:54.174 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T13:19:54.175 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T13:19:54.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T13:19:54.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T13:19:54.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T13:19:54.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T13:19:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/152 2024-04-22T13:19:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/152 2024-04-22T13:19:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:54.803 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/152 2024-04-22T13:19:54.803 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T13:19:54.803 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T13:19:54.803 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:19:54.825 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:19:56.237 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:19:56.237 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.238 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.239 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T13:19:56.240 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T13:19:56.241 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T13:19:56.242 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T13:19:56.243 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T13:19:56.244 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T13:19:56.245 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T13:19:56.246 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T13:19:56.247 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-04-22T13:19:56.248 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-04-22T13:19:56.618 DEBUG:teuthology.parallel:result is None 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:20:00.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/152 2024-04-22T13:20:00.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/152 2024-04-22T13:20:00.293 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/152 2024-04-22T13:20:00.305 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/152 2024-04-22T13:20:00.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T13:20:00.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T13:20:00.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T13:20:00.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T13:20:00.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T13:20:00.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T13:20:00.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T13:20:00.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T13:20:00.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T13:20:00.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/152 2024-04-22T13:20:00.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T13:20:00.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/152 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-04-22T13:20:02.348 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.349 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.350 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.351 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T13:20:02.352 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T13:20:02.353 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T13:20:02.354 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T13:20:02.355 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T13:20:02.356 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T13:20:02.357 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.358 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:20:02.359 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-04-22T13:20:02.735 DEBUG:teuthology.parallel:result is None 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/152 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-04-22T13:20:06.097 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:06.262 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:20:06.277 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 106/152 2024-04-22T13:20:06.277 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:06.277 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T13:20:06.277 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:07.721 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:20:07.746 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/152 2024-04-22T13:20:07.747 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:07.747 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T13:20:07.747 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:20:07.747 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T13:20:07.747 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:07.972 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:20:07.994 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 108/152 2024-04-22T13:20:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T13:20:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:20:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T13:20:07.995 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:08.431 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:20:08.453 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/152 2024-04-22T13:20:08.453 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:08.453 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T13:20:08.453 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:20:08.454 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T13:20:08.454 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:08.561 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T13:20:08.618 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T13:20:08.681 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T13:20:08.866 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T13:20:08.915 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T13:20:08.992 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T13:20:09.089 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T13:20:09.228 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T13:20:09.265 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T13:20:09.298 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T13:20:09.325 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T13:20:09.668 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:20:09.698 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 121/152 2024-04-22T13:20:09.730 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 122/152 2024-04-22T13:20:09.791 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 123/152 2024-04-22T13:20:09.822 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:20:10.857 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 124/152 2024-04-22T13:20:10.907 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:20:11.868 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 125/152 2024-04-22T13:20:11.899 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:20:11.999 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 126/152 2024-04-22T13:20:12.171 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 127/152 2024-04-22T13:20:12.201 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:20:12.224 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 128/152 2024-04-22T13:20:12.224 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:12.224 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T13:20:12.224 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:20:12.225 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T13:20:12.225 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:12.268 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:20:12.279 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 129/152 2024-04-22T13:20:12.377 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/152 2024-04-22T13:20:12.425 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 131/152 2024-04-22T13:20:13.346 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T13:20:13.379 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:20:13.401 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/152 2024-04-22T13:20:13.402 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:13.402 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T13:20:13.402 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:20:13.402 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T13:20:13.402 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:20:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 134/152 2024-04-22T13:20:13.485 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:13.485 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T13:20:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/152 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T13:20:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:13.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:20:13.887 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 136/152 2024-04-22T13:20:13.887 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T13:20:13.887 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T13:20:13.887 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:20:13.888 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T13:20:13.888 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:18.468 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/152 2024-04-22T13:20:18.510 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/152 2024-04-22T13:20:18.550 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/152 2024-04-22T13:20:18.582 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T13:20:18.618 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T13:20:18.798 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 142/152 2024-04-22T13:20:18.848 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T13:20:18.878 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 144/152 2024-04-22T13:20:19.532 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:20:19.575 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:20:19.595 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T13:20:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 146/152 2024-04-22T13:20:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T13:20:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T13:20:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T13:20:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T13:20:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:20:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T13:20:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:20:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:20:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/152 2024-04-22T13:20:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/152 2024-04-22T13:20:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/152 2024-04-22T13:20:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T13:20:26.487 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T13:20:26.488 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T13:20:26.489 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T13:20:26.490 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T13:20:26.491 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T13:20:26.492 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T13:20:26.493 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T13:20:26.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T13:20:26.495 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T13:20:26.495 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T13:20:26.495 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/152 2024-04-22T13:20:26.495 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T13:20:26.495 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/152 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.463 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.464 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.465 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T13:20:28.466 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T13:20:28.467 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T13:20:28.468 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T13:20:28.469 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T13:20:28.470 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T13:20:28.471 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T13:20:28.472 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T13:20:28.473 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T13:20:28.474 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T13:20:28.475 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T13:20:28.476 INFO:teuthology.orchestra.run.smithi087.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T13:20:28.476 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:20:28.476 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-04-22T13:20:28.825 DEBUG:teuthology.parallel:result is None 2024-04-22T13:20:28.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:28.982 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T13:20:29.003 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.004 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.004 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-22T13:20:29.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:29.146 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T13:20:29.171 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.172 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.172 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-22T13:20:29.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:29.317 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T13:20:29.343 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.344 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-22T13:20:29.344 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-22T13:20:29.345 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T13:20:29.346 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:20:29.346 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T13:20:29.371 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:20:29.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T13:20:29.402 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:20:29.402 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T13:20:29.433 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T13:20:29.433 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:20:29.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T13:20:29.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T13:20:29.521 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:20:29.521 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T13:20:29.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T13:20:29.618 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:20:29.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T13:20:29.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T13:20:29.713 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T13:20:29.714 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:20:29.714 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T13:20:29.738 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T13:20:29.803 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:20:29.803 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T13:20:29.827 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T13:20:29.890 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:20:29.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T13:20:29.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T13:20:29.979 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T13:20:29.979 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:20:29.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T13:20:30.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T13:20:30.067 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:20:30.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T13:20:30.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T13:20:30.154 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:20:30.154 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T13:20:30.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T13:20:30.241 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T13:20:30.334 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T13:20:30.334 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:30.335 INFO:tasks.cephadm:Cluster fsid is 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:20:30.335 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T13:20:30.335 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.87', 'mon.b': '172.21.15.120', 'mon.c': '172.21.15.132'} 2024-04-22T13:20:30.335 INFO:tasks.cephadm:First mon is mon.a on smithi087 2024-04-22T13:20:30.335 INFO:tasks.cephadm:First mgr is x 2024-04-22T13:20:30.335 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T13:20:30.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname $(hostname -s) 2024-04-22T13:20:30.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo hostname $(hostname -s) 2024-04-22T13:20:30.383 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-04-22T13:20:30.407 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-22T13:20:30.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:30.552 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T13:20:30.844 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:20:30.845 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-22T13:20:30.845 DEBUG:teuthology.orchestra.run.smithi087:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:31.604 INFO:teuthology.orchestra.run.smithi087.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 22 13:20 /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:31.604 DEBUG:teuthology.orchestra.run.smithi120:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:32.061 INFO:teuthology.orchestra.run.smithi120.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 22 13:20 /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:32.061 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:33.034 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 22 13:20 /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:33.034 DEBUG:teuthology.orchestra.run.smithi087:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:33.053 DEBUG:teuthology.orchestra.run.smithi120:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:33.072 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T13:20:33.105 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-22T13:20:33.105 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-22T13:20:33.108 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-22T13:20:33.115 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-22T13:20:34.594 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-22T13:20:34.632 INFO:teuthology.orchestra.run.smithi087.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-22T13:20:34.659 INFO:teuthology.orchestra.run.smithi120.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-22T13:21:02.764 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout: "repo_digests": [ 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-04-22T13:21:02.765 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-04-22T13:21:07.450 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout: "repo_digests": [ 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:21:07.451 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-22T13:21:22.961 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-22T13:21:22.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2024-04-22T13:21:23.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2024-04-22T13:21:23.053 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-04-22T13:21:23.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /etc/ceph 2024-04-22T13:21:23.107 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /etc/ceph 2024-04-22T13:21:23.134 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-04-22T13:21:23.157 INFO:tasks.cephadm:Writing seed config... 2024-04-22T13:21:23.158 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T13:21:23.158 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T13:21:23.158 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-22T13:21:23.159 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T13:21:23.160 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T13:21:23.161 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T13:21:23.162 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T13:21:23.162 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-22T13:21:23.162 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:21:23.162 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T13:21:23.179 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 176003e8-00ab-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T13:21:23.180 DEBUG:teuthology.orchestra.run.smithi087:mon.a> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T13:21:23.222 DEBUG:teuthology.orchestra.run.smithi087:mgr.x> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T13:21:23.266 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T13:21:23.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.87 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:21:23.456 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-04-22T13:21:23.457 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', '176003e8-00ab-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.87', '--skip-admin-label'] 2024-04-22T13:21:23.457 INFO:teuthology.orchestra.run.smithi087.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T13:21:23.457 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying podman|docker is present... 2024-04-22T13:21:23.482 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T13:21:23.482 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying lvm2 is present... 2024-04-22T13:21:23.482 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying time synchronization is in place... 2024-04-22T13:21:23.489 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T13:21:23.489 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T13:21:23.495 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T13:21:23.495 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:21:23.501 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout enabled 2024-04-22T13:21:23.507 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout active 2024-04-22T13:21:23.507 INFO:teuthology.orchestra.run.smithi087.stdout:Unit chronyd.service is enabled and running 2024-04-22T13:21:23.507 INFO:teuthology.orchestra.run.smithi087.stdout:Repeating the final host check... 2024-04-22T13:21:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T13:21:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T13:21:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl is present 2024-04-22T13:21:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:lvcreate is present 2024-04-22T13:21:23.537 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T13:21:23.537 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T13:21:23.543 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T13:21:23.543 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:21:23.549 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout enabled 2024-04-22T13:21:23.555 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout active 2024-04-22T13:21:23.555 INFO:teuthology.orchestra.run.smithi087.stdout:Unit chronyd.service is enabled and running 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Host looks OK 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Cluster fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Acquiring lock 140140844252320 on /run/cephadm/176003e8-00ab-11ef-bc93-c7b262605968.lock 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Lock 140140844252320 acquired on /run/cephadm/176003e8-00ab-11ef-bc93-c7b262605968.lock 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying IP 172.21.15.87 port 3300 ... 2024-04-22T13:21:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying IP 172.21.15.87 port 6789 ... 2024-04-22T13:21:23.557 INFO:teuthology.orchestra.run.smithi087.stdout:Base mon IP(s) is [172.21.15.87:3300, 172.21.15.87:6789], mon addrv is [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-04-22T13:21:23.559 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.87 metric 100 2024-04-22T13:21:23.560 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.87 metric 100 2024-04-22T13:21:23.562 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T13:21:23.562 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T13:21:23.562 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T13:21:23.564 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T13:21:23.564 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T13:21:23.564 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout inet6 fe80::6bca:fe34:2cb0:648d/64 scope link noprefixroute 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:Mon IP `172.21.15.87` is in CIDR network `172.21.0.0/20` 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:Mon IP `172.21.15.87` is in CIDR network `172.21.0.0/20` 2024-04-22T13:21:23.565 INFO:teuthology.orchestra.run.smithi087.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T13:21:23.566 INFO:teuthology.orchestra.run.smithi087.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T13:21:23.567 INFO:teuthology.orchestra.run.smithi087.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-22T13:21:24.028 INFO:teuthology.orchestra.run.smithi087.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T13:21:24.991 INFO:teuthology.orchestra.run.smithi087.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-22T13:21:24.991 INFO:teuthology.orchestra.run.smithi087.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-22T13:21:24.991 INFO:teuthology.orchestra.run.smithi087.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T13:21:25.955 INFO:teuthology.orchestra.run.smithi087.stdout:stat: stdout 167 167 2024-04-22T13:21:25.955 INFO:teuthology.orchestra.run.smithi087.stdout:Creating initial keys... 2024-04-22T13:21:26.875 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph-authtool: stdout AQBWZCZmy03zEBAAQx8Fb6lBeVhEzl0B1pQ4jA== 2024-04-22T13:21:28.540 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph-authtool: stdout AQBXZCZmi/AgDBAAHH+oQlkCV0iFJ8G0z+GIJQ== 2024-04-22T13:21:37.778 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph-authtool: stdout AQBbZCZmZaQ0GBAAcArKZgGDoc7YQktdlrjlYA== 2024-04-22T13:21:37.779 INFO:teuthology.orchestra.run.smithi087.stdout:Creating initial monmap... 2024-04-22T13:21:41.404 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool for a [v2:172.21.15.87:3300,v1:172.21.15.87:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-04-22T13:21:41.405 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: set fsid to 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:41.406 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T13:21:41.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:21:41.406 INFO:teuthology.orchestra.run.smithi087.stdout:Creating mon... 2024-04-22T13:21:42.393 INFO:teuthology.orchestra.run.smithi087.stdout:create mon.a on 2024-04-22T13:21:42.611 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T13:21:42.756 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T13:21:42.930 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target. 2024-04-22T13:21:42.931 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target. 2024-04-22T13:21:43.140 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a 2024-04-22T13:21:43.140 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Failed to reset failed state of unit ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T13:21:43.301 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target.wants/ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968@.service. 2024-04-22T13:21:43.886 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T13:21:43.886 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout disabled 2024-04-22T13:21:43.893 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T13:21:43.893 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:21:43.893 INFO:teuthology.orchestra.run.smithi087.stdout:firewalld.service is not enabled 2024-04-22T13:21:43.893 INFO:teuthology.orchestra.run.smithi087.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T13:21:43.894 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mon to start... 2024-04-22T13:21:43.894 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mon... 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout id: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout services: 2024-04-22T13:21:45.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.576619s) 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout data: 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.333+0000 7f9594c9c700 1 Processor -- start 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.333+0000 7f9594c9c700 1 -- start start 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f9594c9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f9594c9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f959007bea0 con 0x7f9590108ee0 2024-04-22T13:21:45.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f958e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f958e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57030/0 (socket says 172.21.15.87:57030) 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f958e59c700 1 -- 172.21.15.87:0/475047416 learned_addr learned my addr 172.21.15.87:0/475047416 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.334+0000 7f958e59c700 1 -- 172.21.15.87:0/475047416 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959010b810 con 0x7f9590108ee0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f958e59c700 1 --2- 172.21.15.87:0/475047416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95840098a0 tx=0x7f9584009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407d40ff3c788f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f958d59a700 1 -- 172.21.15.87:0/475047416 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584004160 con 0x7f9590108ee0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f958d59a700 1 -- 172.21.15.87:0/475047416 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584004300 con 0x7f9590108ee0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f958d59a700 1 -- 172.21.15.87:0/475047416 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95840036a0 con 0x7f9590108ee0 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f9594c9c700 1 -- 172.21.15.87:0/475047416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 msgr2=0x7f959010b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:45.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f9594c9c700 1 --2- 172.21.15.87:0/475047416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95840098a0 tx=0x7f9584009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f9594c9c700 1 -- 172.21.15.87:0/475047416 shutdown_connections 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f9594c9c700 1 --2- 172.21.15.87:0/475047416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959010b2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.335+0000 7f9594c9c700 1 -- 172.21.15.87:0/475047416 >> 172.21.15.87:0/475047416 conn(0x7f9590102950 msgr2=0x7f9590104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 -- 172.21.15.87:0/475047416 shutdown_connections 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 -- 172.21.15.87:0/475047416 wait complete. 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 Processor -- start 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 -- start start 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.336+0000 7f9594c9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f959010bc80 con 0x7f9590108ee0 2024-04-22T13:21:45.226 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f958e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f958e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57032/0 (socket says 172.21.15.87:57032) 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f958e59c700 1 -- 172.21.15.87:0/2772328486 learned_addr learned my addr 172.21.15.87:0/2772328486 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f958e59c700 1 -- 172.21.15.87:0/2772328486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959019c070 con 0x7f9590108ee0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f958e59c700 1 --2- 172.21.15.87:0/2772328486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9584000c00 tx=0x7f9584004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584003db0 con 0x7f9590108ee0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584011630 con 0x7f9590108ee0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959019c360 con 0x7f9590108ee0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9584011950 con 0x7f9590108ee0 2024-04-22T13:21:45.227 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959019eff0 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f958401a460 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.337+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f958401b7d0 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.338+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95900640a0 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.340+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f958401b9d0 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.412+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9590063650 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.413+0000 7f957f7fe700 1 -- 172.21.15.87:0/2772328486 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f958401bbb0 con 0x7f9590108ee0 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 msgr2=0x7f959019bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 --2- 172.21.15.87:0/2772328486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9584000c00 tx=0x7f9584004750 comp rx=0 tx=0).stop 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 shutdown_connections 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 --2- 172.21.15.87:0/2772328486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9590108ee0 0x7f959019bb30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:45.228 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 >> 172.21.15.87:0/2772328486 conn(0x7f9590102950 msgr2=0x7f9590104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:45.229 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 shutdown_connections 2024-04-22T13:21:45.229 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:44.414+0000 7f9594c9c700 1 -- 172.21.15.87:0/2772328486 wait complete. 2024-04-22T13:21:45.229 INFO:teuthology.orchestra.run.smithi087.stdout:mon is available 2024-04-22T13:21:45.229 INFO:teuthology.orchestra.run.smithi087.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T13:21:45.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:45 smithi087 ceph-mon[27626]: from='client.? 172.21.15.87:0/1656008047' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout fsid = 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T13:21:46.606 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.663+0000 7f7dbeaf9700 1 Processor -- start 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.663+0000 7f7dbeaf9700 1 -- start start 2024-04-22T13:21:46.607 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.663+0000 7f7dbeaf9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.663+0000 7f7dbeaf9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8108e30 con 0x7f7db8108440 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7dbc895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7dbc895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57040/0 (socket says 172.21.15.87:57040) 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7dbc895700 1 -- 172.21.15.87:0/3997605102 learned_addr learned my addr 172.21.15.87:0/3997605102 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7dbc895700 1 -- 172.21.15.87:0/3997605102 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db81095f0 con 0x7f7db8108440 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7dbc895700 1 --2- 172.21.15.87:0/3997605102 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0098a0 tx=0x7f7dac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d06cb4f5e37721 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7db77fe700 1 -- 172.21.15.87:0/3997605102 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dac004160 con 0x7f7db8108440 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7db77fe700 1 -- 172.21.15.87:0/3997605102 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dac004300 con 0x7f7db8108440 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.664+0000 7f7db77fe700 1 -- 172.21.15.87:0/3997605102 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dac0036a0 con 0x7f7db8108440 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/3997605102 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 msgr2=0x7f7db8108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 --2- 172.21.15.87:0/3997605102 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0098a0 tx=0x7f7dac009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/3997605102 shutdown_connections 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 --2- 172.21.15.87:0/3997605102 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db8108860 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:46.608 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/3997605102 >> 172.21.15.87:0/3997605102 conn(0x7f7db8103ae0 msgr2=0x7f7db8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/3997605102 shutdown_connections 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.665+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/3997605102 wait complete. 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbeaf9700 1 Processor -- start 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbeaf9700 1 -- start start 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbeaf9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbeaf9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8109b20 con 0x7f7db8108440 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbc895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbc895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57050/0 (socket says 172.21.15.87:57050) 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbc895700 1 -- 172.21.15.87:0/1656008047 learned_addr learned my addr 172.21.15.87:0/1656008047 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbc895700 1 -- 172.21.15.87:0/1656008047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db819c070 con 0x7f7db8108440 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbc895700 1 --2- 172.21.15.87:0/1656008047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7dac000c00 tx=0x7f7dac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dac003db0 con 0x7f7db8108440 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dac011630 con 0x7f7db8108440 2024-04-22T13:21:46.609 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dac011950 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.666+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db819c300 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.667+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db819eff0 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.667+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7dac011af0 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.667+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7dac01c450 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.668+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db80640a0 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.670+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7dac032020 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.737+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7db802e100 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.740+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f7dac004260 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.741+0000 7f7db5ffb700 1 -- 172.21.15.87:0/1656008047 <== mon.0 v2:172.21.15.87:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7dac01c6a0 con 0x7f7db8108440 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.741+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 msgr2=0x7f7db819bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.741+0000 7f7dbeaf9700 1 --2- 172.21.15.87:0/1656008047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7dac000c00 tx=0x7f7dac004750 comp rx=0 tx=0).stop 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.742+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 shutdown_connections 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.742+0000 7f7dbeaf9700 1 --2- 172.21.15.87:0/1656008047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7db8108440 0x7f7db819bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.742+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 >> 172.21.15.87:0/1656008047 conn(0x7f7db8103ae0 msgr2=0x7f7db81927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:46.610 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.742+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 shutdown_connections 2024-04-22T13:21:46.611 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:45.742+0000 7f7dbeaf9700 1 -- 172.21.15.87:0/1656008047 wait complete. 2024-04-22T13:21:46.611 INFO:teuthology.orchestra.run.smithi087.stdout:Generating new minimal ceph.conf... 2024-04-22T13:21:47.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:47 smithi087 ceph-mon[27626]: from='client.? 172.21.15.87:0/2676768757' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:21:47.618 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f2b3f700 1 Processor -- start 2024-04-22T13:21:47.618 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f2b3f700 1 -- start start 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f2b3f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f2b3f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f44ec108e30 con 0x7f44ec108440 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f08db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f08db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57064/0 (socket says 172.21.15.87:57064) 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f08db700 1 -- 172.21.15.87:0/3816927793 learned_addr learned my addr 172.21.15.87:0/3816927793 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.015+0000 7f44f08db700 1 -- 172.21.15.87:0/3816927793 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ec1095f0 con 0x7f44ec108440 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f08db700 1 --2- 172.21.15.87:0/3816927793 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f44e00098a0 tx=0x7f44e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b502cdb0c2927e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44eb7fe700 1 -- 172.21.15.87:0/3816927793 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e0004160 con 0x7f44ec108440 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44eb7fe700 1 -- 172.21.15.87:0/3816927793 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44e0004300 con 0x7f44ec108440 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44eb7fe700 1 -- 172.21.15.87:0/3816927793 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e0011af0 con 0x7f44ec108440 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 -- 172.21.15.87:0/3816927793 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 msgr2=0x7f44ec108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 --2- 172.21.15.87:0/3816927793 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f44e00098a0 tx=0x7f44e0009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:47.619 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 -- 172.21.15.87:0/3816927793 shutdown_connections 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 --2- 172.21.15.87:0/3816927793 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec108860 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 -- 172.21.15.87:0/3816927793 >> 172.21.15.87:0/3816927793 conn(0x7f44ec103ae0 msgr2=0x7f44ec105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 -- 172.21.15.87:0/3816927793 shutdown_connections 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.016+0000 7f44f2b3f700 1 -- 172.21.15.87:0/3816927793 wait complete. 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f2b3f700 1 Processor -- start 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f2b3f700 1 -- start start 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f2b3f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f2b3f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f44ec10c9a0 con 0x7f44ec108440 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f08db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f08db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57074/0 (socket says 172.21.15.87:57074) 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.017+0000 7f44f08db700 1 -- 172.21.15.87:0/2676768757 learned_addr learned my addr 172.21.15.87:0/2676768757 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44f08db700 1 -- 172.21.15.87:0/2676768757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ec19c8c0 con 0x7f44ec108440 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44f08db700 1 --2- 172.21.15.87:0/2676768757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f44e0003f40 tx=0x7f44e0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:47.620 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e00038b0 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44e0003a50 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e001a5c0 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ec19cb50 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ec19f840 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44e001ac00 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.018+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f44e0019700 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.019+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ec0640a0 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.021+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f44e0019900 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.088+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f44ec19fa10 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44e9ffb700 1 -- 172.21.15.87:0/2676768757 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f44e0019ae0 con 0x7f44ec108440 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 msgr2=0x7f44ec19c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44f2b3f700 1 --2- 172.21.15.87:0/2676768757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f44e0003f40 tx=0x7f44e0011890 comp rx=0 tx=0).stop 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 shutdown_connections 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44f2b3f700 1 --2- 172.21.15.87:0/2676768757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44ec108440 0x7f44ec19c380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:47.621 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.089+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 >> 172.21.15.87:0/2676768757 conn(0x7f44ec103ae0 msgr2=0x7f44ec1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:47.622 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.090+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 shutdown_connections 2024-04-22T13:21:47.622 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:47.090+0000 7f44f2b3f700 1 -- 172.21.15.87:0/2676768757 wait complete. 2024-04-22T13:21:47.622 INFO:teuthology.orchestra.run.smithi087.stdout:Restarting the monitor... 2024-04-22T13:21:47.882 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:47 smithi087 systemd[1]: Stopping Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:21:48.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a[27622]: 2024-04-22T13:21:47.884+0000 7fca9fe92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:21:48.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a[27622]: 2024-04-22T13:21:47.884+0000 7fca9fe92700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T13:21:48.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 podman[27871]: 2024-04-22 13:21:48.129780502 +0000 UTC m=+0.350203140 container died 985aab37d88fe7d23b64fa93fb53ef93f6962db2e440bed9e671786b3cb7889d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T13:21:48.639 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 podman[27871]: 2024-04-22 13:21:48.425284133 +0000 UTC m=+0.645706768 container cleanup 985aab37d88fe7d23b64fa93fb53ef93f6962db2e440bed9e671786b3cb7889d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T13:21:48.639 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 bash[27871]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a 2024-04-22T13:21:48.975 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 podman[27885]: 2024-04-22 13:21:48.642682557 +0000 UTC m=+0.514323703 container remove 985aab37d88fe7d23b64fa93fb53ef93f6962db2e440bed9e671786b3cb7889d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:21:48.975 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 systemd[1]: ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T13:21:48.975 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 systemd[1]: Stopped Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:21:48.975 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 systemd[1]: Starting Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:21:49.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 podman[27980]: 2024-04-22 13:21:48.881984426 +0000 UTC m=+0.017672716 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:48 smithi087 podman[27980]: 2024-04-22 13:21:48.993610154 +0000 UTC m=+0.129298442 container create 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 podman[27980]: 2024-04-22 13:21:49.17920864 +0000 UTC m=+0.314896931 container init 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 podman[27980]: 2024-04-22 13:21:49.181943484 +0000 UTC m=+0.317631774 container start 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: pidfile_write: ignore empty --pid-file 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: load: jerasure load: lrc 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: RocksDB version: 7.9.2 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Git sha 0 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: DB SUMMARY 2024-04-22T13:21:49.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: DB Session ID: 009D0F8CC4YOMZ8K5Y15 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: CURRENT file: CURRENT 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105124 ; 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.error_if_exists: 0 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.create_if_missing: 0 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.paranoid_checks: 1 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.env: 0x557caee7bcc0 2024-04-22T13:21:49.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.info_log: 0x557cb171dbe0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.statistics: (nil) 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.use_fsync: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_log_file_size: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_fallocate: 1 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.use_direct_reads: 0 2024-04-22T13:21:49.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.db_log_dir: 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.wal_dir: 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.write_buffer_manager: 0x557cb16b10e0 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T13:21:49.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T13:21:49.243 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T13:21:49.243 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.unordered_write: 0 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T13:21:49.244 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.row_cache: None 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.wal_filter: None 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.two_write_queues: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.wal_compression: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.atomic_flush: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T13:21:49.245 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.log_readahead_size: 0 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_background_jobs: 2 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_background_compactions: -1 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_subcompactions: 1 2024-04-22T13:21:49.246 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_open_files: -1 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T13:21:49.247 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_background_flushes: -1 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Compression algorithms supported: 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kZSTD supported: 0 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kXpressCompression supported: 0 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kZlibCompression supported: 1 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kSnappyCompression supported: 1 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kLZ4Compression supported: 1 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: kBZip2Compression supported: 0 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T13:21:49.248 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.merge_operator: 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_filter: None 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557cb171ddc0) 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: cache_index_and_filter_blocks: 1 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: pin_top_level_index_and_filter: 1 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: index_type: 0 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: data_block_index_type: 0 2024-04-22T13:21:49.249 INFO:journalctl@ceph.mon.a.smithi087.stdout: index_shortening: 1 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: checksum: 4 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: no_block_cache: 0 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_cache: 0x557cb16e31f0 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_cache_name: BinnedLRUCache 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_cache_options: 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: capacity : 536870912 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: num_shard_bits : 4 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: strict_capacity_limit : 0 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: high_pri_pool_ratio: 0.000 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_cache_compressed: (nil) 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: persistent_cache: (nil) 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_size: 4096 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_size_deviation: 10 2024-04-22T13:21:49.250 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_restart_interval: 16 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: index_block_restart_interval: 1 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: metadata_block_size: 4096 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: partition_filters: 0 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: use_delta_encoding: 1 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: filter_policy: bloomfilter 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: whole_key_filtering: 1 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: verify_compression: 0 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: read_amp_bytes_per_bit: 0 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: format_version: 5 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: enable_index_compression: 1 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: block_align: 0 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: max_auto_readahead_size: 262144 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: prepopulate_block_cache: 0 2024-04-22T13:21:49.251 INFO:journalctl@ceph.mon.a.smithi087.stdout: initial_auto_readahead_size: 8192 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression: NoCompression 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.num_levels: 7 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T13:21:49.252 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T13:21:49.253 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T13:21:49.254 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T13:21:49.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.inplace_update_support: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.bloom_locality: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.max_successive_merges: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.ttl: 2592000 2024-04-22T13:21:49.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enable_blob_files: false 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.min_blob_size: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6d74c400-433b-489f-863d-bdb954bff3a2 2024-04-22T13:21:49.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792109210364, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792109211389, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100291, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98176, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14169, "raw_average_key_size": 49, "raw_value_size": 90460, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713792109, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6d74c400-433b-489f-863d-bdb954bff3a2", "db_session_id": "009D0F8CC4YOMZ8K5Y15", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792109211470, "job": 1, "event": "recovery_finished"} 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557cb17f8000 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: DB pointer 0x557cb17e2000 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout: ** DB Stats ** 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:21:49.258 INFO:journalctl@ceph.mon.a.smithi087.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: ** Compaction Stats [default] ** 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: L0 2/0 99.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Sum 2/0 99.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: ** Compaction Stats [default] ** 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T13:21:49.259 INFO:journalctl@ceph.mon.a.smithi087.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Cumulative compaction: 0.00 GB write, 31.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Interval compaction: 0.00 GB write, 31.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Block cache BinnedLRUCache@0x557cb16e31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.260 INFO:journalctl@ceph.mon.a.smithi087.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: starting mon.a rank 0 at public addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] at bind addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???) e1 preinit fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).mds e1 new map 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).mds e1 print_map 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: e1 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: legacy client fscid: -1 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout: No filesystems configured 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:21:49.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T13:21:49.262 INFO:teuthology.orchestra.run.smithi087.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 bash[27980]: 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 systemd[1]: Started Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: monmap epoch 1 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: last_changed 2024-04-22T13:21:40.526906+0000 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: min_mon_release 19 (squid) 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: election_strategy: 1 2024-04-22T13:21:49.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:21:49.511 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: fsmap 2024-04-22T13:21:49.511 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T13:21:49.511 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:49 smithi087 ceph-mon[27999]: mgrmap e1: no daemons active 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed1defd700 1 Processor -- start 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed1defd700 1 -- start start 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed1defd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed1defd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fed18107d40 con 0x7fed18107350 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33550/0 (socket says 172.21.15.87:33550) 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed177fe700 1 -- 172.21.15.87:0/3522852981 learned_addr learned my addr 172.21.15.87:0/3522852981 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.691+0000 7fed177fe700 1 -- 172.21.15.87:0/3522852981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18108550 con 0x7fed18107350 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed177fe700 1 --2- 172.21.15.87:0/3522852981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d180 tx=0x7fed0800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d20d992dad5cf15 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed167fc700 1 -- 172.21.15.87:0/3522852981 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08019070 con 0x7fed18107350 2024-04-22T13:21:50.222 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed167fc700 1 -- 172.21.15.87:0/3522852981 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed0800de20 con 0x7fed18107350 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed167fc700 1 -- 172.21.15.87:0/3522852981 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08009a70 con 0x7fed18107350 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 -- 172.21.15.87:0/3522852981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 msgr2=0x7fed18107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 --2- 172.21.15.87:0/3522852981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d180 tx=0x7fed0800d490 comp rx=0 tx=0).stop 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 -- 172.21.15.87:0/3522852981 shutdown_connections 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 --2- 172.21.15.87:0/3522852981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed18107770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 -- 172.21.15.87:0/3522852981 >> 172.21.15.87:0/3522852981 conn(0x7fed18102950 msgr2=0x7fed18104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 -- 172.21.15.87:0/3522852981 shutdown_connections 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.692+0000 7fed1defd700 1 -- 172.21.15.87:0/3522852981 wait complete. 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed1defd700 1 Processor -- start 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed1defd700 1 -- start start 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed1defd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed1defd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fed1807afe0 con 0x7fed18107350 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33558/0 (socket says 172.21.15.87:33558) 2024-04-22T13:21:50.223 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.693+0000 7fed177fe700 1 -- 172.21.15.87:0/473748510 learned_addr learned my addr 172.21.15.87:0/473748510 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed177fe700 1 -- 172.21.15.87:0/473748510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1819eaf0 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed177fe700 1 --2- 172.21.15.87:0/473748510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed0800bd10 tx=0x7fed0800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08019040 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed08003c60 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed0801f7a0 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1819ed80 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed181a1a70 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed0801fa50 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.694+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed08011050 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.695+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed180640a0 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.697+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fed080045b0 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.766+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fed1802e200 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.768+0000 7fed14ff9700 1 -- 172.21.15.87:0/473748510 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fed08004000 con 0x7fed18107350 2024-04-22T13:21:50.224 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.769+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 msgr2=0x7fed1819e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.769+0000 7fed1defd700 1 --2- 172.21.15.87:0/473748510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed0800bd10 tx=0x7fed0800dda0 comp rx=0 tx=0).stop 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.769+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 shutdown_connections 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.769+0000 7fed1defd700 1 --2- 172.21.15.87:0/473748510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed18107350 0x7fed1819e5b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.769+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 >> 172.21.15.87:0/473748510 conn(0x7fed18102950 msgr2=0x7fed18190d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.770+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 shutdown_connections 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:49.770+0000 7fed1defd700 1 -- 172.21.15.87:0/473748510 wait complete. 2024-04-22T13:21:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T13:21:50.247 INFO:teuthology.orchestra.run.smithi087.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:21:50.247 INFO:teuthology.orchestra.run.smithi087.stdout:Creating mgr... 2024-04-22T13:21:50.248 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T13:21:50.248 INFO:teuthology.orchestra.run.smithi087.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T13:21:50.488 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x 2024-04-22T13:21:50.489 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Failed to reset failed state of unit ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-22T13:21:50.638 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968.target.wants/ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-176003e8-00ab-11ef-bc93-c7b262605968@.service. 2024-04-22T13:21:50.765 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:50 smithi087 systemd[1]: Starting Ceph mgr.x for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:21:51.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:50 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/473748510' entity='client.admin' 2024-04-22T13:21:51.086 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:50 smithi087 podman[28220]: 2024-04-22 13:21:50.878644139 +0000 UTC m=+0.110130357 container create e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T13:21:51.086 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:50 smithi087 podman[28220]: 2024-04-22 13:21:50.786257856 +0000 UTC m=+0.017744051 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:21:51.086 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 podman[28220]: 2024-04-22 13:21:51.089225619 +0000 UTC m=+0.320711790 container init e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T13:21:51.191 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T13:21:51.191 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout disabled 2024-04-22T13:21:51.198 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T13:21:51.198 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:21:51.198 INFO:teuthology.orchestra.run.smithi087.stdout:firewalld.service is not enabled 2024-04-22T13:21:51.199 INFO:teuthology.orchestra.run.smithi087.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T13:21:51.207 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T13:21:51.207 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout disabled 2024-04-22T13:21:51.214 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T13:21:51.214 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:21:51.214 INFO:teuthology.orchestra.run.smithi087.stdout:firewalld.service is not enabled 2024-04-22T13:21:51.215 INFO:teuthology.orchestra.run.smithi087.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-22T13:21:51.215 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mgr to start... 2024-04-22T13:21:51.215 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mgr... 2024-04-22T13:21:51.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 podman[28220]: 2024-04-22 13:21:51.0918783 +0000 UTC m=+0.323364473 container start e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:21:51.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 bash[28220]: e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 2024-04-22T13:21:51.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 systemd[1]: Started Ceph mgr.x for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:21:51.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:51.244+0000 7f660652e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:21:51.671 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:51.434+0000 7f660652e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:21:51.672 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:51.542+0000 7f660652e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:21:51.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/478579778' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T13:21:51.947 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:51.864+0000 7f660652e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:21:52.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:51.949+0000 7f660652e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 0 2024-04-22T13:21:52.343 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T13:21:52.344 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T13:21:52.345 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:52.346 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.351 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T13:21:43.838839+0000", 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.683+0000 7f91dfa91700 1 Processor -- start 2024-04-22T13:21:52.352 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.683+0000 7f91dfa91700 1 -- start start 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.683+0000 7f91dfa91700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.683+0000 7f91dfa91700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f91d807be80 con 0x7f91d8108f00 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.684+0000 7f91dd82d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.684+0000 7f91dd82d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33582/0 (socket says 172.21.15.87:33582) 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.684+0000 7f91dd82d700 1 -- 172.21.15.87:0/2704291301 learned_addr learned my addr 172.21.15.87:0/2704291301 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.684+0000 7f91dd82d700 1 -- 172.21.15.87:0/2704291301 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d810b830 con 0x7f91d8108f00 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.684+0000 7f91dd82d700 1 --2- 172.21.15.87:0/2704291301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91d40098a0 tx=0x7f91d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe868d0c6d1f3db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dc82b700 1 -- 172.21.15.87:0/2704291301 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91d4004160 con 0x7f91d8108f00 2024-04-22T13:21:52.353 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dc82b700 1 -- 172.21.15.87:0/2704291301 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f91d4004300 con 0x7f91d8108f00 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dc82b700 1 -- 172.21.15.87:0/2704291301 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91d4011af0 con 0x7f91d8108f00 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 -- 172.21.15.87:0/2704291301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 msgr2=0x7f91d810b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 --2- 172.21.15.87:0/2704291301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91d40098a0 tx=0x7f91d4009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 -- 172.21.15.87:0/2704291301 shutdown_connections 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 --2- 172.21.15.87:0/2704291301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d810b2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 -- 172.21.15.87:0/2704291301 >> 172.21.15.87:0/2704291301 conn(0x7f91d8102950 msgr2=0x7f91d8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 -- 172.21.15.87:0/2704291301 shutdown_connections 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.685+0000 7f91dfa91700 1 -- 172.21.15.87:0/2704291301 wait complete. 2024-04-22T13:21:52.354 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dfa91700 1 Processor -- start 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dfa91700 1 -- start start 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dfa91700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dfa91700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f91d810c7f0 con 0x7f91d8108f00 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dd82d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dd82d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33586/0 (socket says 172.21.15.87:33586) 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.686+0000 7f91dd82d700 1 -- 172.21.15.87:0/478579778 learned_addr learned my addr 172.21.15.87:0/478579778 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91dd82d700 1 -- 172.21.15.87:0/478579778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d819c930 con 0x7f91d8108f00 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91dd82d700 1 --2- 172.21.15.87:0/478579778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009fb0 tx=0x7f91d4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:52.355 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91d40036a0 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f91d401a430 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d819cbc0 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d819f8b0 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91d4019550 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f91d4019b90 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.687+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f91d4024730 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.688+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d80640a0 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.690+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f91d4043e50 con 0x7f91d8108f00 2024-04-22T13:21:52.356 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.763+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f91d819fdf0 con 0x7f91d8108f00 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.763+0000 7f91ceffd700 1 -- 172.21.15.87:0/478579778 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f91d4032030 con 0x7f91d8108f00 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 msgr2=0x7f91d819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 --2- 172.21.15.87:0/478579778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009fb0 tx=0x7f91d4011940 comp rx=0 tx=0).stop 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 shutdown_connections 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 --2- 172.21.15.87:0/478579778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91d8108f00 0x7f91d819c3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 >> 172.21.15.87:0/478579778 conn(0x7f91d8102950 msgr2=0x7f91d81034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 shutdown_connections 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:51.764+0000 7f91dfa91700 1 -- 172.21.15.87:0/478579778 wait complete. 2024-04-22T13:21:52.357 INFO:teuthology.orchestra.run.smithi087.stdout:mgr not available, waiting (1/15)... 2024-04-22T13:21:53.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:53.158+0000 7f660652e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:21:53.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:53.325+0000 7f660652e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:21:53.779 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:53.494+0000 7f660652e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:21:54.073 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:53.782+0000 7f660652e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:21:54.073 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:53.874+0000 7f660652e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:21:54.073 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:54.076+0000 7f660652e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:21:54.477 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:54.267+0000 7f660652e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:21:55.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1933941127' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T13:21:55.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:54.880+0000 7f660652e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T13:21:55.511 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 0 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T13:21:55.512 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T13:21:55.513 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T13:21:55.521 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T13:21:55.522 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T13:21:43.838839+0000", 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f120753c700 1 Processor -- start 2024-04-22T13:21:55.523 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f120753c700 1 -- start start 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f120753c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f120753c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f120007be40 con 0x7f1200108ee0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f12052d8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f12052d8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33596/0 (socket says 172.21.15.87:33596) 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.800+0000 7f12052d8700 1 -- 172.21.15.87:0/2689034981 learned_addr learned my addr 172.21.15.87:0/2689034981 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.801+0000 7f12052d8700 1 -- 172.21.15.87:0/2689034981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120010b810 con 0x7f1200108ee0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.801+0000 7f12052d8700 1 --2- 172.21.15.87:0/2689034981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0098a0 tx=0x7f11fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b28c8aef8ad8c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.801+0000 7f11f7fff700 1 -- 172.21.15.87:0/2689034981 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11fc004160 con 0x7f1200108ee0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.801+0000 7f11f7fff700 1 -- 172.21.15.87:0/2689034981 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f11fc004300 con 0x7f1200108ee0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.801+0000 7f11f7fff700 1 -- 172.21.15.87:0/2689034981 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11fc011af0 con 0x7f1200108ee0 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- 172.21.15.87:0/2689034981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 msgr2=0x7f120010b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 --2- 172.21.15.87:0/2689034981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0098a0 tx=0x7f11fc009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:55.524 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- 172.21.15.87:0/2689034981 shutdown_connections 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 --2- 172.21.15.87:0/2689034981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120010b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- 172.21.15.87:0/2689034981 >> 172.21.15.87:0/2689034981 conn(0x7f1200102910 msgr2=0x7f1200104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- 172.21.15.87:0/2689034981 shutdown_connections 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- 172.21.15.87:0/2689034981 wait complete. 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 Processor -- start 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.802+0000 7f120753c700 1 -- start start 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f120753c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f120753c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f120010c770 con 0x7f1200108ee0 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f12052d8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f12052d8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33604/0 (socket says 172.21.15.87:33604) 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f12052d8700 1 -- 172.21.15.87:0/1933941127 learned_addr learned my addr 172.21.15.87:0/1933941127 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f12052d8700 1 -- 172.21.15.87:0/1933941127 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120019c940 con 0x7f1200108ee0 2024-04-22T13:21:55.525 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f12052d8700 1 --2- 172.21.15.87:0/1933941127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009fb0 tx=0x7f11fc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11fc0036a0 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f120753c700 1 -- 172.21.15.87:0/1933941127 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120019cbd0 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f11fc01a430 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f120753c700 1 -- 172.21.15.87:0/1933941127 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120019f8c0 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.803+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11fc019550 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.804+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f11fc019be0 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.804+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f11fc024900 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.805+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12000640a0 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.807+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f11fc043e50 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.878+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1200063650 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11f67fc700 1 -- 172.21.15.87:0/1933941127 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f11fc032020 con 0x7f1200108ee0 2024-04-22T13:21:55.526 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 msgr2=0x7f120019c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11ebfff700 1 --2- 172.21.15.87:0/1933941127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009fb0 tx=0x7f11fc011940 comp rx=0 tx=0).stop 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 shutdown_connections 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11ebfff700 1 --2- 172.21.15.87:0/1933941127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1200108ee0 0x7f120019c400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.879+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 >> 172.21.15.87:0/1933941127 conn(0x7f1200102910 msgr2=0x7f12001034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.880+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 shutdown_connections 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:54.880+0000 7f11ebfff700 1 -- 172.21.15.87:0/1933941127 wait complete. 2024-04-22T13:21:55.527 INFO:teuthology.orchestra.run.smithi087.stdout:mgr not available, waiting (2/15)... 2024-04-22T13:21:55.607 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:55.309+0000 7f660652e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:21:55.607 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:55.405+0000 7f660652e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:21:55.607 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:55.610+0000 7f660652e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:21:56.476 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.149+0000 7f660652e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:21:56.476 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.240+0000 7f660652e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:21:56.738 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.478+0000 7f660652e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:21:56.738 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.610+0000 7f660652e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:21:56.738 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.705+0000 7f660652e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:21:57.117 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:56.820+0000 7f660652e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:21:57.117 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:57.120+0000 7f660652e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:21:57.933 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:57.727+0000 7f660652e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:21:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:21:57.948+0000 7f660652e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: Activating manager daemon x 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: mgrmap e2: x(active, starting, since 0.00346141s) 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:21:58.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: Manager daemon x is now available 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' 2024-04-22T13:21:58.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:58 smithi087 ceph-mon[27999]: from='mgr.14100 172.21.15.87:0/2701723157' entity='mgr.x' 2024-04-22T13:21:58.575 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:21:58.575 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:21:58.575 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:21:58.575 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T13:21:58.575 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 0 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:58.576 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T13:21:58.577 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T13:21:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T13:21:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T13:21:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T13:21:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T13:21:58.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T13:21:43.838839+0000", 2024-04-22T13:21:58.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f38228f8700 1 Processor -- start 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f38228f8700 1 -- start start 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f38228f8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f38228f8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f381c107ce0 con 0x7f381c1072f0 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f381bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f381bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33610/0 (socket says 172.21.15.87:33610) 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.952+0000 7f381bfff700 1 -- 172.21.15.87:0/1776480708 learned_addr learned my addr 172.21.15.87:0/1776480708 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.953+0000 7f381bfff700 1 -- 172.21.15.87:0/1776480708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1084f0 con 0x7f381c1072f0 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.953+0000 7f381bfff700 1 --2- 172.21.15.87:0/1776480708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f380c0098a0 tx=0x7f380c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5090ec9672f6c03e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:58.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.953+0000 7f381affd700 1 -- 172.21.15.87:0/1776480708 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f380c004160 con 0x7f381c1072f0 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.953+0000 7f381affd700 1 -- 172.21.15.87:0/1776480708 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f380c004300 con 0x7f381c1072f0 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.953+0000 7f381affd700 1 -- 172.21.15.87:0/1776480708 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f380c011af0 con 0x7f381c1072f0 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 -- 172.21.15.87:0/1776480708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 msgr2=0x7f381c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 --2- 172.21.15.87:0/1776480708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f380c0098a0 tx=0x7f380c009bb0 comp rx=0 tx=0).stop 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 -- 172.21.15.87:0/1776480708 shutdown_connections 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 --2- 172.21.15.87:0/1776480708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c107710 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 -- 172.21.15.87:0/1776480708 >> 172.21.15.87:0/1776480708 conn(0x7f381c102910 msgr2=0x7f381c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 -- 172.21.15.87:0/1776480708 shutdown_connections 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.954+0000 7f38228f8700 1 -- 172.21.15.87:0/1776480708 wait complete. 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.955+0000 7f38228f8700 1 Processor -- start 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.955+0000 7f38228f8700 1 -- start start 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.955+0000 7f38228f8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.955+0000 7f38228f8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10b8a0 con 0x7f381c1072f0 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f381bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:21:58.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f381bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33638/0 (socket says 172.21.15.87:33638) 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f381bfff700 1 -- 172.21.15.87:0/2594093277 learned_addr learned my addr 172.21.15.87:0/2594093277 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f381bfff700 1 -- 172.21.15.87:0/2594093277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1929a0 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f381bfff700 1 --2- 172.21.15.87:0/2594093277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f380c009fb0 tx=0x7f380c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f380c0036a0 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f380c01a430 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c190b90 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f380c019550 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.956+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c1910d0 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.957+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7f380c019850 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.957+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c0640a0 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.957+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f380c003840 con 0x7f381c1072f0 2024-04-22T13:21:58.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:57.960+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f380c023c90 con 0x7f381c1072f0 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.035+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f381c1918c0 con 0x7f381c1072f0 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.036+0000 7f38197fa700 1 -- 172.21.15.87:0/2594093277 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f380c00fda0 con 0x7f381c1072f0 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.036+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 msgr2=0x7f381c192460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.036+0000 7f38228f8700 1 --2- 172.21.15.87:0/2594093277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f380c009fb0 tx=0x7f380c011940 comp rx=0 tx=0).stop 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.037+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 shutdown_connections 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.037+0000 7f38228f8700 1 --2- 172.21.15.87:0/2594093277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f381c1072f0 0x7f381c192460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.037+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 >> 172.21.15.87:0/2594093277 conn(0x7f381c102910 msgr2=0x7f381c1034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.037+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 shutdown_connections 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:21:58.037+0000 7f38228f8700 1 -- 172.21.15.87:0/2594093277 wait complete. 2024-04-22T13:21:58.591 INFO:teuthology.orchestra.run.smithi087.stdout:mgr not available, waiting (3/15)... 2024-04-22T13:21:59.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2594093277' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T13:21:59.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:21:59 smithi087 ceph-mon[27999]: mgrmap e3: x(active, since 1.00699s) 2024-04-22T13:22:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:01 smithi087 ceph-mon[27999]: mgrmap e4: x(active, since 2s) 2024-04-22T13:22:01.905 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:22:01.905 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:22:01.905 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 0 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-22T13:22:01.906 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T13:22:01.907 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T13:22:01.915 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T13:22:01.915 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T13:22:01.915 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T13:22:01.916 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ], 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T13:22:01.917 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T13:21:43.838839+0000", 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout }, 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.003+0000 7fededf41700 1 Processor -- start 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.003+0000 7fededf41700 1 -- start start 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.003+0000 7fededf41700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.003+0000 7fededf41700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fede81070e0 con 0x7fede81066f0 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.003+0000 7fede77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53364/0 (socket says 172.21.15.87:53364) 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede77fe700 1 -- 172.21.15.87:0/1269901109 learned_addr learned my addr 172.21.15.87:0/1269901109 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede77fe700 1 -- 172.21.15.87:0/1269901109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede81096d0 con 0x7fede81066f0 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede77fe700 1 --2- 172.21.15.87:0/1269901109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543ce26b8177d294 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:01.918 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede67fc700 1 -- 172.21.15.87:0/1269901109 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedd8004160 con 0x7fede81066f0 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede67fc700 1 -- 172.21.15.87:0/1269901109 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fedd8004300 con 0x7fede81066f0 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.004+0000 7fede67fc700 1 -- 172.21.15.87:0/1269901109 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedd8011af0 con 0x7fede81066f0 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 -- 172.21.15.87:0/1269901109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 msgr2=0x7fede8106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 --2- 172.21.15.87:0/1269901109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd8009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 -- 172.21.15.87:0/1269901109 shutdown_connections 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 --2- 172.21.15.87:0/1269901109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8106b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 -- 172.21.15.87:0/1269901109 >> 172.21.15.87:0/1269901109 conn(0x7fede8102110 msgr2=0x7fede8104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 -- 172.21.15.87:0/1269901109 shutdown_connections 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 -- 172.21.15.87:0/1269901109 wait complete. 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.005+0000 7fededf41700 1 Processor -- start 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fededf41700 1 -- start start 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fededf41700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fededf41700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fede810ca80 con 0x7fede81066f0 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fede77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:01.919 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fede77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53378/0 (socket says 172.21.15.87:53378) 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fede77fe700 1 -- 172.21.15.87:0/3911270196 learned_addr learned my addr 172.21.15.87:0/3911270196 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fede77fe700 1 -- 172.21.15.87:0/3911270196 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede8109490 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.006+0000 7fede77fe700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd8003f40 tx=0x7fedd8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedd80038b0 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fedd8003a50 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedd801a5c0 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede8107680 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede8107bc0 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.008+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fedd801a870 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.009+0000 7fede4ff9700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 0x7fedd003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.009+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fedd804daa0 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.009+0000 7fede6ffd700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 0x7fedd003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.009+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede80640a0 con 0x7fede81066f0 2024-04-22T13:22:01.920 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.009+0000 7fede6ffd700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 0x7fedd003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.014+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedd8019c70 con 0x7fede81066f0 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.292+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fede8103730 con 0x7fede81066f0 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.292+0000 7fede4ff9700 1 -- 172.21.15.87:0/3911270196 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fedd804d560 con 0x7fede81066f0 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 msgr2=0x7fedd003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 0x7fedd003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 msgr2=0x7fede8108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd8003f40 tx=0x7fedd8011890 comp rx=0 tx=0).stop 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 shutdown_connections 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7fedd0039560 0x7fedd003ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 --2- 172.21.15.87:0/3911270196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fede81066f0 0x7fede8108f50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 >> 172.21.15.87:0/3911270196 conn(0x7fede8102110 msgr2=0x7fede8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.295+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 shutdown_connections 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:01.296+0000 7fededf41700 1 -- 172.21.15.87:0/3911270196 wait complete. 2024-04-22T13:22:01.921 INFO:teuthology.orchestra.run.smithi087.stdout:mgr is available 2024-04-22T13:22:02.296 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3911270196' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T13:22:03.115 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:03 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2833957706' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout fsid = 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T13:22:03.129 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T13:22:03.130 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.330+0000 7f2062db6700 1 Processor -- start 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.330+0000 7f2062db6700 1 -- start start 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.330+0000 7f2062db6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:03.131 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.330+0000 7f2062db6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f205c1092a0 con 0x7f205c1088b0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f2060b52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f2060b52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53382/0 (socket says 172.21.15.87:53382) 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f2060b52700 1 -- 172.21.15.87:0/1515896493 learned_addr learned my addr 172.21.15.87:0/1515896493 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f2060b52700 1 -- 172.21.15.87:0/1515896493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c107510 con 0x7f205c1088b0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f2060b52700 1 --2- 172.21.15.87:0/1515896493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f204c009e80 tx=0x7f204c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c5212e4af9df1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f205b7fe700 1 -- 172.21.15.87:0/1515896493 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f204c017070 con 0x7f205c1088b0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f205b7fe700 1 -- 172.21.15.87:0/1515896493 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f204c00b8c0 con 0x7f205c1088b0 2024-04-22T13:22:03.132 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.331+0000 7f205b7fe700 1 -- 172.21.15.87:0/1515896493 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f204c004490 con 0x7f205c1088b0 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- 172.21.15.87:0/1515896493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 msgr2=0x7f205c108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 --2- 172.21.15.87:0/1515896493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f204c009e80 tx=0x7f204c00b330 comp rx=0 tx=0).stop 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- 172.21.15.87:0/1515896493 shutdown_connections 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 --2- 172.21.15.87:0/1515896493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c108cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- 172.21.15.87:0/1515896493 >> 172.21.15.87:0/1515896493 conn(0x7f205c102080 msgr2=0x7f205c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- 172.21.15.87:0/1515896493 shutdown_connections 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- 172.21.15.87:0/1515896493 wait complete. 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 Processor -- start 2024-04-22T13:22:03.133 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.332+0000 7f2062db6700 1 -- start start 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2062db6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2062db6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f205c10b8d0 con 0x7f205c1088b0 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2060b52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2060b52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53392/0 (socket says 172.21.15.87:53392) 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2060b52700 1 -- 172.21.15.87:0/2833957706 learned_addr learned my addr 172.21.15.87:0/2833957706 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2060b52700 1 -- 172.21.15.87:0/2833957706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c192a60 con 0x7f205c1088b0 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2060b52700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f204c00b7b0 tx=0x7f204c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:03.134 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f204c020020 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f204c012430 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205c190c50 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f204c017040 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.333+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205c191190 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.334+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f204c023020 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.334+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c0640a0 con 0x7f205c1088b0 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.335+0000 7f2059ffb700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 0x7f204803b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:03.135 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.335+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f204c00f050 con 0x7f205c1088b0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.335+0000 7f205bfff700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 0x7f204803b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.335+0000 7f205bfff700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 0x7f204803b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2044006fd0 tx=0x7f2044006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.340+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f204c02a020 con 0x7f205c1088b0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.534+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f205c1962e0 con 0x7f205c1088b0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.535+0000 7f2059ffb700 1 -- 172.21.15.87:0/2833957706 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f204c025dc0 con 0x7f205c1088b0 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 msgr2=0x7f204803b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 0x7f204803b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2044006fd0 tx=0x7f2044006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:03.136 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 msgr2=0x7f205c192520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f204c00b7b0 tx=0x7f204c00be20 comp rx=0 tx=0).stop 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 shutdown_connections 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f2048039510 0x7f204803b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 --2- 172.21.15.87:0/2833957706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c1088b0 0x7f205c192520 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 >> 172.21.15.87:0/2833957706 conn(0x7f205c102080 msgr2=0x7f205c102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 shutdown_connections 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:02.537+0000 7f2062db6700 1 -- 172.21.15.87:0/2833957706 wait complete. 2024-04-22T13:22:03.137 INFO:teuthology.orchestra.run.smithi087.stdout:Enabling cephadm module... 2024-04-22T13:22:04.173 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/834508103' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T13:22:04.173 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setuser ceph since I am not root 2024-04-22T13:22:04.173 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setgroup ceph since I am not root 2024-04-22T13:22:04.173 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:04.176+0000 7f6b1dba5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:22:04.461 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:04.362+0000 7f6b1dba5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:22:04.461 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:04.463+0000 7f6b1dba5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:22:04.546 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f619111a700 1 Processor -- start 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f619111a700 1 -- start start 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f619111a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f619111a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f618c07be80 con 0x7f618c108f00 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53400/0 (socket says 172.21.15.87:53400) 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f618ad9d700 1 -- 172.21.15.87:0/3239178595 learned_addr learned my addr 172.21.15.87:0/3239178595 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.567+0000 7f618ad9d700 1 -- 172.21.15.87:0/3239178595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c10b830 con 0x7f618c108f00 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f618ad9d700 1 --2- 172.21.15.87:0/3239178595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f617c0098a0 tx=0x7f617c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d648b7932703fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:04.547 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f6189d9b700 1 -- 172.21.15.87:0/3239178595 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f617c004160 con 0x7f618c108f00 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f6189d9b700 1 -- 172.21.15.87:0/3239178595 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f617c004300 con 0x7f618c108f00 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f6189d9b700 1 -- 172.21.15.87:0/3239178595 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f617c011af0 con 0x7f618c108f00 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 -- 172.21.15.87:0/3239178595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 msgr2=0x7f618c10b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 --2- 172.21.15.87:0/3239178595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f617c0098a0 tx=0x7f617c009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 -- 172.21.15.87:0/3239178595 shutdown_connections 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 --2- 172.21.15.87:0/3239178595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c10b2f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 -- 172.21.15.87:0/3239178595 >> 172.21.15.87:0/3239178595 conn(0x7f618c102950 msgr2=0x7f618c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 -- 172.21.15.87:0/3239178595 shutdown_connections 2024-04-22T13:22:04.548 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.568+0000 7f619111a700 1 -- 172.21.15.87:0/3239178595 wait complete. 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.569+0000 7f619111a700 1 Processor -- start 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.569+0000 7f619111a700 1 -- start start 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.569+0000 7f619111a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.569+0000 7f619111a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c7f0 con 0x7f618c108f00 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53412/0 (socket says 172.21.15.87:53412) 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f618ad9d700 1 -- 172.21.15.87:0/834508103 learned_addr learned my addr 172.21.15.87:0/834508103 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f618ad9d700 1 -- 172.21.15.87:0/834508103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c19c900 con 0x7f618c108f00 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f618ad9d700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f617c011ac0 tx=0x7f617c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:04.549 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f617c003970 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f617c003b10 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c19cb90 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f617c024610 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.570+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c19f880 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.571+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f617c024890 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.571+0000 7f6173fff700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 0x7f617403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.571+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f617c04d8e0 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.571+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c0640a0 con 0x7f618c108f00 2024-04-22T13:22:04.550 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.571+0000 7f618a59c700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 0x7f617403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.572+0000 7f618a59c700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 0x7f617403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6180006fd0 tx=0x7f6180006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.576+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f617c0172b0 con 0x7f618c108f00 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:03.828+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f618c19fe50 con 0x7f618c108f00 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.011+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f617c04d300 con 0x7f618c108f00 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.013+0000 7f6173fff700 1 -- 172.21.15.87:0/834508103 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f617c02d910 con 0x7f618c108f00 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 msgr2=0x7f617403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:04.551 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 0x7f617403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6180006fd0 tx=0x7f6180006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 msgr2=0x7f618c19c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f617c011ac0 tx=0x7f617c011ba0 comp rx=0 tx=0).stop 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 shutdown_connections 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f6174039560 0x7f617403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 --2- 172.21.15.87:0/834508103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f618c108f00 0x7f618c19c3c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 >> 172.21.15.87:0/834508103 conn(0x7f618c102950 msgr2=0x7f618c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 shutdown_connections 2024-04-22T13:22:04.552 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.014+0000 7f619111a700 1 -- 172.21.15.87:0/834508103 wait complete. 2024-04-22T13:22:05.196 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/834508103' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T13:22:05.196 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:05 smithi087 ceph-mon[27999]: mgrmap e5: x(active, since 6s) 2024-04-22T13:22:05.196 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:04.776+0000 7f6b1dba5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:22:05.196 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:04.864+0000 7f6b1dba5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:22:05.887 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f409e448700 1 Processor -- start 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f409e448700 1 -- start start 2024-04-22T13:22:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f409e448700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f409e448700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f409807be60 con 0x7f4098108f20 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f4097fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f4097fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53438/0 (socket says 172.21.15.87:53438) 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f4097fff700 1 -- 172.21.15.87:0/3399811965 learned_addr learned my addr 172.21.15.87:0/3399811965 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.993+0000 7f4097fff700 1 -- 172.21.15.87:0/3399811965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409810b850 con 0x7f4098108f20 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.994+0000 7f4097fff700 1 --2- 172.21.15.87:0/3399811965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40880098a0 tx=0x7f4088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fafb773f3b90b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.994+0000 7f4096ffd700 1 -- 172.21.15.87:0/3399811965 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4088004160 con 0x7f4098108f20 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.994+0000 7f4096ffd700 1 -- 172.21.15.87:0/3399811965 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4088004300 con 0x7f4098108f20 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.994+0000 7f4096ffd700 1 -- 172.21.15.87:0/3399811965 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4088011af0 con 0x7f4098108f20 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 -- 172.21.15.87:0/3399811965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 msgr2=0x7f409810b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 --2- 172.21.15.87:0/3399811965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40880098a0 tx=0x7f4088009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 -- 172.21.15.87:0/3399811965 shutdown_connections 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 --2- 172.21.15.87:0/3399811965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409810b310 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 -- 172.21.15.87:0/3399811965 >> 172.21.15.87:0/3399811965 conn(0x7f4098102950 msgr2=0x7f4098104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 -- 172.21.15.87:0/3399811965 shutdown_connections 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.995+0000 7f409e448700 1 -- 172.21.15.87:0/3399811965 wait complete. 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f409e448700 1 Processor -- start 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f409e448700 1 -- start start 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f409e448700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f409e448700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4098107ab0 con 0x7f4098108f20 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f4097fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:05.890 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f4097fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53450/0 (socket says 172.21.15.87:53450) 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f4097fff700 1 -- 172.21.15.87:0/416467404 learned_addr learned my addr 172.21.15.87:0/416467404 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.996+0000 7f4097fff700 1 -- 172.21.15.87:0/416467404 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409819a720 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f4097fff700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4088009fb0 tx=0x7f4088011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40880036a0 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f408801a430 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409819a9b0 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4088019550 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.997+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409819d6a0 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4088019850 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40980640a0 con 0x7f4098108f20 2024-04-22T13:22:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f40957fa700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 0x7f408003fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f408804d910 con 0x7f4098108f20 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f40977fe700 1 -- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 msgr2=0x7f408003fe30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:04.998+0000 7f40977fe700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 0x7f408003fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.003+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f408804d330 con 0x7f4098108f20 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.199+0000 7f40977fe700 1 -- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 msgr2=0x7f408003fe30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.199+0000 7f40977fe700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 0x7f408003fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.300+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f409802e240 con 0x7f4098108f20 2024-04-22T13:22:05.892 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.301+0000 7f40957fa700 1 -- 172.21.15.87:0/416467404 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f408800fda0 con 0x7f4098108f20 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.304+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 msgr2=0x7f408003fe30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.304+0000 7f409e448700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 0x7f408003fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.304+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 msgr2=0x7f409819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.304+0000 7f409e448700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4088009fb0 tx=0x7f4088011940 comp rx=0 tx=0).stop 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.305+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 shutdown_connections 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.305+0000 7f409e448700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f408003d970 0x7f408003fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.305+0000 7f409e448700 1 --2- 172.21.15.87:0/416467404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4098108f20 0x7f409819a1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.305+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 >> 172.21.15.87:0/416467404 conn(0x7f4098102950 msgr2=0x7f40981035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.305+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 shutdown_connections 2024-04-22T13:22:05.893 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:05.306+0000 7f409e448700 1 -- 172.21.15.87:0/416467404 wait complete. 2024-04-22T13:22:05.894 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for the mgr to restart... 2024-04-22T13:22:05.894 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mgr epoch 5... 2024-04-22T13:22:06.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/416467404' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T13:22:06.347 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.072+0000 7f6b1dba5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:22:06.347 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.245+0000 7f6b1dba5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:22:06.703 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.419+0000 7f6b1dba5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:22:06.703 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.705+0000 7f6b1dba5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:22:06.981 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.796+0000 7f6b1dba5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:22:07.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:06.995+0000 7f6b1dba5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:22:07.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:07.185+0000 7f6b1dba5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:22:08.139 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:07.775+0000 7f6b1dba5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:22:08.410 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:08.196+0000 7f6b1dba5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:22:08.410 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:08.296+0000 7f6b1dba5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:22:08.737 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:08.507+0000 7f6b1dba5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:22:09.364 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.043+0000 7f6b1dba5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:22:09.365 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.138+0000 7f6b1dba5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:22:09.728 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.388+0000 7f6b1dba5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:22:09.728 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.520+0000 7f6b1dba5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:22:09.728 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.615+0000 7f6b1dba5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:22:09.728 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:09.729+0000 7f6b1dba5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:22:10.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:10.025+0000 7f6b1dba5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:22:10.882 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:10.616+0000 7f6b1dba5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:22:10.882 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:10.827+0000 7f6b1dba5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: Active manager daemon x restarted 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: Activating manager daemon x 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: mgrmap e6: x(active, starting, since 0.00342486s) 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:22:11.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: Manager daemon x is now available 2024-04-22T13:22:11.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:11.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:11.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:11.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:11.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:10 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:22:12.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: Found migration_current of "None". Setting to last migration. 2024-04-22T13:22:12.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:22:12.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:12.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:12.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:12.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:11 smithi087 ceph-mon[27999]: mgrmap e7: x(active, since 1.00569s) 2024-04-22T13:22:12.477 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d353a700 1 Processor -- start 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d353a700 1 -- start start 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d353a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d353a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc07be80 con 0x7f81cc108ea0 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d12d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d12d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53460/0 (socket says 172.21.15.87:53460) 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.377+0000 7f81d12d6700 1 -- 172.21.15.87:0/217281366 learned_addr learned my addr 172.21.15.87:0/217281366 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d12d6700 1 -- 172.21.15.87:0/217281366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc10b7d0 con 0x7f81cc108ea0 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d12d6700 1 --2- 172.21.15.87:0/217281366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098a0 tx=0x7f81c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7132c29eb24519ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:12.478 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81c3fff700 1 -- 172.21.15.87:0/217281366 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c8004160 con 0x7f81cc108ea0 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81c3fff700 1 -- 172.21.15.87:0/217281366 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81c8004300 con 0x7f81cc108ea0 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81c3fff700 1 -- 172.21.15.87:0/217281366 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c8011af0 con 0x7f81cc108ea0 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d353a700 1 -- 172.21.15.87:0/217281366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 msgr2=0x7f81cc10b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d353a700 1 --2- 172.21.15.87:0/217281366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098a0 tx=0x7f81c8009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d353a700 1 -- 172.21.15.87:0/217281366 shutdown_connections 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.378+0000 7f81d353a700 1 --2- 172.21.15.87:0/217281366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc10b290 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 -- 172.21.15.87:0/217281366 >> 172.21.15.87:0/217281366 conn(0x7f81cc102910 msgr2=0x7f81cc104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 -- 172.21.15.87:0/217281366 shutdown_connections 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 -- 172.21.15.87:0/217281366 wait complete. 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 Processor -- start 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 -- start start 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d353a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc10c790 con 0x7f81cc108ea0 2024-04-22T13:22:12.479 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d12d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d12d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53466/0 (socket says 172.21.15.87:53466) 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.379+0000 7f81d12d6700 1 -- 172.21.15.87:0/2232154545 learned_addr learned my addr 172.21.15.87:0/2232154545 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81d12d6700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc108c60 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81d12d6700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81c8011ac0 tx=0x7f81c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c8003970 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81c8003b10 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc106e50 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c8024610 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.380+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc107390 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f81c8024840 con 0x7f81cc108ea0 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81c27fc700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f81b803c1d0 con 0x7f81b8039600 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81d0ad5700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:12.480 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f81c804d8a0 con 0x7f81cc108ea0 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.381+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.582+0000 7f81d0ad5700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.582+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.983+0000 7f81d0ad5700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:06.983+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:07.784+0000 7f81d0ad5700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:07.784+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:09.385+0000 7f81d0ad5700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/4093217616 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:09.385+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:10.831+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f81c8024b30 con 0x7f81cc108ea0 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:10.831+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 msgr2=0x7f81b803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:10.831+0000 7f81c27fc700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.833+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f81c804f4f0 con 0x7f81cc108ea0 2024-04-22T13:22:12.481 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.834+0000 7f81c27fc700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 0x7f81b803f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.834+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f81b803c1d0 con 0x7f81b803cef0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.834+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 0x7f81b803f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.835+0000 7f81d0ad5700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 0x7f81b803f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f81bc003a10 tx=0x7f81bc0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.835+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f81b803c1d0 con 0x7f81b803cef0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f81cc0640a0 con 0x7f81b803cef0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81c27fc700 1 -- 172.21.15.87:0/2232154545 <== mgr.14120 v2:172.21.15.87:6800/35245616 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f81cc0640a0 con 0x7f81b803cef0 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 msgr2=0x7f81b803f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:12.482 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 0x7f81b803f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f81bc003a10 tx=0x7f81bc0092b0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 msgr2=0x7f81cc108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81c8011ac0 tx=0x7f81c8011ba0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 shutdown_connections 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f81b803cef0 0x7f81b803f2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81cc108ea0 0x7f81cc108720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 --2- 172.21.15.87:0/2232154545 >> [v2:172.21.15.87:6800/4093217616,v1:172.21.15.87:6801/4093217616] conn(0x7f81b8039600 0x7f81b803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 >> 172.21.15.87:0/2232154545 conn(0x7f81cc102910 msgr2=0x7f81cc1034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 shutdown_connections 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:11.842+0000 7f81d353a700 1 -- 172.21.15.87:0/2232154545 wait complete. 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:mgr epoch 5 is available 2024-04-22T13:22:12.483 INFO:teuthology.orchestra.run.smithi087.stdout:Setting orchestrator backend to cephadm... 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:11] ENGINE Bus STARTING 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:11] ENGINE Serving on https://172.21.15.87:7150 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:11] ENGINE Serving on http://172.21.15.87:8765 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:11] ENGINE Bus STARTED 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T13:22:13.107 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:12 smithi087 ceph-mon[27999]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T13:22:13.753 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.883+0000 7f7513ad8700 1 Processor -- start 2024-04-22T13:22:13.753 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.883+0000 7f7513ad8700 1 -- start start 2024-04-22T13:22:13.753 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.883+0000 7f7513ad8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:13.753 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.883+0000 7f7513ad8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f750c07be40 con 0x7f750c106d30 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7511874700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7511874700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54412/0 (socket says 172.21.15.87:54412) 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7511874700 1 -- 172.21.15.87:0/3267354920 learned_addr learned my addr 172.21.15.87:0/3267354920 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7511874700 1 -- 172.21.15.87:0/3267354920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c107690 con 0x7f750c106d30 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7511874700 1 --2- 172.21.15.87:0/3267354920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7508009e80 tx=0x7f750800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0046a01238f9a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7510872700 1 -- 172.21.15.87:0/3267354920 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7508017070 con 0x7f750c106d30 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7510872700 1 -- 172.21.15.87:0/3267354920 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f750800b8c0 con 0x7f750c106d30 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.884+0000 7f7510872700 1 -- 172.21.15.87:0/3267354920 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7508004490 con 0x7f750c106d30 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 -- 172.21.15.87:0/3267354920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 msgr2=0x7f750c107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:13.754 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3267354920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7508009e80 tx=0x7f750800b330 comp rx=0 tx=0).stop 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 -- 172.21.15.87:0/3267354920 shutdown_connections 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3267354920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c107150 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 -- 172.21.15.87:0/3267354920 >> 172.21.15.87:0/3267354920 conn(0x7f750c102950 msgr2=0x7f750c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 -- 172.21.15.87:0/3267354920 shutdown_connections 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.885+0000 7f7513ad8700 1 -- 172.21.15.87:0/3267354920 wait complete. 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7513ad8700 1 Processor -- start 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7513ad8700 1 -- start start 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7513ad8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7513ad8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f750c1086c0 con 0x7f750c106d30 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7511874700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7511874700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54418/0 (socket says 172.21.15.87:54418) 2024-04-22T13:22:13.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7511874700 1 -- 172.21.15.87:0/3785776945 learned_addr learned my addr 172.21.15.87:0/3785776945 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7511874700 1 -- 172.21.15.87:0/3785776945 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c19ead0 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.886+0000 7f7511874700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f750800b7b0 tx=0x7f750800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7508020020 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7508003d10 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7508017040 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750c19ed60 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750c1a1a50 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.890+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f7508023020 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.891+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f750c0640a0 con 0x7f750c106d30 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.891+0000 7f7502ffd700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 0x7f74f803b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:13.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.891+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f750800f050 con 0x7f750c106d30 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.891+0000 7f7511073700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 0x7f74f803b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.891+0000 7f7511073700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 0x7f74f803b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74fc006fd0 tx=0x7f74fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.896+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f750802a020 con 0x7f750c106d30 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:12.897+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f750802daa0 con 0x7f750c106d30 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.110+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f750c0008d0 con 0x7f74f80394e0 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.115+0000 7f7502ffd700 1 -- 172.21.15.87:0/3785776945 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f750c0008d0 con 0x7f74f80394e0 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 msgr2=0x7f74f803b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 0x7f74f803b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74fc006fd0 tx=0x7f74fc006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 msgr2=0x7f750c19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f750800b7b0 tx=0x7f750800be90 comp rx=0 tx=0).stop 2024-04-22T13:22:13.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 shutdown_connections 2024-04-22T13:22:13.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f74f80394e0 0x7f74f803b9a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:13.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 --2- 172.21.15.87:0/3785776945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f750c106d30 0x7f750c19e590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:13.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 >> 172.21.15.87:0/3785776945 conn(0x7f750c102950 msgr2=0x7f750c1951f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:13.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 shutdown_connections 2024-04-22T13:22:13.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:13.117+0000 7f7513ad8700 1 -- 172.21.15.87:0/3785776945 wait complete. 2024-04-22T13:22:14.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:13 smithi087 ceph-mon[27999]: mgrmap e8: x(active, since 2s) 2024-04-22T13:22:14.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:13 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:14.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:13 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:14.853 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c2237d700 1 Processor -- start 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c2237d700 1 -- start start 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c2237d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c2237d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c107d40 con 0x7f2c1c107350 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54430/0 (socket says 172.21.15.87:54430) 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.137+0000 7f2c1bfff700 1 -- 172.21.15.87:0/1245588223 learned_addr learned my addr 172.21.15.87:0/1245588223 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c1bfff700 1 -- 172.21.15.87:0/1245588223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c108550 con 0x7f2c1c107350 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c1bfff700 1 --2- 172.21.15.87:0/1245588223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0098a0 tx=0x7f2c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a71c79bbe4765b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c1affd700 1 -- 172.21.15.87:0/1245588223 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c004160 con 0x7f2c1c107350 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c1affd700 1 -- 172.21.15.87:0/1245588223 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2c0c004300 con 0x7f2c1c107350 2024-04-22T13:22:14.854 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c1affd700 1 -- 172.21.15.87:0/1245588223 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c011af0 con 0x7f2c1c107350 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 -- 172.21.15.87:0/1245588223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 msgr2=0x7f2c1c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 --2- 172.21.15.87:0/1245588223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0098a0 tx=0x7f2c0c009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 -- 172.21.15.87:0/1245588223 shutdown_connections 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 --2- 172.21.15.87:0/1245588223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c107770 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 -- 172.21.15.87:0/1245588223 >> 172.21.15.87:0/1245588223 conn(0x7f2c1c102950 msgr2=0x7f2c1c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.138+0000 7f2c2237d700 1 -- 172.21.15.87:0/1245588223 shutdown_connections 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c2237d700 1 -- 172.21.15.87:0/1245588223 wait complete. 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c2237d700 1 Processor -- start 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c2237d700 1 -- start start 2024-04-22T13:22:14.855 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c2237d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c2237d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c10b900 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.139+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54440/0 (socket says 172.21.15.87:54440) 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c1bfff700 1 -- 172.21.15.87:0/2280845949 learned_addr learned my addr 172.21.15.87:0/2280845949 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c1bfff700 1 -- 172.21.15.87:0/2280845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c19c9c0 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c1bfff700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009fb0 tx=0x7f2c0c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c0036a0 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2c0c01a430 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1c19cc50 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c0c019550 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.140+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1c19f8f0 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.141+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2c0c003840 con 0x7f2c1c107350 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.141+0000 7f2c197fa700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 0x7f2c0403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:14.856 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.141+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c1c0640a0 con 0x7f2c1c107350 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.141+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2c0c04d8a0 con 0x7f2c1c107350 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.141+0000 7f2c1b7fe700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 0x7f2c0403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.142+0000 7f2c1b7fe700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 0x7f2c0403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2c10006fd0 tx=0x7f2c10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.146+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c0c003b30 con 0x7f2c1c107350 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.345+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2c1c02e2e0 con 0x7f2c040395b0 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.346+0000 7f2c197fa700 1 -- 172.21.15.87:0/2280845949 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2c1c02e2e0 con 0x7f2c040395b0 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 msgr2=0x7f2c0403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 0x7f2c0403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2c10006fd0 tx=0x7f2c10006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 msgr2=0x7f2c1c19c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009fb0 tx=0x7f2c0c011940 comp rx=0 tx=0).stop 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 shutdown_connections 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f2c040395b0 0x7f2c0403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:14.857 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 --2- 172.21.15.87:0/2280845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c1c107350 0x7f2c1c19c480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:14.858 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 >> 172.21.15.87:0/2280845949 conn(0x7f2c1c102950 msgr2=0x7f2c1c103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:14.858 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 shutdown_connections 2024-04-22T13:22:14.858 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:14.349+0000 7f2c2237d700 1 -- 172.21.15.87:0/2280845949 wait complete. 2024-04-22T13:22:14.858 INFO:teuthology.orchestra.run.smithi087.stdout:Generating ssh key... 2024-04-22T13:22:15.190 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:14 smithi087 ceph-mon[27999]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: Generating public/private rsa key pair. 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: Your identification has been saved in /tmp/tmpygk9orlq/key. 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: Your public key has been saved in /tmp/tmpygk9orlq/key.pub. 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: The key fingerprint is: 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: SHA256:TysWYREc/ARadb+t37YkseBoouUhNn9cFG4qsnNC0ko ceph-176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:15.895 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: The key's randomart image is: 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: +---[RSA 3072]----+ 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | o*=. . | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | oo..o . | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | . oo. . . | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | . ..+ o | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | . S =. .. .| 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | E + . =oo. o. | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | . ++o+=o+. o.. | 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | ..+Bo++ o.o| 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: | .+o. o+| 2024-04-22T13:22:15.896 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: +----[SHA256]-----+ 2024-04-22T13:22:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-mon[27999]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:15 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:16.252 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.292+0000 7fdc36c7c700 1 Processor -- start 2024-04-22T13:22:16.252 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.292+0000 7fdc36c7c700 1 -- start start 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.292+0000 7fdc36c7c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.292+0000 7fdc36c7c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3007bea0 con 0x7fdc30108e80 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc34a18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc34a18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54450/0 (socket says 172.21.15.87:54450) 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc34a18700 1 -- 172.21.15.87:0/4065164803 learned_addr learned my addr 172.21.15.87:0/4065164803 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc34a18700 1 -- 172.21.15.87:0/4065164803 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3010b7b0 con 0x7fdc30108e80 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc34a18700 1 --2- 172.21.15.87:0/4065164803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009e80 tx=0x7fdc2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200c51611134cd36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc2f7fe700 1 -- 172.21.15.87:0/4065164803 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28017070 con 0x7fdc30108e80 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc2f7fe700 1 -- 172.21.15.87:0/4065164803 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdc2800b8c0 con 0x7fdc30108e80 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.293+0000 7fdc2f7fe700 1 -- 172.21.15.87:0/4065164803 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28004490 con 0x7fdc30108e80 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- 172.21.15.87:0/4065164803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 msgr2=0x7fdc3010b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/4065164803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009e80 tx=0x7fdc2800b330 comp rx=0 tx=0).stop 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- 172.21.15.87:0/4065164803 shutdown_connections 2024-04-22T13:22:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/4065164803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3010b270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- 172.21.15.87:0/4065164803 >> 172.21.15.87:0/4065164803 conn(0x7fdc30102910 msgr2=0x7fdc30104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- 172.21.15.87:0/4065164803 shutdown_connections 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- 172.21.15.87:0/4065164803 wait complete. 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 Processor -- start 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- start start 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.294+0000 7fdc36c7c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3010c700 con 0x7fdc30108e80 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc34a18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc34a18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54466/0 (socket says 172.21.15.87:54466) 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc34a18700 1 -- 172.21.15.87:0/3033629779 learned_addr learned my addr 172.21.15.87:0/3033629779 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc34a18700 1 -- 172.21.15.87:0/3033629779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3019ea70 con 0x7fdc30108e80 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc34a18700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc28004220 tx=0x7fdc2800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:16.254 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28020020 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdc28012430 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3019ed00 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28017040 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.295+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc301a19f0 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.296+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdc28023020 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.296+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc300640a0 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.296+0000 7fdc2dffb700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 0x7fdc1c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.296+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc2800f050 con 0x7fdc30108e80 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.296+0000 7fdc2ffff700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 0x7fdc1c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.297+0000 7fdc2ffff700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 0x7fdc1c03ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdc20006fd0 tx=0x7fdc20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:16.255 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.301+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc2802a020 con 0x7fdc30108e80 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.499+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fdc30103ad0 con 0x7fdc1c0395b0 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.674+0000 7fdc2dffb700 1 -- 172.21.15.87:0/3033629779 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdc30103ad0 con 0x7fdc1c0395b0 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 msgr2=0x7fdc1c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 0x7fdc1c03ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdc20006fd0 tx=0x7fdc20006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 msgr2=0x7fdc3019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc28004220 tx=0x7fdc2800be90 comp rx=0 tx=0).stop 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 shutdown_connections 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fdc1c0395b0 0x7fdc1c03ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 --2- 172.21.15.87:0/3033629779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc30108e80 0x7fdc3019e530 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 >> 172.21.15.87:0/3033629779 conn(0x7fdc30102910 msgr2=0x7fdc301033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 shutdown_connections 2024-04-22T13:22:16.256 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:15.676+0000 7fdc36c7c700 1 -- 172.21.15.87:0/3033629779 wait complete. 2024-04-22T13:22:17.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:16 smithi087 ceph-mon[27999]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:17.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:16 smithi087 ceph-mon[27999]: Generating ssh key... 2024-04-22T13:22:17.515 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:17.515 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.646+0000 7f37bfc2e700 1 Processor -- start 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bfc2e700 1 -- start start 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bfc2e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bfc2e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37b8107d40 con 0x7f37b8107350 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bd9ca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bd9ca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54478/0 (socket says 172.21.15.87:54478) 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bd9ca700 1 -- 172.21.15.87:0/2033851952 learned_addr learned my addr 172.21.15.87:0/2033851952 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bd9ca700 1 -- 172.21.15.87:0/2033851952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b8108550 con 0x7f37b8107350 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bd9ca700 1 --2- 172.21.15.87:0/2033851952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37b40098a0 tx=0x7f37b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d096c21370ec59f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.647+0000 7f37bc9c8700 1 -- 172.21.15.87:0/2033851952 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37b4004160 con 0x7f37b8107350 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bc9c8700 1 -- 172.21.15.87:0/2033851952 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f37b4004300 con 0x7f37b8107350 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bc9c8700 1 -- 172.21.15.87:0/2033851952 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37b4011af0 con 0x7f37b8107350 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 -- 172.21.15.87:0/2033851952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 msgr2=0x7f37b8107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/2033851952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37b40098a0 tx=0x7f37b4009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:17.516 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 -- 172.21.15.87:0/2033851952 shutdown_connections 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/2033851952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b8107770 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 -- 172.21.15.87:0/2033851952 >> 172.21.15.87:0/2033851952 conn(0x7f37b8102950 msgr2=0x7f37b8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 -- 172.21.15.87:0/2033851952 shutdown_connections 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.648+0000 7f37bfc2e700 1 -- 172.21.15.87:0/2033851952 wait complete. 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bfc2e700 1 Processor -- start 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bfc2e700 1 -- start start 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bfc2e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bfc2e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37b810b900 con 0x7f37b8107350 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bd9ca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bd9ca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54482/0 (socket says 172.21.15.87:54482) 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bd9ca700 1 -- 172.21.15.87:0/3076665 learned_addr learned my addr 172.21.15.87:0/3076665 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bd9ca700 1 -- 172.21.15.87:0/3076665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b819c970 con 0x7f37b8107350 2024-04-22T13:22:17.517 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.649+0000 7f37bd9ca700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37b40117b0 tx=0x7f37b4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37b4003af0 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f37b4020070 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b819cc00 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37b4019630 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b819f8f0 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.650+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f37b40198a0 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.651+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b80640a0 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.651+0000 7f37aeffd700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 0x7f37a403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.651+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f37b404dd40 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.651+0000 7f37bd1c9700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 0x7f37a403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.651+0000 7f37bd1c9700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 0x7f37a403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f37a8006fd0 tx=0x7f37a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.656+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37b402bd00 con 0x7f37b8107350 2024-04-22T13:22:17.518 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.854+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f37b80008d0 con 0x7f37a4039600 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.855+0000 7f37aeffd700 1 -- 172.21.15.87:0/3076665 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f37b80008d0 con 0x7f37a4039600 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 msgr2=0x7f37a403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 0x7f37a403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f37a8006fd0 tx=0x7f37a8006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 msgr2=0x7f37b819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37b40117b0 tx=0x7f37b4011b70 comp rx=0 tx=0).stop 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 shutdown_connections 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f37a4039600 0x7f37a403bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 --2- 172.21.15.87:0/3076665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8107350 0x7f37b819c430 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.857+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 >> 172.21.15.87:0/3076665 conn(0x7f37b8102950 msgr2=0x7f37b8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.858+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 shutdown_connections 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:16.858+0000 7f37bfc2e700 1 -- 172.21.15.87:0/3076665 wait complete. 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T13:22:17.519 INFO:teuthology.orchestra.run.smithi087.stdout:Adding host smithi087... 2024-04-22T13:22:18.142 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:17 smithi087 ceph-mon[27999]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:19.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:19 smithi087 ceph-mon[27999]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:19.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:19 smithi087 ceph-mon[27999]: Deploying cephadm binary to smithi087 2024-04-22T13:22:22.715 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout Added host 'smithi087' with addr '172.21.15.87' 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02d07fd700 1 Processor -- start 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02d07fd700 1 -- start start 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02d07fd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02d07fd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8108e30 con 0x7f02c8108440 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02ce599700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02ce599700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54492/0 (socket says 172.21.15.87:54492) 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.932+0000 7f02ce599700 1 -- 172.21.15.87:0/1374486587 learned_addr learned my addr 172.21.15.87:0/1374486587 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02ce599700 1 -- 172.21.15.87:0/1374486587 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c81095f0 con 0x7f02c8108440 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02ce599700 1 --2- 172.21.15.87:0/1374486587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02c40098a0 tx=0x7f02c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e081769527649f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02cd597700 1 -- 172.21.15.87:0/1374486587 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c4004160 con 0x7f02c8108440 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02cd597700 1 -- 172.21.15.87:0/1374486587 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f02c4004300 con 0x7f02c8108440 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02cd597700 1 -- 172.21.15.87:0/1374486587 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c4011af0 con 0x7f02c8108440 2024-04-22T13:22:22.716 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02d07fd700 1 -- 172.21.15.87:0/1374486587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 msgr2=0x7f02c8108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.933+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1374486587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02c40098a0 tx=0x7f02c4009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- 172.21.15.87:0/1374486587 shutdown_connections 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1374486587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c8108860 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- 172.21.15.87:0/1374486587 >> 172.21.15.87:0/1374486587 conn(0x7f02c8103ae0 msgr2=0x7f02c8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- 172.21.15.87:0/1374486587 shutdown_connections 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- 172.21.15.87:0/1374486587 wait complete. 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 Processor -- start 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- start start 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02d07fd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810c9a0 con 0x7f02c8108440 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02ce599700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02ce599700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54498/0 (socket says 172.21.15.87:54498) 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.934+0000 7f02ce599700 1 -- 172.21.15.87:0/1861510451 learned_addr learned my addr 172.21.15.87:0/1861510451 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:22.717 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02ce599700 1 -- 172.21.15.87:0/1861510451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c807e0c0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02ce599700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02c4003f40 tx=0x7f02c4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c40038b0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c807c2b0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f02c4003a50 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c401a5c0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.935+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c807c7f0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f02c401a8c0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02bf7fe700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 0x7f02b403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f02c404daf0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c80640a0 con 0x7f02c8108440 2024-04-22T13:22:22.718 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02cdd98700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 0x7f02b403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.936+0000 7f02cdd98700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 0x7f02b403ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02b8006fd0 tx=0x7f02b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:17.941+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02c4019c70 con 0x7f02c8108440 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:18.145+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}) -- 0x7f02c807cf70 con 0x7f02b4039560 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.161+0000 7f02bf7fe700 1 -- 172.21.15.87:0/1861510451 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f02c807cf70 con 0x7f02b4039560 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 msgr2=0x7f02b403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 0x7f02b403ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02b8006fd0 tx=0x7f02b8006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 msgr2=0x7f02c807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02c4003f40 tx=0x7f02c4011890 comp rx=0 tx=0).stop 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 shutdown_connections 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f02b4039560 0x7f02b403ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 --2- 172.21.15.87:0/1861510451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02c8108440 0x7f02c807db80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:22.719 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 >> 172.21.15.87:0/1861510451 conn(0x7f02c8103ae0 msgr2=0x7f02c81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:22.720 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 shutdown_connections 2024-04-22T13:22:22.720 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:22.164+0000 7f02d07fd700 1 -- 172.21.15.87:0/1861510451 wait complete. 2024-04-22T13:22:22.720 INFO:teuthology.orchestra.run.smithi087.stdout:Deploying unmanaged mon service... 2024-04-22T13:22:23.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:23 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:23.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:23 smithi087 ceph-mon[27999]: Added host smithi087 2024-04-22T13:22:23.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:23 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:24.014 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.206+0000 7f9231147700 1 Processor -- start 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.206+0000 7f9231147700 1 -- start start 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f9231147700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f9231147700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f922c07bde0 con 0x7f922c108fa0 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f922ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f922ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44736/0 (socket says 172.21.15.87:44736) 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f922ad9d700 1 -- 172.21.15.87:0/1808427335 learned_addr learned my addr 172.21.15.87:0/1808427335 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f922ad9d700 1 -- 172.21.15.87:0/1808427335 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c10b8d0 con 0x7f922c108fa0 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f922ad9d700 1 --2- 172.21.15.87:0/1808427335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098a0 tx=0x7f921c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df069cc0da248eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f9229d9b700 1 -- 172.21.15.87:0/1808427335 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f921c004160 con 0x7f922c108fa0 2024-04-22T13:22:24.015 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f9229d9b700 1 -- 172.21.15.87:0/1808427335 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f921c004300 con 0x7f922c108fa0 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.207+0000 7f9229d9b700 1 -- 172.21.15.87:0/1808427335 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f921c011af0 con 0x7f922c108fa0 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 -- 172.21.15.87:0/1808427335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 msgr2=0x7f922c10b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 --2- 172.21.15.87:0/1808427335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098a0 tx=0x7f921c009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 -- 172.21.15.87:0/1808427335 shutdown_connections 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 --2- 172.21.15.87:0/1808427335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c10b390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 -- 172.21.15.87:0/1808427335 >> 172.21.15.87:0/1808427335 conn(0x7f922c102950 msgr2=0x7f922c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 -- 172.21.15.87:0/1808427335 shutdown_connections 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.208+0000 7f9231147700 1 -- 172.21.15.87:0/1808427335 wait complete. 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f9231147700 1 Processor -- start 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f9231147700 1 -- start start 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f9231147700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f9231147700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f922c10c7d0 con 0x7f922c108fa0 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f922ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:24.016 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f922ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44746/0 (socket says 172.21.15.87:44746) 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.209+0000 7f922ad9d700 1 -- 172.21.15.87:0/3715790464 learned_addr learned my addr 172.21.15.87:0/3715790464 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f922ad9d700 1 -- 172.21.15.87:0/3715790464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c19c910 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f922ad9d700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f921c009fb0 tx=0x7f921c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f921c0036a0 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f921c01a430 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922c19cba0 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f921c019550 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.210+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922c19f890 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f921c019850 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922c0640a0 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f9213fff700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 0x7f921403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f921c04d8f0 con 0x7f922c108fa0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f922a59c700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 0x7f921403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.211+0000 7f922a59c700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 0x7f921403ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9220006fd0 tx=0x7f9220006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:24.017 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.216+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f921c00f2e0 con 0x7f922c108fa0 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.434+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f922c103c10 con 0x7f9214039560 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.442+0000 7f9213fff700 1 -- 172.21.15.87:0/3715790464 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f922c103c10 con 0x7f9214039560 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 msgr2=0x7f921403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 0x7f921403ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9220006fd0 tx=0x7f9220006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 msgr2=0x7f922c19c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f921c009fb0 tx=0x7f921c011940 comp rx=0 tx=0).stop 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 shutdown_connections 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f9214039560 0x7f921403ba20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 --2- 172.21.15.87:0/3715790464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c108fa0 0x7f922c19c3d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.444+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 >> 172.21.15.87:0/3715790464 conn(0x7f922c102950 msgr2=0x7f922c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.445+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 shutdown_connections 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:23.445+0000 7f9231147700 1 -- 172.21.15.87:0/3715790464 wait complete. 2024-04-22T13:22:24.018 INFO:teuthology.orchestra.run.smithi087.stdout:Deploying unmanaged mgr service... 2024-04-22T13:22:24.670 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:24 smithi087 ceph-mon[27999]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:24.670 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:24 smithi087 ceph-mon[27999]: Saving service mon spec with placement count:5 2024-04-22T13:22:24.670 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:24 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:25.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:25 smithi087 ceph-mon[27999]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:25.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:25 smithi087 ceph-mon[27999]: Saving service mgr spec with placement count:2 2024-04-22T13:22:25.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:25 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:25.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:25 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:25.943 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T13:22:25.943 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.457+0000 7fea4f95d700 1 Processor -- start 2024-04-22T13:22:25.943 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.457+0000 7fea4f95d700 1 -- start start 2024-04-22T13:22:25.943 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4f95d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4f95d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fea4807bea0 con 0x7fea48108e80 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4d6f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4d6f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44758/0 (socket says 172.21.15.87:44758) 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4d6f9700 1 -- 172.21.15.87:0/2335354441 learned_addr learned my addr 172.21.15.87:0/2335354441 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4d6f9700 1 -- 172.21.15.87:0/2335354441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4810b7b0 con 0x7fea48108e80 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea4d6f9700 1 --2- 172.21.15.87:0/2335354441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea380098a0 tx=0x7fea38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3025ba933c05d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea3ffff700 1 -- 172.21.15.87:0/2335354441 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea38004160 con 0x7fea48108e80 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea3ffff700 1 -- 172.21.15.87:0/2335354441 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea38004300 con 0x7fea48108e80 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.458+0000 7fea3ffff700 1 -- 172.21.15.87:0/2335354441 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea38011af0 con 0x7fea48108e80 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 -- 172.21.15.87:0/2335354441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 msgr2=0x7fea4810b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 --2- 172.21.15.87:0/2335354441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea380098a0 tx=0x7fea38009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 -- 172.21.15.87:0/2335354441 shutdown_connections 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 --2- 172.21.15.87:0/2335354441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4810b270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:25.944 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 -- 172.21.15.87:0/2335354441 >> 172.21.15.87:0/2335354441 conn(0x7fea48102910 msgr2=0x7fea48104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 -- 172.21.15.87:0/2335354441 shutdown_connections 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.459+0000 7fea4f95d700 1 -- 172.21.15.87:0/2335354441 wait complete. 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4f95d700 1 Processor -- start 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4f95d700 1 -- start start 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4f95d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4f95d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fea4810c700 con 0x7fea48108e80 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4d6f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4d6f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44762/0 (socket says 172.21.15.87:44762) 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4d6f9700 1 -- 172.21.15.87:0/3263634353 learned_addr learned my addr 172.21.15.87:0/3263634353 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4d6f9700 1 -- 172.21.15.87:0/3263634353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4819c860 con 0x7fea48108e80 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4d6f9700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fb0 tx=0x7fea38011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea380036a0 con 0x7fea48108e80 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea3801a430 con 0x7fea48108e80 2024-04-22T13:22:25.945 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.460+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4819cb50 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.461+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea38019550 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.461+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4819f7e0 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.461+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fea38019800 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.461+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea480640a0 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.462+0000 7fea3e7fc700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 0x7fea3403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.462+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fea3804da20 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.462+0000 7fea4cef8700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 0x7fea3403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.462+0000 7fea4cef8700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 0x7fea3403ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea44006fd0 tx=0x7fea44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.467+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea38003840 con 0x7fea48108e80 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.685+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fea4819f9b0 con 0x7fea340395b0 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.692+0000 7fea3e7fc700 1 -- 172.21.15.87:0/3263634353 <== mgr.14120 v2:172.21.15.87:6800/35245616 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fea4819f9b0 con 0x7fea340395b0 2024-04-22T13:22:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 msgr2=0x7fea3403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 0x7fea3403ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fea44006fd0 tx=0x7fea44006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 msgr2=0x7fea4819c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fb0 tx=0x7fea38011940 comp rx=0 tx=0).stop 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 shutdown_connections 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fea340395b0 0x7fea3403ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 --2- 172.21.15.87:0/3263634353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea48108e80 0x7fea4819c320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 >> 172.21.15.87:0/3263634353 conn(0x7fea48102910 msgr2=0x7fea48103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 shutdown_connections 2024-04-22T13:22:25.947 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:24.694+0000 7fea4f95d700 1 -- 172.21.15.87:0/3263634353 wait complete. 2024-04-22T13:22:27.150 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.362+0000 7fac8aff1700 1 Processor -- start 2024-04-22T13:22:27.150 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.362+0000 7fac8aff1700 1 -- start start 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac8aff1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac8aff1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fac84108e30 con 0x7fac84108440 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac88d8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac88d8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44778/0 (socket says 172.21.15.87:44778) 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac88d8d700 1 -- 172.21.15.87:0/1007995485 learned_addr learned my addr 172.21.15.87:0/1007995485 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac88d8d700 1 -- 172.21.15.87:0/1007995485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac841095f0 con 0x7fac84108440 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac88d8d700 1 --2- 172.21.15.87:0/1007995485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac780098a0 tx=0x7fac78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c5675784f409f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac837fe700 1 -- 172.21.15.87:0/1007995485 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac78004160 con 0x7fac84108440 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac837fe700 1 -- 172.21.15.87:0/1007995485 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac78004300 con 0x7fac84108440 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.363+0000 7fac837fe700 1 -- 172.21.15.87:0/1007995485 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac78011a20 con 0x7fac84108440 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 -- 172.21.15.87:0/1007995485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 msgr2=0x7fac84108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1007995485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac780098a0 tx=0x7fac78009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:27.151 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 -- 172.21.15.87:0/1007995485 shutdown_connections 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1007995485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac84108860 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 -- 172.21.15.87:0/1007995485 >> 172.21.15.87:0/1007995485 conn(0x7fac84103a00 msgr2=0x7fac84105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 -- 172.21.15.87:0/1007995485 shutdown_connections 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.364+0000 7fac8aff1700 1 -- 172.21.15.87:0/1007995485 wait complete. 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac8aff1700 1 Processor -- start 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac8aff1700 1 -- start start 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac8aff1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac8aff1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fac8410c9a0 con 0x7fac84108440 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac88d8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac88d8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44786/0 (socket says 172.21.15.87:44786) 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.365+0000 7fac88d8d700 1 -- 172.21.15.87:0/1938974241 learned_addr learned my addr 172.21.15.87:0/1938974241 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac88d8d700 1 -- 172.21.15.87:0/1938974241 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8419c850 con 0x7fac84108440 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac88d8d700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac78009520 tx=0x7fac78011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:27.152 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac780039e0 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8419cae0 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac7801b8f0 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac78019770 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.366+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8419f7d0 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.367+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fac78019a70 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.367+0000 7fac81ffb700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 0x7fac7403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.367+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac840640a0 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.367+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fac7804ef70 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.368+0000 7fac83fff700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 0x7fac7403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.368+0000 7fac83fff700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 0x7fac7403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac70006fd0 tx=0x7fac70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.373+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac7801f030 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.577+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fac8419fd50 con 0x7fac84108440 2024-04-22T13:22:27.153 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.582+0000 7fac81ffb700 1 -- 172.21.15.87:0/1938974241 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fac780172b0 con 0x7fac84108440 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.584+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 msgr2=0x7fac7403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.584+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 0x7fac7403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac70006fd0 tx=0x7fac70006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.584+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 msgr2=0x7fac8419c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.584+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac78009520 tx=0x7fac78011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 shutdown_connections 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fac740395b0 0x7fac7403ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 --2- 172.21.15.87:0/1938974241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac84108440 0x7fac8419c310 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 >> 172.21.15.87:0/1938974241 conn(0x7fac84103a00 msgr2=0x7fac841044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 shutdown_connections 2024-04-22T13:22:27.154 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:26.585+0000 7fac8aff1700 1 -- 172.21.15.87:0/1938974241 wait complete. 2024-04-22T13:22:27.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:27 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:27.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:27 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1938974241' entity='client.admin' 2024-04-22T13:22:28.968 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:28 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3885526835' entity='client.admin' 2024-04-22T13:22:28.968 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:28 smithi087 ceph-mon[27999]: from='mgr.14120 172.21.15.87:0/506497744' entity='mgr.x' 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.572+0000 7fca8eef7700 1 Processor -- start 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.572+0000 7fca8eef7700 1 -- start start 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.572+0000 7fca8eef7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.572+0000 7fca8eef7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fca88107d40 con 0x7fca88107350 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca8cc93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:28.976 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca8cc93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44792/0 (socket says 172.21.15.87:44792) 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca8cc93700 1 -- 172.21.15.87:0/911563097 learned_addr learned my addr 172.21.15.87:0/911563097 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca8cc93700 1 -- 172.21.15.87:0/911563097 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca88108550 con 0x7fca88107350 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca8cc93700 1 --2- 172.21.15.87:0/911563097 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e80 tx=0x7fca8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bf52bf38a907a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca7f7fe700 1 -- 172.21.15.87:0/911563097 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca84017070 con 0x7fca88107350 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca7f7fe700 1 -- 172.21.15.87:0/911563097 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca8400b8c0 con 0x7fca88107350 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.573+0000 7fca7f7fe700 1 -- 172.21.15.87:0/911563097 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca84004490 con 0x7fca88107350 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- 172.21.15.87:0/911563097 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 msgr2=0x7fca88107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 --2- 172.21.15.87:0/911563097 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e80 tx=0x7fca8400b330 comp rx=0 tx=0).stop 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- 172.21.15.87:0/911563097 shutdown_connections 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 --2- 172.21.15.87:0/911563097 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca88107770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- 172.21.15.87:0/911563097 >> 172.21.15.87:0/911563097 conn(0x7fca88102950 msgr2=0x7fca88104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- 172.21.15.87:0/911563097 shutdown_connections 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- 172.21.15.87:0/911563097 wait complete. 2024-04-22T13:22:28.977 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 Processor -- start 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.574+0000 7fca8eef7700 1 -- start start 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8eef7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8eef7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810b900 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8cc93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8cc93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44802/0 (socket says 172.21.15.87:44802) 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8cc93700 1 -- 172.21.15.87:0/3885526835 learned_addr learned my addr 172.21.15.87:0/3885526835 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8cc93700 1 -- 172.21.15.87:0/3885526835 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8819f180 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8cc93700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e50 tx=0x7fca8400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca84020020 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca84003b50 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca84017040 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8819f350 con 0x7fca88107350 2024-04-22T13:22:28.978 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.575+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8819f770 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.576+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fca84023020 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.576+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca880640a0 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.577+0000 7fca7dffb700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 0x7fca7003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.577+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fca8400f050 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.577+0000 7fca7ffff700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 0x7fca7003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.577+0000 7fca7ffff700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 0x7fca7003ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca78006fd0 tx=0x7fca78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.582+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca8402a020 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.799+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fca8819ff10 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.802+0000 7fca7dffb700 1 -- 172.21.15.87:0/3885526835 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fca8402c300 con 0x7fca88107350 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 msgr2=0x7fca7003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 0x7fca7003ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca78006fd0 tx=0x7fca78006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 msgr2=0x7fca8807ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca84009e50 tx=0x7fca8400bd50 comp rx=0 tx=0).stop 2024-04-22T13:22:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 shutdown_connections 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fca70039560 0x7fca7003ba20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 --2- 172.21.15.87:0/3885526835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca88107350 0x7fca8807ba60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 >> 172.21.15.87:0/3885526835 conn(0x7fca88102950 msgr2=0x7fca881034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.804+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 shutdown_connections 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:27.805+0000 7fca8eef7700 1 -- 172.21.15.87:0/3885526835 wait complete. 2024-04-22T13:22:28.980 INFO:teuthology.orchestra.run.smithi087.stdout:Enabling the dashboard module... 2024-04-22T13:22:29.981 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:29 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2610335563' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T13:22:29.981 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:29 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setuser ceph since I am not root 2024-04-22T13:22:29.981 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:29 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setgroup ceph since I am not root 2024-04-22T13:22:29.981 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:29 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:29.983+0000 7fa4351c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:22:30.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:30.170+0000 7fa4351c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:22:30.381 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a2d2d700 1 Processor -- start 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a2d2d700 1 -- start start 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a2d2d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a2d2d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f639c107ce0 con 0x7f639c1072f0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a0ac9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a0ac9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44810/0 (socket says 172.21.15.87:44810) 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.413+0000 7f63a0ac9700 1 -- 172.21.15.87:0/2227888265 learned_addr learned my addr 172.21.15.87:0/2227888265 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.414+0000 7f63a0ac9700 1 -- 172.21.15.87:0/2227888265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c1084f0 con 0x7f639c1072f0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.414+0000 7f63a0ac9700 1 --2- 172.21.15.87:0/2227888265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63900098a0 tx=0x7f6390009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a21506e79e066f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.414+0000 7f639b7fe700 1 -- 172.21.15.87:0/2227888265 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6390004160 con 0x7f639c1072f0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.414+0000 7f639b7fe700 1 -- 172.21.15.87:0/2227888265 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6390004300 con 0x7f639c1072f0 2024-04-22T13:22:30.382 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.414+0000 7f639b7fe700 1 -- 172.21.15.87:0/2227888265 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6390011a20 con 0x7f639c1072f0 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2227888265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 msgr2=0x7f639c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2227888265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63900098a0 tx=0x7f6390009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2227888265 shutdown_connections 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2227888265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c107710 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2227888265 >> 172.21.15.87:0/2227888265 conn(0x7f639c102910 msgr2=0x7f639c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2227888265 shutdown_connections 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.415+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2227888265 wait complete. 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a2d2d700 1 Processor -- start 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a2d2d700 1 -- start start 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a2d2d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a2d2d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f639c10b8a0 con 0x7f639c1072f0 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a0ac9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a0ac9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44820/0 (socket says 172.21.15.87:44820) 2024-04-22T13:22:30.383 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a0ac9700 1 -- 172.21.15.87:0/2610335563 learned_addr learned my addr 172.21.15.87:0/2610335563 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a0ac9700 1 -- 172.21.15.87:0/2610335563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c19f0d0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a0ac9700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f63900119f0 tx=0x7f6390011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63900039e0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f639001b8f0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639c19f2a0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.416+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63900237a0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.417+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639c19f640 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f639001b440 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639c0640a0 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f6399ffb700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 0x7f638c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f639004df00 con 0x7f639c1072f0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f639bfff700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 0x7f638c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.418+0000 7f639bfff700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 0x7f638c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6388006fd0 tx=0x7f6388006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:30.384 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.423+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f639001f030 con 0x7f639c1072f0 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.689+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f639c19fb00 con 0x7f639c1072f0 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.805+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f6390018d80 con 0x7f639c1072f0 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.807+0000 7f6399ffb700 1 -- 172.21.15.87:0/2610335563 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f6390019750 con 0x7f639c1072f0 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.808+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 msgr2=0x7f638c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.808+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 0x7f638c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6388006fd0 tx=0x7f6388006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.808+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 msgr2=0x7f639c07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.808+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f63900119f0 tx=0x7f6390011ad0 comp rx=0 tx=0).stop 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 shutdown_connections 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f638c0395b0 0x7f638c03ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 --2- 172.21.15.87:0/2610335563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f639c1072f0 0x7f639c07ba60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 >> 172.21.15.87:0/2610335563 conn(0x7f639c102910 msgr2=0x7f639c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 shutdown_connections 2024-04-22T13:22:30.385 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:29.809+0000 7f63a2d2d700 1 -- 172.21.15.87:0/2610335563 wait complete. 2024-04-22T13:22:30.585 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:30.272+0000 7fa4351c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:22:30.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2610335563' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T13:22:30.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-mon[27999]: mgrmap e9: x(active, since 18s) 2024-04-22T13:22:30.988 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:30.588+0000 7fa4351c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:22:30.988 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:30 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:30.679+0000 7fa4351c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:22:31.755 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4f01d4700 1 Processor -- start 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4f01d4700 1 -- start start 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4f01d4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4f01d4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e810a3d0 con 0x7fe4e81099e0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4edf70700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4edf70700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44444/0 (socket says 172.21.15.87:44444) 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.799+0000 7fe4edf70700 1 -- 172.21.15.87:0/3864534073 learned_addr learned my addr 172.21.15.87:0/3864534073 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4edf70700 1 -- 172.21.15.87:0/3864534073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e810a550 con 0x7fe4e81099e0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4edf70700 1 --2- 172.21.15.87:0/3864534073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40098a0 tx=0x7fe4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916642065b5aca11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4ecf6e700 1 -- 172.21.15.87:0/3864534073 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4e4004160 con 0x7fe4e81099e0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4ecf6e700 1 -- 172.21.15.87:0/3864534073 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4e4004300 con 0x7fe4e81099e0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4ecf6e700 1 -- 172.21.15.87:0/3864534073 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4e4011a20 con 0x7fe4e81099e0 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4f01d4700 1 -- 172.21.15.87:0/3864534073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 msgr2=0x7fe4e8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:31.756 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.800+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/3864534073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40098a0 tx=0x7fe4e4009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- 172.21.15.87:0/3864534073 shutdown_connections 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/3864534073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e8109e00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- 172.21.15.87:0/3864534073 >> 172.21.15.87:0/3864534073 conn(0x7fe4e8102080 msgr2=0x7fe4e81044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- 172.21.15.87:0/3864534073 shutdown_connections 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- 172.21.15.87:0/3864534073 wait complete. 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 Processor -- start 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- start start 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.801+0000 7fe4f01d4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e81072a0 con 0x7fe4e81099e0 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4edf70700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4edf70700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44448/0 (socket says 172.21.15.87:44448) 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4edf70700 1 -- 172.21.15.87:0/2878546725 learned_addr learned my addr 172.21.15.87:0/2878546725 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4edf70700 1 -- 172.21.15.87:0/2878546725 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e819c870 con 0x7fe4e81099e0 2024-04-22T13:22:31.757 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4edf70700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40119f0 tx=0x7fe4e4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4e40039e0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4e401b8f0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4e40237a0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e819cb60 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.802+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e819f7f0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fe4e40193d0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e80640a0 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4deffd700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 0x7fe4d403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe4e404df30 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4ed76f700 1 -- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 msgr2=0x7fe4d403bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.803+0000 7fe4ed76f700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 0x7fe4d403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:30.808+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4e401f030 con 0x7fe4e81099e0 2024-04-22T13:22:31.758 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.004+0000 7fe4ed76f700 1 -- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 msgr2=0x7fe4d403bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.004+0000 7fe4ed76f700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 0x7fe4d403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.085+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe4e819ff00 con 0x7fe4e81099e0 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.085+0000 7fe4deffd700 1 -- 172.21.15.87:0/2878546725 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe4e400f2e0 con 0x7fe4e81099e0 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 msgr2=0x7fe4d403bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 0x7fe4d403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 msgr2=0x7fe4e819c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40119f0 tx=0x7fe4e4011ad0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 shutdown_connections 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7fe4d4039610 0x7fe4d403bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 --2- 172.21.15.87:0/2878546725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe4e81099e0 0x7fe4e819c330 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40119f0 tx=0x7fe4e4011ad0 comp rx=0 tx=0).stop 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.088+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 >> 172.21.15.87:0/2878546725 conn(0x7fe4e8102080 msgr2=0x7fe4e8102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.089+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 shutdown_connections 2024-04-22T13:22:31.759 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:31.089+0000 7fe4f01d4700 1 -- 172.21.15.87:0/2878546725 wait complete. 2024-04-22T13:22:31.760 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for the mgr to restart... 2024-04-22T13:22:31.760 INFO:teuthology.orchestra.run.smithi087.stdout:Waiting for mgr epoch 9... 2024-04-22T13:22:32.005 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:31 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2878546725' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T13:22:32.005 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:31 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:31.894+0000 7fa4351c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:22:32.392 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:32 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:32.072+0000 7fa4351c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:22:32.392 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:32 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:32.249+0000 7fa4351c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:22:32.738 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:32 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:32.538+0000 7fa4351c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:22:32.738 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:32 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:32.631+0000 7fa4351c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:22:33.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:32 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:32.834+0000 7fa4351c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:22:33.238 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:33 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:33.026+0000 7fa4351c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:22:33.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:33 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:33.621+0000 7fa4351c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:22:34.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:34.044+0000 7fa4351c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:22:34.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:34.140+0000 7fa4351c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:22:34.339 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:34.342+0000 7fa4351c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:22:35.195 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:34.860+0000 7fa4351c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:22:35.195 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:34.953+0000 7fa4351c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:22:35.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:35.198+0000 7fa4351c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:22:35.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:35.330+0000 7fa4351c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:22:35.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:35.427+0000 7fa4351c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:22:35.843 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:35.545+0000 7fa4351c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:22:35.843 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:35.846+0000 7fa4351c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:22:36.716 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:36.447+0000 7fa4351c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:22:36.716 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:22:36.665+0000 7fa4351c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: Active manager daemon x restarted 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: Activating manager daemon x 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: mgrmap e10: x(active, starting, since 0.00331119s) 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: Manager daemon x is now available 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:36.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:22:37.880 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:22:37.880 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:37 smithi087 ceph-mon[27999]: mgrmap e11: x(active, since 1.00682s) 2024-04-22T13:22:38.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout { 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout } 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e746a700 1 Processor -- start 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e746a700 1 -- start start 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e746a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e746a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f40e007be60 con 0x7f40e0108ef0 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e5206700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.189+0000 7f40e5206700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44460/0 (socket says 172.21.15.87:44460) 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40e5206700 1 -- 172.21.15.87:0/247915927 learned_addr learned my addr 172.21.15.87:0/247915927 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:38.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40e5206700 1 -- 172.21.15.87:0/247915927 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e010b820 con 0x7f40e0108ef0 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40e5206700 1 --2- 172.21.15.87:0/247915927 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40d40098a0 tx=0x7f40d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f2770df903edad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40d3fff700 1 -- 172.21.15.87:0/247915927 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d4004160 con 0x7f40e0108ef0 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40d3fff700 1 -- 172.21.15.87:0/247915927 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40d4004300 con 0x7f40e0108ef0 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.190+0000 7f40d3fff700 1 -- 172.21.15.87:0/247915927 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d4011a20 con 0x7f40e0108ef0 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 -- 172.21.15.87:0/247915927 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 msgr2=0x7f40e010b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 --2- 172.21.15.87:0/247915927 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40d40098a0 tx=0x7f40d4009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 -- 172.21.15.87:0/247915927 shutdown_connections 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 --2- 172.21.15.87:0/247915927 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e010b2e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 -- 172.21.15.87:0/247915927 >> 172.21.15.87:0/247915927 conn(0x7f40e01028a0 msgr2=0x7f40e0104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 -- 172.21.15.87:0/247915927 shutdown_connections 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.191+0000 7f40e746a700 1 -- 172.21.15.87:0/247915927 wait complete. 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e746a700 1 Processor -- start 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e746a700 1 -- start start 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e746a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e746a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f40e010c7d0 con 0x7f40e0108ef0 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e5206700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e5206700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44462/0 (socket says 172.21.15.87:44462) 2024-04-22T13:22:38.586 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e5206700 1 -- 172.21.15.87:0/306037932 learned_addr learned my addr 172.21.15.87:0/306037932 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e5206700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e019c8d0 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.192+0000 7f40e5206700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40d40116e0 tx=0x7f40d4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d4003b20 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40d401b8f0 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d40238c0 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e019cb60 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e019f850 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.193+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f40d4023a60 con 0x7f40e0108ef0 2024-04-22T13:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.194+0000 7f40d27fc700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.194+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40cc03c1e0 con 0x7f40cc039610 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.194+0000 7f40e4a05700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.194+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f40d404e060 con 0x7f40e0108ef0 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.194+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.395+0000 7f40e4a05700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.395+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.796+0000 7f40e4a05700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:32.796+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:33.597+0000 7f40e4a05700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:33.597+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:35.198+0000 7f40e4a05700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/35245616 2024-04-22T13:22:38.588 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:35.198+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:36.669+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f40d401baa0 con 0x7f40e0108ef0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:36.669+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 msgr2=0x7f40cc03bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:36.669+0000 7f40d27fc700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.671+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f40d4050170 con 0x7f40e0108ef0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.672+0000 7f40d27fc700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 0x7f40cc03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.672+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40cc03c1e0 con 0x7f40cc03cff0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.672+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 0x7f40cc03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.673+0000 7f40e4a05700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 0x7f40cc03f3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f40dc003a10 tx=0x7f40dc009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.673+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f40cc03c1e0 con 0x7f40cc03cff0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.679+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f40e00640a0 con 0x7f40cc03cff0 2024-04-22T13:22:38.589 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40d27fc700 1 -- 172.21.15.87:0/306037932 <== mgr.14152 v2:172.21.15.87:6800/1421536657 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f40e00640a0 con 0x7f40cc03cff0 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 msgr2=0x7f40cc03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 0x7f40cc03f3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f40dc003a10 tx=0x7f40dc009270 comp rx=0 tx=0).stop 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 msgr2=0x7f40e019c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40d40116e0 tx=0x7f40d4011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 shutdown_connections 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40e0108ef0 0x7f40e019c390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f40cc03cff0 0x7f40cc03f3e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.590 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 --2- 172.21.15.87:0/306037932 >> [v2:172.21.15.87:6800/35245616,v1:172.21.15.87:6801/35245616] conn(0x7f40cc039610 0x7f40cc03bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 >> 172.21.15.87:0/306037932 conn(0x7f40e01028a0 msgr2=0x7f40e0103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 shutdown_connections 2024-04-22T13:22:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:37.680+0000 7f40e746a700 1 -- 172.21.15.87:0/306037932 wait complete. 2024-04-22T13:22:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:mgr epoch 9 is available 2024-04-22T13:22:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:Generating a dashboard self-signed certificate... 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:37] ENGINE Bus STARTING 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:37] ENGINE Serving on https://172.21.15.87:7150 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:37] ENGINE Serving on http://172.21.15.87:8765 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: [22/Apr/2024:13:22:37] ENGINE Bus STARTED 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T13:22:38.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:39.966 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T13:22:39.966 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042ceaf700 1 Processor -- start 2024-04-22T13:22:39.966 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042ceaf700 1 -- start start 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042ceaf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042ceaf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f042807bec0 con 0x7f0428108e60 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44522/0 (socket says 172.21.15.87:44522) 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.016+0000 7f042659c700 1 -- 172.21.15.87:0/2374944932 learned_addr learned my addr 172.21.15.87:0/2374944932 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.017+0000 7f042659c700 1 -- 172.21.15.87:0/2374944932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042810b790 con 0x7f0428108e60 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.017+0000 7f042659c700 1 --2- 172.21.15.87:0/2374944932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04180098a0 tx=0x7f0418009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d372fb58f863d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.017+0000 7f042559a700 1 -- 172.21.15.87:0/2374944932 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0418004160 con 0x7f0428108e60 2024-04-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.017+0000 7f042559a700 1 -- 172.21.15.87:0/2374944932 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0418004300 con 0x7f0428108e60 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.017+0000 7f042559a700 1 -- 172.21.15.87:0/2374944932 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0418011a20 con 0x7f0428108e60 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 -- 172.21.15.87:0/2374944932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 msgr2=0x7f042810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2374944932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04180098a0 tx=0x7f0418009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 -- 172.21.15.87:0/2374944932 shutdown_connections 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2374944932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042810b250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 -- 172.21.15.87:0/2374944932 >> 172.21.15.87:0/2374944932 conn(0x7f0428102910 msgr2=0x7f0428104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 -- 172.21.15.87:0/2374944932 shutdown_connections 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 -- 172.21.15.87:0/2374944932 wait complete. 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.018+0000 7f042ceaf700 1 Processor -- start 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042ceaf700 1 -- start start 2024-04-22T13:22:39.968 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042ceaf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042ceaf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f042810c7a0 con 0x7f0428108e60 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44530/0 (socket says 172.21.15.87:44530) 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042659c700 1 -- 172.21.15.87:0/2063402724 learned_addr learned my addr 172.21.15.87:0/2063402724 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042659c700 1 -- 172.21.15.87:0/2063402724 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042819c860 con 0x7f0428108e60 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042659c700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04180116e0 tx=0x7f0418011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0418003b20 con 0x7f0428108e60 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042819caf0 con 0x7f0428108e60 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f041801b8f0 con 0x7f0428108e60 2024-04-22T13:22:39.969 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.019+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f042819f7e0 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.020+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04180238c0 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.020+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f0418023b90 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.021+0000 7f04177fe700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 0x7f041003b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.021+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f041804ee90 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.021+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04280640a0 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.021+0000 7f0425d9b700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 0x7f041003b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.021+0000 7f0425d9b700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 0x7f041003b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f041c006fd0 tx=0x7f041c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.027+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f041801f030 con 0x7f0428108e60 2024-04-22T13:22:39.970 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.229+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f041801baa0 con 0x7f0428108e60 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.238+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f0428103b10 con 0x7f04100394a0 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.375+0000 7f04177fe700 1 -- 172.21.15.87:0/2063402724 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f0428103b10 con 0x7f04100394a0 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 msgr2=0x7f041003b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 0x7f041003b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f041c006fd0 tx=0x7f041c006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 msgr2=0x7f042819c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f04180116e0 tx=0x7f0418011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 shutdown_connections 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04100394a0 0x7f041003b960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 --2- 172.21.15.87:0/2063402724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0428108e60 0x7f042819c320 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 >> 172.21.15.87:0/2063402724 conn(0x7f0428102910 msgr2=0x7f0428103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:39.971 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 shutdown_connections 2024-04-22T13:22:39.972 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:39.377+0000 7f042ceaf700 1 -- 172.21.15.87:0/2063402724 wait complete. 2024-04-22T13:22:39.972 INFO:teuthology.orchestra.run.smithi087.stdout:Creating initial admin user... 2024-04-22T13:22:40.376 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:40 smithi087 ceph-mon[27999]: mgrmap e12: x(active, since 2s) 2024-04-22T13:22:40.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:40 smithi087 ceph-mon[27999]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:40.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:40.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:40.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:41.658 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Fvm4o/Gm17GPm8M4a3suHOnQGq4flyuPvNh7TSJb5mbSEp1GPXRo2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713792160, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd321be4700 1 Processor -- start 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd321be4700 1 -- start start 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd321be4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd321be4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c107d40 con 0x7fd31c107350 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd31b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd31b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42948/0 (socket says 172.21.15.87:42948) 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd31b7fe700 1 -- 172.21.15.87:0/2314047961 learned_addr learned my addr 172.21.15.87:0/2314047961 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.411+0000 7fd31b7fe700 1 -- 172.21.15.87:0/2314047961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c108550 con 0x7fd31c107350 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd31b7fe700 1 --2- 172.21.15.87:0/2314047961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0098a0 tx=0x7fd30c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d1fcd465163512 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd31a7fc700 1 -- 172.21.15.87:0/2314047961 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c004160 con 0x7fd31c107350 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd31a7fc700 1 -- 172.21.15.87:0/2314047961 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c004300 con 0x7fd31c107350 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd31a7fc700 1 -- 172.21.15.87:0/2314047961 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c011a20 con 0x7fd31c107350 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd321be4700 1 -- 172.21.15.87:0/2314047961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 msgr2=0x7fd31c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.412+0000 7fd321be4700 1 --2- 172.21.15.87:0/2314047961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0098a0 tx=0x7fd30c009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 -- 172.21.15.87:0/2314047961 shutdown_connections 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 --2- 172.21.15.87:0/2314047961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c107770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 -- 172.21.15.87:0/2314047961 >> 172.21.15.87:0/2314047961 conn(0x7fd31c102950 msgr2=0x7fd31c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 -- 172.21.15.87:0/2314047961 shutdown_connections 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 -- 172.21.15.87:0/2314047961 wait complete. 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 Processor -- start 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.413+0000 7fd321be4700 1 -- start start 2024-04-22T13:22:41.660 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd321be4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd321be4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10b900 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd31b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd31b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42960/0 (socket says 172.21.15.87:42960) 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd31b7fe700 1 -- 172.21.15.87:0/1792088302 learned_addr learned my addr 172.21.15.87:0/1792088302 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd31b7fe700 1 -- 172.21.15.87:0/1792088302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c19f110 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd31b7fe700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c003aa0 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c19f2e0 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c01b8f0 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c023890 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.414+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c19f680 con 0x7fd31c107350 2024-04-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.415+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fd30c02c3f0 con 0x7fd31c107350 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.416+0000 7fd318ff9700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 0x7fd30403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.416+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c0640a0 con 0x7fd31c107350 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.416+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd30c04f040 con 0x7fd31c107350 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.416+0000 7fd31affd700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 0x7fd30403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.416+0000 7fd31affd700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 0x7fd30403bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd310006fd0 tx=0x7fd310006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.421+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd30c01f030 con 0x7fd31c107350 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.639+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd31c103b30 con 0x7fd304039600 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.865+0000 7fd318ff9700 1 -- 172.21.15.87:0/1792088302 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd31c103b30 con 0x7fd304039600 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 msgr2=0x7fd30403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 0x7fd30403bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd310006fd0 tx=0x7fd310006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 msgr2=0x7fd31c07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd30c000c00 tx=0x7fd30c011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 shutdown_connections 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd304039600 0x7fd30403bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 --2- 172.21.15.87:0/1792088302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd31c107350 0x7fd31c07ba60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 >> 172.21.15.87:0/1792088302 conn(0x7fd31c102950 msgr2=0x7fd31c103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 shutdown_connections 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:40.867+0000 7fd321be4700 1 -- 172.21.15.87:0/1792088302 wait complete. 2024-04-22T13:22:41.663 INFO:teuthology.orchestra.run.smithi087.stdout:Fetching dashboard port number... 2024-04-22T13:22:41.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:41 smithi087 ceph-mon[27999]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:41.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:43.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4056689277' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T13:22:43.368 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T13:22:43.368 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.575+0000 7f7e7d2f6700 1 Processor -- start 2024-04-22T13:22:43.368 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.575+0000 7f7e7d2f6700 1 -- start start 2024-04-22T13:22:43.368 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.575+0000 7f7e7d2f6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.575+0000 7f7e7d2f6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781070e0 con 0x7f7e781066f0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42968/0 (socket says 172.21.15.87:42968) 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e76ffd700 1 -- 172.21.15.87:0/1905344207 learned_addr learned my addr 172.21.15.87:0/1905344207 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e76ffd700 1 -- 172.21.15.87:0/1905344207 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e781096d0 con 0x7f7e781066f0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e76ffd700 1 --2- 172.21.15.87:0/1905344207 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2219a8fdfce579b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e767fc700 1 -- 172.21.15.87:0/1905344207 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e68004160 con 0x7f7e781066f0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e767fc700 1 -- 172.21.15.87:0/1905344207 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e68004300 con 0x7f7e781066f0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.576+0000 7f7e767fc700 1 -- 172.21.15.87:0/1905344207 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e68011a20 con 0x7f7e781066f0 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/1905344207 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 msgr2=0x7f7e78106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/1905344207 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e68009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/1905344207 shutdown_connections 2024-04-22T13:22:43.369 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/1905344207 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e78106b10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/1905344207 >> 172.21.15.87:0/1905344207 conn(0x7f7e78102110 msgr2=0x7f7e78104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/1905344207 shutdown_connections 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/1905344207 wait complete. 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.577+0000 7f7e7d2f6700 1 Processor -- start 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e7d2f6700 1 -- start start 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e7d2f6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e7d2f6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7810ca80 con 0x7f7e781066f0 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42984/0 (socket says 172.21.15.87:42984) 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e76ffd700 1 -- 172.21.15.87:0/4056689277 learned_addr learned my addr 172.21.15.87:0/4056689277 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e76ffd700 1 -- 172.21.15.87:0/4056689277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7819c890 con 0x7f7e781066f0 2024-04-22T13:22:43.370 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e76ffd700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009520 tx=0x7f7e68011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.578+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e680039e0 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.579+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7819cb20 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.579+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e6801b8f0 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.579+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7819f810 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.579+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e68019770 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.579+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f7e68019a90 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.580+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e780640a0 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.580+0000 7f7e74ff9700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 0x7f7e6403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.580+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7e6804e150 con 0x7f7e781066f0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.580+0000 7f7e6e7fc700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 0x7f7e6403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.580+0000 7f7e6e7fc700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 0x7f7e6403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e60006fd0 tx=0x7f7e60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:43.371 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.585+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e6801f030 con 0x7f7e781066f0 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.781+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f7e7819fe00 con 0x7f7e781066f0 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.782+0000 7f7e74ff9700 1 -- 172.21.15.87:0/4056689277 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7e6801baa0 con 0x7f7e781066f0 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 msgr2=0x7f7e6403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 0x7f7e6403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e60006fd0 tx=0x7f7e60006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 msgr2=0x7f7e7819c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009520 tx=0x7f7e68011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 shutdown_connections 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7e64039600 0x7f7e6403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 --2- 172.21.15.87:0/4056689277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e781066f0 0x7f7e7819c350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 >> 172.21.15.87:0/4056689277 conn(0x7f7e78102110 msgr2=0x7f7e78102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 shutdown_connections 2024-04-22T13:22:43.372 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:42.784+0000 7f7e7d2f6700 1 -- 172.21.15.87:0/4056689277 wait complete. 2024-04-22T13:22:43.375 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T13:22:43.375 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout disabled 2024-04-22T13:22:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T13:22:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:systemctl: stdout inactive 2024-04-22T13:22:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:firewalld.service is not enabled 2024-04-22T13:22:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout:Ceph Dashboard is now available at: 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout: URL: https://smithi087:8443/ 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout: User: admin 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout: Password: lm70ds8a6y 2024-04-22T13:22:43.385 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:43.386 INFO:teuthology.orchestra.run.smithi087.stdout:Saving cluster configuration to /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config directory 2024-04-22T13:22:43.386 INFO:teuthology.orchestra.run.smithi087.stdout:Enabling autotune for osd_memory_target 2024-04-22T13:22:43.982 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:43 smithi087 ceph-mon[27999]: mgrmap e13: x(active, since 6s) 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb2a82700 1 Processor -- start 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb2a82700 1 -- start start 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb2a82700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb2a82700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac1070e0 con 0x7fdfac1066f0 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb081e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb081e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42990/0 (socket says 172.21.15.87:42990) 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.779+0000 7fdfb081e700 1 -- 172.21.15.87:0/1476695988 learned_addr learned my addr 172.21.15.87:0/1476695988 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb081e700 1 -- 172.21.15.87:0/1476695988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac1096d0 con 0x7fdfac1066f0 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb081e700 1 --2- 172.21.15.87:0/1476695988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098a0 tx=0x7fdfa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657af92d430fbe58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfab7fe700 1 -- 172.21.15.87:0/1476695988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa0004160 con 0x7fdfac1066f0 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfab7fe700 1 -- 172.21.15.87:0/1476695988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfa0004300 con 0x7fdfac1066f0 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfab7fe700 1 -- 172.21.15.87:0/1476695988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa0011a20 con 0x7fdfac1066f0 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb2a82700 1 -- 172.21.15.87:0/1476695988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 msgr2=0x7fdfac106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/1476695988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098a0 tx=0x7fdfa0009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb2a82700 1 -- 172.21.15.87:0/1476695988 shutdown_connections 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/1476695988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac106b10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.780+0000 7fdfb2a82700 1 -- 172.21.15.87:0/1476695988 >> 172.21.15.87:0/1476695988 conn(0x7fdfac102110 msgr2=0x7fdfac104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 -- 172.21.15.87:0/1476695988 shutdown_connections 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 -- 172.21.15.87:0/1476695988 wait complete. 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 Processor -- start 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 -- start start 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb2a82700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac10ca80 con 0x7fdfac1066f0 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb081e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:44.583 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb081e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42996/0 (socket says 172.21.15.87:42996) 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.781+0000 7fdfb081e700 1 -- 172.21.15.87:0/3362213075 learned_addr learned my addr 172.21.15.87:0/3362213075 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfb081e700 1 -- 172.21.15.87:0/3362213075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac19c860 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfb081e700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00116e0 tx=0x7fdfa0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa00039e0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfa001b8f0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa0023760 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac19caf0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.782+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac19f7e0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.783+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fdfa001b440 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.783+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac0640a0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.783+0000 7fdfa9ffb700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 0x7fdf9c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.783+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdfa004e0a0 con 0x7fdfac1066f0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.783+0000 7fdfabfff700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 0x7fdf9c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.784+0000 7fdfabfff700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 0x7fdf9c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf98006fd0 tx=0x7fdf98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.788+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdfa001f030 con 0x7fdfac1066f0 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.986+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fdfac19f9b0 con 0x7fdfac1066f0 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.986+0000 7fdfa9ffb700 1 -- 172.21.15.87:0/3362213075 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fdfa001a3d0 con 0x7fdfac1066f0 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 msgr2=0x7fdf9c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 0x7fdf9c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf98006fd0 tx=0x7fdf98006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 msgr2=0x7fdfac19c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00116e0 tx=0x7fdfa0011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 shutdown_connections 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf9c0396b0 0x7fdf9c03bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 --2- 172.21.15.87:0/3362213075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac1066f0 0x7fdfac19c320 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 >> 172.21.15.87:0/3362213075 conn(0x7fdfac102110 msgr2=0x7fdfac102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 shutdown_connections 2024-04-22T13:22:44.585 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:43.988+0000 7fdfb2a82700 1 -- 172.21.15.87:0/3362213075 wait complete. 2024-04-22T13:22:46.006 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabd212d700 1 Processor -- start 2024-04-22T13:22:46.006 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabd212d700 1 -- start start 2024-04-22T13:22:46.006 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabd212d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:46.006 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabd212d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc07bea0 con 0x7fabcc108ee0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabcb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabcb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43012/0 (socket says 172.21.15.87:43012) 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.049+0000 7fabcb7fe700 1 -- 172.21.15.87:0/1749774038 learned_addr learned my addr 172.21.15.87:0/1749774038 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabcb7fe700 1 -- 172.21.15.87:0/1749774038 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc10b810 con 0x7fabcc108ee0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabcb7fe700 1 --2- 172.21.15.87:0/1749774038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0098a0 tx=0x7fabbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44fd796da6c1e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabca7fc700 1 -- 172.21.15.87:0/1749774038 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabbc004160 con 0x7fabcc108ee0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabca7fc700 1 -- 172.21.15.87:0/1749774038 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabbc004300 con 0x7fabcc108ee0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabca7fc700 1 -- 172.21.15.87:0/1749774038 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabbc011a20 con 0x7fabcc108ee0 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabd212d700 1 -- 172.21.15.87:0/1749774038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 msgr2=0x7fabcc10b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.050+0000 7fabd212d700 1 --2- 172.21.15.87:0/1749774038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0098a0 tx=0x7fabbc009bb0 comp rx=0 tx=0).stop 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- 172.21.15.87:0/1749774038 shutdown_connections 2024-04-22T13:22:46.007 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 --2- 172.21.15.87:0/1749774038 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc10b2d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- 172.21.15.87:0/1749774038 >> 172.21.15.87:0/1749774038 conn(0x7fabcc102950 msgr2=0x7fabcc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- 172.21.15.87:0/1749774038 shutdown_connections 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- 172.21.15.87:0/1749774038 wait complete. 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 Processor -- start 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- start start 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.051+0000 7fabd212d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc10c7b0 con 0x7fabcc108ee0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabcb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabcb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43018/0 (socket says 172.21.15.87:43018) 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabcb7fe700 1 -- 172.21.15.87:0/516005252 learned_addr learned my addr 172.21.15.87:0/516005252 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabcb7fe700 1 -- 172.21.15.87:0/516005252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc19c870 con 0x7fabcc108ee0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabcb7fe700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0116e0 tx=0x7fabbc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabbc003b20 con 0x7fabcc108ee0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabbc01b8f0 con 0x7fabcc108ee0 2024-04-22T13:22:46.008 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc19cb60 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabbc0238c0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.052+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc19f7f0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.053+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fabbc01baa0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.053+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc0640a0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.053+0000 7fabc8ff9700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 0x7fabb403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.053+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fabbc04e1f0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.053+0000 7fabcaffd700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 0x7fabb403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.054+0000 7fabcaffd700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 0x7fabb403bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabc0006fd0 tx=0x7fabc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.058+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabbc01f030 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.368+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fabcc19f9c0 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.370+0000 7fabc8ff9700 1 -- 172.21.15.87:0/516005252 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fabbc02bb90 con 0x7fabcc108ee0 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.372+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 msgr2=0x7fabb403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.372+0000 7fabd212d700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 0x7fabb403bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabc0006fd0 tx=0x7fabc0006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:46.009 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.372+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 msgr2=0x7fabcc19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.372+0000 7fabd212d700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0116e0 tx=0x7fabbc011aa0 comp rx=0 tx=0).stop 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 shutdown_connections 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fabb4039650 0x7fabb403bb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 --2- 172.21.15.87:0/516005252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fabcc108ee0 0x7fabcc19c330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 >> 172.21.15.87:0/516005252 conn(0x7fabcc102950 msgr2=0x7fabcc103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 shutdown_connections 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr 2024-04-22T13:22:45.373+0000 7fabd212d700 1 -- 172.21.15.87:0/516005252 wait complete. 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T13:22:46.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: ceph telemetry on 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout:For more information see: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:22:46.011 INFO:teuthology.orchestra.run.smithi087.stdout:Bootstrap complete. 2024-04-22T13:22:46.051 INFO:tasks.cephadm:Fetching config... 2024-04-22T13:22:46.052 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:22:46.052 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T13:22:46.068 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T13:22:46.068 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:22:46.069 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T13:22:46.124 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T13:22:46.124 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:22:46.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T13:22:46.189 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T13:22:46.189 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:22:46.189 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T13:22:46.244 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T13:22:46.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T13:22:46.316 INFO:teuthology.orchestra.run.smithi087.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:46.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T13:22:46.366 INFO:teuthology.orchestra.run.smithi120.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:46.377 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T13:22:46.416 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzIyLbp+T+h3LAayH9u2I2JZBjtmAoRB3/3IlgCeuYCr+Zqrl40BNMRu2u0igZKzDjqpJvi9S9GdWv52Nk0gWUmZIdyWlSRSoujx/BOPuduYjbLANjD7u8TnQzVwptQhhi6rwbRiXOgU62rsciNb0YlRi8d7bEOpFSrO1+/DdMmZKFDzzNi7tlHL7GgFV/3+t2Ydt8l95jB/OE2zf0/AA48NP4xe9HQuV/wiIVvh6n/1YamKR/pYqfkrTD51LTv3Rt0nBidw6b/7uYratSnijVfv0d8GsDGm69gNbTlGplqYL40FxScPfkGaVKir7NccKmIx/AomMhiQ9F3XaJ09iHD/QsY/0r/Wnan6rBUfKihiKmAvXuUTsFLCSe5wnmHxmXJpigLhv4nN/LS7nKP0pa3Q7iFXw6HYvbqBe5+hobLRRLuY7f/L9++RifOAkeQ8MbyV+dwqwpzYAElx9kJnYA2x45+bAU+biZ0s8JSonc4EyGaclkbO7cLaXGtGJGR/s= ceph-176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:22:46.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T13:22:46.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:46 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/516005252' entity='client.admin' 2024-04-22T13:22:46.667 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 -- 172.21.15.87:0/667463258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 msgr2=0x7f8a28103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 --2- 172.21.15.87:0/667463258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28103fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009910 tx=0x7f8a18009c20 comp rx=0 tx=0).stop 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 -- 172.21.15.87:0/667463258 shutdown_connections 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 --2- 172.21.15.87:0/667463258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28103fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 -- 172.21.15.87:0/667463258 >> 172.21.15.87:0/667463258 conn(0x7f8a280fda90 msgr2=0x7f8a280ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 -- 172.21.15.87:0/667463258 shutdown_connections 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.919+0000 7f8a30199700 1 -- 172.21.15.87:0/667463258 wait complete. 2024-04-22T13:22:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a30199700 1 Processor -- start 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a30199700 1 -- start start 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a30199700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a30199700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8a28108520 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a2df35700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a2df35700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43038/0 (socket says 172.21.15.87:43038) 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.920+0000 7f8a2df35700 1 -- 172.21.15.87:0/3346602800 learned_addr learned my addr 172.21.15.87:0/3346602800 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a2df35700 1 -- 172.21.15.87:0/3346602800 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a28198140 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a2df35700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005800 tx=0x7f8a18009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a18027070 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a281983d0 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a1800be10 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a1800f460 con 0x7f8a28103bb0 2024-04-22T13:22:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.921+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2819b0c0 con 0x7f8a28103bb0 2024-04-22T13:22:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8a1802a070 con 0x7f8a28103bb0 2024-04-22T13:22:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a1effd700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 0x7f8a1403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8a1801e030 con 0x7f8a28103bb0 2024-04-22T13:22:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a2d734700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 0x7f8a1403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a28108ca0 con 0x7f8a28103bb0 2024-04-22T13:22:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.922+0000 7f8a2d734700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 0x7f8a1403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006fd0 tx=0x7f8a24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:48.927+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a18031070 con 0x7f8a28103bb0 2024-04-22T13:22:49.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.128+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f8a28190d50 con 0x7f8a28103bb0 2024-04-22T13:22:49.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.130+0000 7f8a1effd700 1 -- 172.21.15.87:0/3346602800 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f8a18013070 con 0x7f8a28103bb0 2024-04-22T13:22:49.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.132+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 msgr2=0x7f8a1403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:49.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.132+0000 7f8a30199700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 0x7f8a1403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006fd0 tx=0x7f8a24006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.132+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 msgr2=0x7f8a28197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005800 tx=0x7f8a18009f60 comp rx=0 tx=0).stop 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 shutdown_connections 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8a14039650 0x7f8a1403bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 --2- 172.21.15.87:0/3346602800 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a28103bb0 0x7f8a28197c00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 >> 172.21.15.87:0/3346602800 conn(0x7f8a280fda90 msgr2=0x7f8a280fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 shutdown_connections 2024-04-22T13:22:49.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:49.133+0000 7f8a30199700 1 -- 172.21.15.87:0/3346602800 wait complete. 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:49.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:49.741 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T13:22:49.742 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T13:22:50.021 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:22:50.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:50 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3346602800' entity='client.admin' 2024-04-22T13:22:50.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:50.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 -- 172.21.15.87:0/3119560758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 msgr2=0x7fb0841017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 --2- 172.21.15.87:0/3119560758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb0841017c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb078009910 tx=0x7fb078009c20 comp rx=0 tx=0).stop 2024-04-22T13:22:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 -- 172.21.15.87:0/3119560758 shutdown_connections 2024-04-22T13:22:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 --2- 172.21.15.87:0/3119560758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb0841017c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 -- 172.21.15.87:0/3119560758 >> 172.21.15.87:0/3119560758 conn(0x7fb0840fc940 msgr2=0x7fb0840fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 -- 172.21.15.87:0/3119560758 shutdown_connections 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.580+0000 7fb088faa700 1 -- 172.21.15.87:0/3119560758 wait complete. 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb088faa700 1 Processor -- start 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb088faa700 1 -- start start 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb088faa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb088faa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb0841072e0 con 0x7fb0841013a0 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb08259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb08259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42820/0 (socket says 172.21.15.87:42820) 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.581+0000 7fb08259c700 1 -- 172.21.15.87:0/310013831 learned_addr learned my addr 172.21.15.87:0/310013831 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb08259c700 1 -- 172.21.15.87:0/310013831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08418e950 con 0x7fb0841013a0 2024-04-22T13:22:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb08259c700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb078005800 tx=0x7fb078009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb078027070 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb07800be10 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08418cb40 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb07800f460 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.582+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08418d080 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb07802a070 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb0737fe700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 0x7fb06c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb07801e030 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb081d9b700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 0x7fb06c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb084107a60 con 0x7fb0841013a0 2024-04-22T13:22:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.583+0000 7fb081d9b700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 0x7fb06c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:51.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.588+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb078031070 con 0x7fb0841013a0 2024-04-22T13:22:51.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.792+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fb08418d740 con 0x7fb06c0396b0 2024-04-22T13:22:51.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.795+0000 7fb0737fe700 1 -- 172.21.15.87:0/310013831 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb08418d740 con 0x7fb06c0396b0 2024-04-22T13:22:51.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 msgr2=0x7fb06c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:51.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 0x7fb06c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb074006fd0 tx=0x7fb074006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:51.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 msgr2=0x7fb08418e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:51.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb078005800 tx=0x7fb078009f60 comp rx=0 tx=0).stop 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 shutdown_connections 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0396b0 0x7fb06c03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 --2- 172.21.15.87:0/310013831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0841013a0 0x7fb08418e410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 >> 172.21.15.87:0/310013831 conn(0x7fb0840fc940 msgr2=0x7fb08406a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 shutdown_connections 2024-04-22T13:22:51.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:51.797+0000 7fb088faa700 1 -- 172.21.15.87:0/310013831 wait complete. 2024-04-22T13:22:52.525 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi120 2024-04-22T13:22:52.525 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:22:52.525 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.conf 2024-04-22T13:22:52.543 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:22:52.543 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:22:52.601 INFO:tasks.cephadm:Adding host smithi120 to orchestrator... 2024-04-22T13:22:52.601 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch host add smithi120 2024-04-22T13:22:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:22:53.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:53.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:53.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:53.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:53 smithi087 ceph-mon[27999]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:22:53.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:53 smithi087 ceph-mon[27999]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:22:53.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:53 smithi087 ceph-mon[27999]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:22:54.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.330+0000 7f7c84905700 1 -- 172.21.15.87:0/1704702544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 msgr2=0x7f7c801088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.330+0000 7f7c84905700 1 --2- 172.21.15.87:0/1704702544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c801088c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009910 tx=0x7f7c70009c20 comp rx=0 tx=0).stop 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.330+0000 7f7c84905700 1 -- 172.21.15.87:0/1704702544 shutdown_connections 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.330+0000 7f7c84905700 1 --2- 172.21.15.87:0/1704702544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c801088c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.330+0000 7f7c84905700 1 -- 172.21.15.87:0/1704702544 >> 172.21.15.87:0/1704702544 conn(0x7f7c800fdf80 msgr2=0x7f7c801003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 -- 172.21.15.87:0/1704702544 shutdown_connections 2024-04-22T13:22:54.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 -- 172.21.15.87:0/1704702544 wait complete. 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 Processor -- start 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 -- start start 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.331+0000 7f7c84905700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8010c940 con 0x7f7c801084e0 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c7e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c7e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42836/0 (socket says 172.21.15.87:42836) 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c7e59c700 1 -- 172.21.15.87:0/3161484358 learned_addr learned my addr 172.21.15.87:0/3161484358 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c7e59c700 1 -- 172.21.15.87:0/3161484358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c80076c30 con 0x7f7c801084e0 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c7e59c700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c700056d0 tx=0x7f7c700057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c70026070 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c7000bc00 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c800752e0 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c70003920 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.332+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c800757a0 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.333+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7c70029070 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.333+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c80103b90 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.333+0000 7f7c6f7fe700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 0x7f7c6803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.333+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7c7001e070 con 0x7f7c801084e0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.333+0000 7f7c7dd9b700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 0x7f7c6803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:22:54.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.334+0000 7f7c7dd9b700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 0x7f7c6803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c74006fd0 tx=0x7f7c74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:22:54.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.338+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c70013070 con 0x7f7c801084e0 2024-04-22T13:22:54.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:54.541+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}) -- 0x7f7c800762e0 con 0x7f7c68039650 2024-04-22T13:22:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:54 smithi087 ceph-mon[27999]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:22:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:55 smithi087 ceph-mon[27999]: Deploying cephadm binary to smithi120 2024-04-22T13:22:57.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:57 smithi087 ceph-mon[27999]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:22:58.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.712+0000 7f7c6f7fe700 1 -- 172.21.15.87:0/3161484358 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7c800762e0 con 0x7f7c68039650 2024-04-22T13:22:58.709 INFO:teuthology.orchestra.run.smithi087.stdout:Added host 'smithi120' with addr '172.21.15.120' 2024-04-22T13:22:58.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 msgr2=0x7f7c6803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:58.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 0x7f7c6803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7c74006fd0 tx=0x7f7c74006e40 comp rx=0 tx=0).stop 2024-04-22T13:22:58.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 msgr2=0x7f7c80074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c700056d0 tx=0x7f7c700057b0 comp rx=0 tx=0).stop 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 shutdown_connections 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c68039650 0x7f7c6803bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 --2- 172.21.15.87:0/3161484358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c801084e0 0x7f7c80074d40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 >> 172.21.15.87:0/3161484358 conn(0x7f7c800fdf80 msgr2=0x7f7c8010a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 shutdown_connections 2024-04-22T13:22:58.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:22:58.714+0000 7f7c84905700 1 -- 172.21.15.87:0/3161484358 wait complete. 2024-04-22T13:22:59.301 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T13:22:59.534 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:22:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:59 smithi087 ceph-mon[27999]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:22:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:22:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:59 smithi087 ceph-mon[27999]: Added host smithi120 2024-04-22T13:22:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:22:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:22:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 -- 172.21.15.87:0/373511806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 msgr2=0x7f9708103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 --2- 172.21.15.87:0/373511806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708103a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009910 tx=0x7f96fc009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 -- 172.21.15.87:0/373511806 shutdown_connections 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 --2- 172.21.15.87:0/373511806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708103a90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 -- 172.21.15.87:0/373511806 >> 172.21.15.87:0/373511806 conn(0x7f97080fce40 msgr2=0x7f97080ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 -- 172.21.15.87:0/373511806 shutdown_connections 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.002+0000 7f970eaf2700 1 -- 172.21.15.87:0/373511806 wait complete. 2024-04-22T13:23:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970eaf2700 1 Processor -- start 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970eaf2700 1 -- start start 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970eaf2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970eaf2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f970810c9b0 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970c88e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970c88e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56964/0 (socket says 172.21.15.87:56964) 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.003+0000 7f970c88e700 1 -- 172.21.15.87:0/2013919264 learned_addr learned my addr 172.21.15.87:0/2013919264 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f970c88e700 1 -- 172.21.15.87:0/2013919264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9708103200 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f970c88e700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0056d0 tx=0x7f96fc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96fc026070 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96fc004370 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9708101850 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96fc0039b0 con 0x7f97081036b0 2024-04-22T13:23:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.004+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9708101d10 con 0x7f97081036b0 2024-04-22T13:23:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.005+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f96fc029070 con 0x7f97081036b0 2024-04-22T13:23:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.005+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f970806b0f0 con 0x7f97081036b0 2024-04-22T13:23:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.005+0000 7f9705ffb700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 0x7f96f803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.005+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f96fc01e070 con 0x7f97081036b0 2024-04-22T13:23:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.005+0000 7f9707fff700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 0x7f96f803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.006+0000 7f9707fff700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 0x7f96f803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96f4006fd0 tx=0x7f96f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.010+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96fc013070 con 0x7f97081036b0 2024-04-22T13:23:01.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.212+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9708102840 con 0x7f96f80396b0 2024-04-22T13:23:01.209 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:01.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.213+0000 7f9705ffb700 1 -- 172.21.15.87:0/2013919264 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f9708102840 con 0x7f96f80396b0 2024-04-22T13:23:01.210 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:23:01.210 INFO:teuthology.orchestra.run.smithi087.stdout:[{"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}] 2024-04-22T13:23:01.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 msgr2=0x7f96f803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 0x7f96f803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96f4006fd0 tx=0x7f96f4006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 msgr2=0x7f9708101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0056d0 tx=0x7f96fc0057b0 comp rx=0 tx=0).stop 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 shutdown_connections 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f96f80396b0 0x7f96f803bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 --2- 172.21.15.87:0/2013919264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97081036b0 0x7f9708101310 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 >> 172.21.15.87:0/2013919264 conn(0x7f97080fce40 msgr2=0x7f9708104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 shutdown_connections 2024-04-22T13:23:01.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:01.216+0000 7f970eaf2700 1 -- 172.21.15.87:0/2013919264 wait complete. 2024-04-22T13:23:01.852 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-04-22T13:23:01.853 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:23:01.853 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-04-22T13:23:01.872 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:23:01.872 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:01.930 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-04-22T13:23:01.930 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch host add smithi132 2024-04-22T13:23:02.159 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:02.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:01 smithi087 ceph-mon[27999]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:02.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:01 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:02.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:02 smithi087 ceph-mon[27999]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- 172.21.15.87:0/680900262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 msgr2=0x7f6c38103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 --2- 172.21.15.87:0/680900262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38103ab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009910 tx=0x7f6c34009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- 172.21.15.87:0/680900262 shutdown_connections 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 --2- 172.21.15.87:0/680900262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38103ab0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- 172.21.15.87:0/680900262 >> 172.21.15.87:0/680900262 conn(0x7f6c380fce60 msgr2=0x7f6c380ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- 172.21.15.87:0/680900262 shutdown_connections 2024-04-22T13:23:03.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- 172.21.15.87:0/680900262 wait complete. 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 Processor -- start 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.690+0000 7f6c4040f700 1 -- start start 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c4040f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c4040f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3810c9a0 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c3e1ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c3e1ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56990/0 (socket says 172.21.15.87:56990) 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c3e1ab700 1 -- 172.21.15.87:0/2829439858 learned_addr learned my addr 172.21.15.87:0/2829439858 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c3e1ab700 1 -- 172.21.15.87:0/2829439858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c38198450 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c3e1ab700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c340056d0 tx=0x7f6c340057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c34026070 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c381986e0 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c34004370 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c340039b0 con 0x7f6c381036d0 2024-04-22T13:23:03.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.691+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c38198b40 con 0x7f6c381036d0 2024-04-22T13:23:03.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.692+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6c34029070 con 0x7f6c381036d0 2024-04-22T13:23:03.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.692+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3806b110 con 0x7f6c381036d0 2024-04-22T13:23:03.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.692+0000 7f6c2f7fe700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 0x7f6c2403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:03.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.692+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c3401e070 con 0x7f6c381036d0 2024-04-22T13:23:03.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.692+0000 7f6c3d9aa700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 0x7f6c2403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:03.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.693+0000 7f6c3d9aa700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 0x7f6c2403bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c28006fd0 tx=0x7f6c28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:03.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.697+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c34013070 con 0x7f6c381036d0 2024-04-22T13:23:03.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:03.899+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}) -- 0x7f6c381992e0 con 0x7f6c24039650 2024-04-22T13:23:03.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:03 smithi087 ceph-mon[27999]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: Deploying cephadm binary to smithi132 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:05.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:05.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:05 smithi087 ceph-mon[27999]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:07.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.958+0000 7f6c2f7fe700 1 -- 172.21.15.87:0/2829439858 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f6c381992e0 con 0x7f6c24039650 2024-04-22T13:23:07.955 INFO:teuthology.orchestra.run.smithi087.stdout:Added host 'smithi132' with addr '172.21.15.132' 2024-04-22T13:23:07.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 msgr2=0x7f6c2403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:07.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 0x7f6c2403bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6c28006fd0 tx=0x7f6c28006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 msgr2=0x7f6c38197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c340056d0 tx=0x7f6c340057b0 comp rx=0 tx=0).stop 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 shutdown_connections 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6c24039650 0x7f6c2403bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 --2- 172.21.15.87:0/2829439858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c381036d0 0x7f6c38197f10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.960+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 >> 172.21.15.87:0/2829439858 conn(0x7f6c380fce60 msgr2=0x7f6c38104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.961+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 shutdown_connections 2024-04-22T13:23:07.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:07.961+0000 7f6c4040f700 1 -- 172.21.15.87:0/2829439858 wait complete. 2024-04-22T13:23:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:07 smithi087 ceph-mon[27999]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:08.490 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T13:23:08.720 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:09.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:10.115 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:09 smithi087 ceph-mon[27999]: Added host smithi132 2024-04-22T13:23:10.115 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:09 smithi087 ceph-mon[27999]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:10.115 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:10.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 -- 172.21.15.87:0/649976841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 msgr2=0x7f61bc1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:10.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 --2- 172.21.15.87:0/649976841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc1019d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009910 tx=0x7f61b0009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:10.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 -- 172.21.15.87:0/649976841 shutdown_connections 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 --2- 172.21.15.87:0/649976841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc1019d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 -- 172.21.15.87:0/649976841 >> 172.21.15.87:0/649976841 conn(0x7f61bc076ea0 msgr2=0x7f61bc0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 -- 172.21.15.87:0/649976841 shutdown_connections 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.356+0000 7f61c309a700 1 -- 172.21.15.87:0/649976841 wait complete. 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c309a700 1 Processor -- start 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c309a700 1 -- start start 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c309a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c309a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc10b820 con 0x7f61bc1015f0 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c0e36700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c0e36700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56604/0 (socket says 172.21.15.87:56604) 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c0e36700 1 -- 172.21.15.87:0/1829450432 learned_addr learned my addr 172.21.15.87:0/1829450432 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c0e36700 1 -- 172.21.15.87:0/1829450432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc199290 con 0x7f61bc1015f0 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.357+0000 7f61c0e36700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f61b00056d0 tx=0x7f61b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:10.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.358+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b0026070 con 0x7f61bc1015f0 2024-04-22T13:23:10.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.358+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b000bba0 con 0x7f61bc1015f0 2024-04-22T13:23:10.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.358+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc199580 con 0x7f61bc1015f0 2024-04-22T13:23:10.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.358+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61b0003920 con 0x7f61bc1015f0 2024-04-22T13:23:10.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.358+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc199a40 con 0x7f61bc1015f0 2024-04-22T13:23:10.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f61b0029070 con 0x7f61bc1015f0 2024-04-22T13:23:10.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61b9ffb700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 0x7f61ac03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:10.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f61b001e070 con 0x7f61bc1015f0 2024-04-22T13:23:10.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61bbfff700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 0x7f61ac03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:10.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61bc102a80 con 0x7f61bc1015f0 2024-04-22T13:23:10.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.359+0000 7f61bbfff700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 0x7f61ac03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:10.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.364+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61b0030070 con 0x7f61bc1015f0 2024-04-22T13:23:10.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.578+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61bc0640a0 con 0x7f61ac039650 2024-04-22T13:23:10.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.579+0000 7f61b9ffb700 1 -- 172.21.15.87:0/1829450432 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f61bc0640a0 con 0x7f61ac039650 2024-04-22T13:23:10.575 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:23:10.576 INFO:teuthology.orchestra.run.smithi087.stdout:[{"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 msgr2=0x7f61ac03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 0x7f61ac03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61a8006fd0 tx=0x7f61a8006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 msgr2=0x7f61bc198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f61b00056d0 tx=0x7f61b00057b0 comp rx=0 tx=0).stop 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 shutdown_connections 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f61ac039650 0x7f61ac03bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 --2- 172.21.15.87:0/1829450432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61bc1015f0 0x7f61bc198d50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 >> 172.21.15.87:0/1829450432 conn(0x7f61bc076ea0 msgr2=0x7f61bc102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:10.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 shutdown_connections 2024-04-22T13:23:10.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:10.581+0000 7f61c309a700 1 -- 172.21.15.87:0/1829450432 wait complete. 2024-04-22T13:23:11.242 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T13:23:11.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T13:23:11.476 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:11.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:11.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:11 smithi087 ceph-mon[27999]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:11.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:11 smithi087 ceph-mon[27999]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:13.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 -- 172.21.15.87:0/4215395410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 msgr2=0x7ff1dc1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/4215395410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc1088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009910 tx=0x7ff1cc009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 -- 172.21.15.87:0/4215395410 shutdown_connections 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/4215395410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc1088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 -- 172.21.15.87:0/4215395410 >> 172.21.15.87:0/4215395410 conn(0x7ff1dc0fdf80 msgr2=0x7ff1dc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 -- 172.21.15.87:0/4215395410 shutdown_connections 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.016+0000 7ff1e1e68700 1 -- 172.21.15.87:0/4215395410 wait complete. 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1e1e68700 1 Processor -- start 2024-04-22T13:23:13.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1e1e68700 1 -- start start 2024-04-22T13:23:13.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1e1e68700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:13.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1e1e68700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc10c940 con 0x7ff1dc1084e0 2024-04-22T13:23:13.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:13.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56614/0 (socket says 172.21.15.87:56614) 2024-04-22T13:23:13.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.017+0000 7ff1db7fe700 1 -- 172.21.15.87:0/1328077680 learned_addr learned my addr 172.21.15.87:0/1328077680 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1db7fe700 1 -- 172.21.15.87:0/1328077680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc076c30 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1db7fe700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0056d0 tx=0x7ff1cc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc026070 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1cc00bc00 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc0752e0 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc003920 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.018+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc0757a0 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff1cc029070 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1dc103b90 con 0x7ff1dc1084e0 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1d8ff9700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 0x7ff1c403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff1cc01e070 con 0x7ff1dc1084e0 2024-04-22T13:23:13.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1daffd700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 0x7ff1c403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:13.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.019+0000 7ff1daffd700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 0x7ff1c403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0006fd0 tx=0x7ff1d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:13.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.024+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1cc013070 con 0x7ff1dc1084e0 2024-04-22T13:23:13.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.225+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff1dc076270 con 0x7ff1dc1084e0 2024-04-22T13:23:13.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.730+0000 7ff1d8ff9700 1 -- 172.21.15.87:0/1328077680 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7ff1cc004390 con 0x7ff1dc1084e0 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 msgr2=0x7ff1c403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 0x7ff1c403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0006fd0 tx=0x7ff1d0006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 msgr2=0x7ff1dc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0056d0 tx=0x7ff1cc0057b0 comp rx=0 tx=0).stop 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 shutdown_connections 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff1c4039650 0x7ff1c403bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 --2- 172.21.15.87:0/1328077680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1dc1084e0 0x7ff1dc074d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 >> 172.21.15.87:0/1328077680 conn(0x7ff1dc0fdf80 msgr2=0x7ff1dc10a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 shutdown_connections 2024-04-22T13:23:13.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:13.733+0000 7ff1e1e68700 1 -- 172.21.15.87:0/1328077680 wait complete. 2024-04-22T13:23:13.731 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-04-22T13:23:13.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:13 smithi087 ceph-mon[27999]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:13.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:13 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T13:23:14.441 INFO:tasks.cephadm:Adding mon.a on smithi087 2024-04-22T13:23:14.441 INFO:tasks.cephadm:Adding mon.b on smithi120 2024-04-22T13:23:14.441 INFO:tasks.cephadm:Adding mon.c on smithi132 2024-04-22T13:23:14.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c' 2024-04-22T13:23:14.682 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:14.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:15.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:15 smithi087 ceph-mon[27999]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:15.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:15 smithi087 ceph-mon[27999]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:15.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:15 smithi087 ceph-mon[27999]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:15.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:15 smithi087 ceph-mon[27999]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:16.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.275+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1041336021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 msgr2=0x7f5a78101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:16.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.275+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1041336021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009910 tx=0x7f5a68009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:16.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1041336021 shutdown_connections 2024-04-22T13:23:16.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1041336021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:16.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1041336021 >> 172.21.15.132:0/1041336021 conn(0x7f5a780fce20 msgr2=0x7f5a780ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1041336021 shutdown_connections 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1041336021 wait complete. 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 Processor -- start 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- start start 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.276+0000 7f5a7e9da700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7810b830 con 0x7f5a781015b0 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a77fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a77fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.132:44604/0 (socket says 172.21.15.132:44604) 2024-04-22T13:23:16.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a77fff700 1 -- 172.21.15.132:0/1799869831 learned_addr learned my addr 172.21.15.132:0/1799869831 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a77fff700 1 -- 172.21.15.132:0/1799869831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a78199210 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a77fff700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a68006010 tx=0x7f5a68005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a68026070 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a78199500 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6800bba0 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a68003910 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.277+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a781999c0 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.278+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5a68029070 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.278+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a78102a90 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.278+0000 7f5a757fa700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 0x7f5a6003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.278+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5a6801e070 con 0x7f5a781015b0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.279+0000 7f5a777fe700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 0x7f5a6003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:16.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.279+0000 7f5a777fe700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 0x7f5a6003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:16.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.284+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a68013070 con 0x7f5a781015b0 2024-04-22T13:23:16.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.493+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}) -- 0x7f5a7806af00 con 0x7f5a60039650 2024-04-22T13:23:16.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.500+0000 7f5a757fa700 1 -- 172.21.15.132:0/1799869831 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5a7806af00 con 0x7f5a60039650 2024-04-22T13:23:16.497 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mon update... 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 msgr2=0x7f5a6003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 0x7f5a6003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 msgr2=0x7f5a78198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a68006010 tx=0x7f5a68005780 comp rx=0 tx=0).stop 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 shutdown_connections 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5a60039650 0x7f5a6003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 --2- 172.21.15.132:0/1799869831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a781015b0 0x7f5a78198cd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 >> 172.21.15.132:0/1799869831 conn(0x7f5a780fce20 msgr2=0x7f5a78102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 shutdown_connections 2024-04-22T13:23:16.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:16.502+0000 7f5a7e9da700 1 -- 172.21.15.132:0/1799869831 wait complete. 2024-04-22T13:23:17.155 DEBUG:teuthology.orchestra.run.smithi120:mon.b> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T13:23:17.158 DEBUG:teuthology.orchestra.run.smithi132:mon.c> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T13:23:17.160 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T13:23:17.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: Saving service mon spec with placement smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c;count:3 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: Deploying daemon mon.c on smithi132 2024-04-22T13:23:17.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:17 smithi087 ceph-mon[27999]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:18.249 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:19.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:19 smithi087 ceph-mon[27999]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 -- 172.21.15.132:0/1245024907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 msgr2=0x7f20c4107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/1245024907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4107610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00b0a0 tx=0x7f20bc00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 -- 172.21.15.132:0/1245024907 shutdown_connections 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/1245024907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4107610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 -- 172.21.15.132:0/1245024907 >> 172.21.15.132:0/1245024907 conn(0x7f20c4076cf0 msgr2=0x7f20c4077100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 -- 172.21.15.132:0/1245024907 shutdown_connections 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.206+0000 7f20c8d9e700 1 -- 172.21.15.132:0/1245024907 wait complete. 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.207+0000 7f20c8d9e700 1 Processor -- start 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.207+0000 7f20c8d9e700 1 -- start start 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.207+0000 7f20c8d9e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:21.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.207+0000 7f20c8d9e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f20c410b670 con 0x7f20c4107230 2024-04-22T13:23:21.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:21.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.132:58102/0 (socket says 172.21.15.132:58102) 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c37fe700 1 -- 172.21.15.132:0/2825468449 learned_addr learned my addr 172.21.15.132:0/2825468449 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c37fe700 1 -- 172.21.15.132:0/2825468449 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c4198fe0 con 0x7f20c4107230 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c37fe700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20bc006130 tx=0x7f20bc0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20bc021020 con 0x7f20c4107230 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c4199270 con 0x7f20c4107230 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20bc0079b0 con 0x7f20c4107230 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20bc017040 con 0x7f20c4107230 2024-04-22T13:23:21.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.208+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c4199730 con 0x7f20c4107230 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.209+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f20bc024020 con 0x7f20c4107230 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.210+0000 7f20c0ff9700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 0x7f20b003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.210+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f20bc00f050 con 0x7f20c4107230 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.210+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c41028e0 con 0x7f20c4107230 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.210+0000 7f20c2ffd700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 0x7f20b003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:21.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.210+0000 7f20c2ffd700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 0x7f20b003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f20ac006fd0 tx=0x7f20ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:21.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.215+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20bc02b020 con 0x7f20c4107230 2024-04-22T13:23:21.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.516+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f20c40640a0 con 0x7f20c4107230 2024-04-22T13:23:21.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.516+0000 7f20c0ff9700 1 -- 172.21.15.132:0/2825468449 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f20bc01b7b0 con 0x7f20c4107230 2024-04-22T13:23:21.513 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:23:21.513 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","modified":"2024-04-22T13:21:40.526906Z","created":"2024-04-22T13:21:40.526906Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T13:23:21.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.518+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 msgr2=0x7f20b003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:21.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.518+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 0x7f20b003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f20ac006fd0 tx=0x7f20ac006e40 comp rx=0 tx=0).stop 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.518+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 msgr2=0x7f20c4198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20bc006130 tx=0x7f20bc0097d0 comp rx=0 tx=0).stop 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 shutdown_connections 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20b0039650 0x7f20b003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 --2- 172.21.15.132:0/2825468449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20c4107230 0x7f20c4198aa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 >> 172.21.15.132:0/2825468449 conn(0x7f20c4076cf0 msgr2=0x7f20c40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 shutdown_connections 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:21.519+0000 7f20c8d9e700 1 -- 172.21.15.132:0/2825468449 wait complete. 2024-04-22T13:23:21.516 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-04-22T13:23:21.598 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 podman[31496]: 2024-04-22 13:23:21.57840363 +0000 UTC m=+0.364342697 container start 16324ba06b2f1ce725bce1a8a1e6786c8150cdfad5673f37c5ef347d5bdf200e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T13:23:21.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T13:23:21.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-22T13:23:21.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pidfile_write: ignore empty --pid-file 2024-04-22T13:23:21.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: load: jerasure load: lrc 2024-04-22T13:23:21.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: RocksDB version: 7.9.2 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Git sha 0 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: DB SUMMARY 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: DB Session ID: TS01CL5646KRGFSDOE4K 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: CURRENT file: CURRENT 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.error_if_exists: 0 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.create_if_missing: 0 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.paranoid_checks: 1 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T13:23:21.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.env: 0x555de0bc9cc0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.info_log: 0x555de24cbbe0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.statistics: (nil) 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.use_fsync: 0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_log_file_size: 0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_fallocate: 1 2024-04-22T13:23:21.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.use_direct_reads: 0 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.db_log_dir: 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.wal_dir: 2024-04-22T13:23:21.967 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.write_buffer_manager: 0x555de245f0e0 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T13:23:21.969 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.unordered_write: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.row_cache: None 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.wal_filter: None 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.two_write_queues: 0 2024-04-22T13:23:21.970 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.wal_compression: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.atomic_flush: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.log_readahead_size: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T13:23:21.971 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_background_jobs: 2 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_background_compactions: -1 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_subcompactions: 1 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T13:23:21.972 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_open_files: -1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_background_flushes: -1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Compression algorithms supported: 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kZSTD supported: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kXpressCompression supported: 0 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kZlibCompression supported: 1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kSnappyCompression supported: 1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kLZ4Compression supported: 1 2024-04-22T13:23:21.973 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: kBZip2Compression supported: 0 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.merge_operator: 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_filter: None 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555de24cbdc0) 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-04-22T13:23:21.974 INFO:journalctl@ceph.mon.c.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_type: 0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: data_block_index_type: 0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_shortening: 1 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: checksum: 4 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: no_block_cache: 0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache: 0x555de24911f0 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_options: 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: capacity : 536870912 2024-04-22T13:23:21.975 INFO:journalctl@ceph.mon.c.smithi132.stdout: num_shard_bits : 4 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: strict_capacity_limit : 0 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_compressed: (nil) 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: persistent_cache: (nil) 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_size: 4096 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_size_deviation: 10 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_restart_interval: 16 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_block_restart_interval: 1 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: metadata_block_size: 4096 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: partition_filters: 0 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: use_delta_encoding: 1 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: filter_policy: bloomfilter 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: whole_key_filtering: 1 2024-04-22T13:23:21.976 INFO:journalctl@ceph.mon.c.smithi132.stdout: verify_compression: 0 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: format_version: 5 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: enable_index_compression: 1 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_align: 0 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: max_auto_readahead_size: 262144 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: prepopulate_block_cache: 0 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: initial_auto_readahead_size: 8192 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression: NoCompression 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T13:23:21.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.num_levels: 7 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T13:23:21.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T13:23:21.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T13:23:21.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.inplace_update_support: 0 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T13:23:21.981 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.bloom_locality: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.max_successive_merges: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.ttl: 2592000 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T13:23:21.982 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enable_blob_files: false 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.min_blob_size: 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 50491dbf-1c65-4c22-90a5-3e47e9bb2223 2024-04-22T13:23:21.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792201618888, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792201619299, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713792201, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "50491dbf-1c65-4c22-90a5-3e47e9bb2223", "db_session_id": "TS01CL5646KRGFSDOE4K", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792201619378, "job": 1, "event": "recovery_finished"} 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555de25a6000 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: DB pointer 0x555de2590000 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** DB Stats ** 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:23:21.984 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** Compaction Stats [default] ** 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** Compaction Stats [default] ** 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T13:23:21.985 INFO:journalctl@ceph.mon.c.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Block cache BinnedLRUCache@0x555de24911f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T13:23:21.986 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: using public_addr v2:172.21.15.132:0/0 -> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: starting mon.c rank -1 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(???) e0 preinit fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).mds e1 new map 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: e1 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: legacy client fscid: -1 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout: No filesystems configured 2024-04-22T13:23:21.987 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:21.988 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Deploying cephadm binary to smithi132 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.989 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Added host smithi132 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T13:23:21.990 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Saving service mon spec with placement smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c;count:3 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: Deploying daemon mon.c on smithi132 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:21.993 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 ceph-mon[31531]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T13:23:21.993 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 bash[31496]: 16324ba06b2f1ce725bce1a8a1e6786c8150cdfad5673f37c5ef347d5bdf200e 2024-04-22T13:23:21.993 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:21 smithi132 systemd[1]: Started Ceph mon.c for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:23:23.247 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T13:23:23.247 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T13:23:23.490 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: mon.a calling monitor election 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: mon.c calling monitor election 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: monmap epoch 2 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: last_changed 2024-04-22T13:23:21.642695+0000 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: min_mon_release 19 (squid) 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: election_strategy: 1 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: fsmap 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:26.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: mgrmap e13: x(active, since 49s) 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: overall HEALTH_OK 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:26.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: mon.a calling monitor election 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: mon.c calling monitor election 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: monmap epoch 2 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: last_changed 2024-04-22T13:23:21.642695+0000 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: min_mon_release 19 (squid) 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: election_strategy: 1 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: fsmap 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: mgrmap e13: x(active, since 49s) 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: overall HEALTH_OK 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:26.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:26.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:27 smithi087 ceph-mon[27999]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:27.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:27 smithi087 ceph-mon[27999]: Deploying daemon mon.b on smithi120 2024-04-22T13:23:27.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:27.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.998+0000 7ff697806700 1 -- 172.21.15.132:0/3128198842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff67c0035c0 msgr2=0x7ff67c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.998+0000 7ff697806700 1 --2- 172.21.15.132:0/3128198842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff67c0035c0 0x7ff67c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff690078b70 tx=0x7ff68000feb0 comp rx=0 tx=0).stop 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.999+0000 7ff697806700 1 -- 172.21.15.132:0/3128198842 shutdown_connections 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.999+0000 7ff697806700 1 --2- 172.21.15.132:0/3128198842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff67c0035c0 0x7ff67c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.999+0000 7ff697806700 1 -- 172.21.15.132:0/3128198842 >> 172.21.15.132:0/3128198842 conn(0x7ff690077860 msgr2=0x7ff690077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.999+0000 7ff697806700 1 -- 172.21.15.132:0/3128198842 shutdown_connections 2024-04-22T13:23:27.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:27.999+0000 7ff697806700 1 -- 172.21.15.132:0/3128198842 wait complete. 2024-04-22T13:23:27.997 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:27 smithi132 ceph-mon[31531]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:27.997 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:27 smithi132 ceph-mon[31531]: Deploying daemon mon.b on smithi120 2024-04-22T13:23:27.997 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:27.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 Processor -- start 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 -- start start 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 0x7ff69019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff69010a4c0 con 0x7ff6901984b0 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff697806700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff69010a340 con 0x7ff67c0035c0 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59936/0 (socket says 172.21.15.132:59936) 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 -- 172.21.15.132:0/3409949126 learned_addr learned my addr 172.21.15.132:0/3409949126 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 -- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 msgr2=0x7ff690197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 -- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 msgr2=0x7ff690197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T13:23:28.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff694da1700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 0x7ff69019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.000+0000 7ff6955a2700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff694da1700 1 -- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 msgr2=0x7ff690197f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff694da1700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff694da1700 1 -- 172.21.15.132:0/3409949126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff690198af0 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff694da1700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 0x7ff69019c840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00b4d0 tx=0x7ff68c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff68c011630 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff68c011c80 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69019cd80 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff68c010da0 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.001+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69019d360 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.002+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff68c018020 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.002+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69019d670 con 0x7ff6901984b0 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.002+0000 7ff6867fc700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 0x7ff67403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:28.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.002+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff68c04c220 con 0x7ff6901984b0 2024-04-22T13:23:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.003+0000 7ff6955a2700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 0x7ff67403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.003+0000 7ff6955a2700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 0x7ff67403b950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff68000fa60 tx=0x7ff680005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:28.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.008+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff69019d670 con 0x7ff6901984b0 2024-04-22T13:23:28.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.292+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff6901992d0 con 0x7ff6901984b0 2024-04-22T13:23:28.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.292+0000 7ff6867fc700 1 -- 172.21.15.132:0/3409949126 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7ff68c01f020 con 0x7ff6901984b0 2024-04-22T13:23:28.290 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:23:28.291 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":2,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","modified":"2024-04-22T13:23:21.642695Z","created":"2024-04-22T13:21:40.526906Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T13:23:28.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 msgr2=0x7ff67403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:28.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 0x7ff67403b950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff68000fa60 tx=0x7ff680005a90 comp rx=0 tx=0).stop 2024-04-22T13:23:28.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 msgr2=0x7ff69019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 0x7ff69019c840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00b4d0 tx=0x7ff68c00b890 comp rx=0 tx=0).stop 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 shutdown_connections 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff674039490 0x7ff67403b950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff67c0035c0 0x7ff690197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.294+0000 7ff697806700 1 --2- 172.21.15.132:0/3409949126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6901984b0 0x7ff69019c840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.295+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 >> 172.21.15.132:0/3409949126 conn(0x7ff690077860 msgr2=0x7ff6900ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.295+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 shutdown_connections 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:28.295+0000 7ff697806700 1 -- 172.21.15.132:0/3409949126 wait complete. 2024-04-22T13:23:28.294 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 2 2024-04-22T13:23:28.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:28 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/3409949126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:28.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:28 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/3409949126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:29.911 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T13:23:29.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T13:23:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:29 smithi087 ceph-mon[27999]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:23:30.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:29 smithi132 ceph-mon[31531]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.372 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 podman[31082]: 2024-04-22 13:23:30.268278977 +0000 UTC m=+0.127182265 container create 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 podman[31082]: 2024-04-22 13:23:30.462567163 +0000 UTC m=+0.321470460 container init 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 podman[31082]: 2024-04-22 13:23:30.465665449 +0000 UTC m=+0.324568745 container start 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pidfile_write: ignore empty --pid-file 2024-04-22T13:23:30.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: load: jerasure load: lrc 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: RocksDB version: 7.9.2 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Git sha 0 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: DB SUMMARY 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: DB Session ID: T9OH28SJ4DC90MIDFLCY 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: CURRENT file: CURRENT 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.error_if_exists: 0 2024-04-22T13:23:30.672 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.create_if_missing: 0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.paranoid_checks: 1 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.env: 0x55cbc8695cc0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.info_log: 0x55cbc90a3be0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.statistics: (nil) 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.use_fsync: 0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_log_file_size: 0 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T13:23:30.673 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_fallocate: 1 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.use_direct_reads: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.db_log_dir: 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.wal_dir: 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T13:23:30.674 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.write_buffer_manager: 0x55cbc90370e0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T13:23:30.675 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.unordered_write: 0 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.row_cache: None 2024-04-22T13:23:30.676 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.wal_filter: None 2024-04-22T13:23:30.677 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.two_write_queues: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.wal_compression: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.atomic_flush: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.log_readahead_size: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T13:23:30.678 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_background_jobs: 2 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_background_compactions: -1 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_subcompactions: 1 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_open_files: -1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_background_flushes: -1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Compression algorithms supported: 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kZSTD supported: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kXpressCompression supported: 0 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kZlibCompression supported: 1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kSnappyCompression supported: 1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kLZ4Compression supported: 1 2024-04-22T13:23:30.680 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: kBZip2Compression supported: 0 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.merge_operator: 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_filter: None 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cbc90a3dc0) 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout: cache_index_and_filter_blocks: 1 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T13:23:30.681 INFO:journalctl@ceph.mon.b.smithi120.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: pin_top_level_index_and_filter: 1 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_type: 0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: data_block_index_type: 0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_shortening: 1 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: checksum: 4 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: no_block_cache: 0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache: 0x55cbc90691f0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_name: BinnedLRUCache 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_options: 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: capacity : 536870912 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: num_shard_bits : 4 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: strict_capacity_limit : 0 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: high_pri_pool_ratio: 0.000 2024-04-22T13:23:30.682 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_compressed: (nil) 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: persistent_cache: (nil) 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_size: 4096 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_size_deviation: 10 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_restart_interval: 16 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_block_restart_interval: 1 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: metadata_block_size: 4096 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: partition_filters: 0 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: use_delta_encoding: 1 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: filter_policy: bloomfilter 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: whole_key_filtering: 1 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: verify_compression: 0 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: read_amp_bytes_per_bit: 0 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: format_version: 5 2024-04-22T13:23:30.683 INFO:journalctl@ceph.mon.b.smithi120.stdout: enable_index_compression: 1 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_align: 0 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout: max_auto_readahead_size: 262144 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout: prepopulate_block_cache: 0 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout: initial_auto_readahead_size: 8192 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression: NoCompression 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.num_levels: 7 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T13:23:30.684 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T13:23:30.685 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T13:23:30.686 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T13:23:30.687 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.inplace_update_support: 0 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T13:23:30.688 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.bloom_locality: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.max_successive_merges: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.ttl: 2592000 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enable_blob_files: false 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.min_blob_size: 0 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T13:23:30.689 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a0fb61ae-4f57-45c4-bc9b-ce2b35d8c410 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792210495561, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792210495975, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713792210, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a0fb61ae-4f57-45c4-bc9b-ce2b35d8c410", "db_session_id": "T9OH28SJ4DC90MIDFLCY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713792210496054, "job": 1, "event": "recovery_finished"} 2024-04-22T13:23:30.690 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cbc917e000 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: DB pointer 0x55cbc9168000 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** DB Stats ** 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.691 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** Compaction Stats [default] ** 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** Compaction Stats [default] ** 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T13:23:30.692 INFO:journalctl@ceph.mon.b.smithi120.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: Block cache BinnedLRUCache@0x55cbc90691f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: using public_addr v2:172.21.15.120:0/0 -> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-04-22T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: starting mon.b rank -1 at public addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] at bind addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(???) e0 preinit fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).mds e1 new map 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: e1 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: legacy client fscid: -1 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout: No filesystems configured 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T13:23:30.694 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Deploying cephadm binary to smithi132 2024-04-22T13:23:30.695 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:30.696 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Added host smithi132 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:23:30.697 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1328077680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:30.698 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Saving service mon spec with placement smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c;count:3 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:30.699 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Deploying daemon mon.c on smithi132 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.a calling monitor election 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.c calling monitor election 2024-04-22T13:23:30.700 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: monmap epoch 2 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: last_changed 2024-04-22T13:23:21.642695+0000 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: min_mon_release 19 (squid) 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: election_strategy: 1 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:30.701 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: fsmap 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mgrmap e13: x(active, since 49s) 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: overall HEALTH_OK 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: Deploying daemon mon.b on smithi120 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/3409949126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:30.702 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:30.703 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 ceph-mon[31101]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T13:23:30.703 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 bash[31082]: 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 2024-04-22T13:23:30.703 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:30 smithi120 systemd[1]: Started Ceph mon.b for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:23:35.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:35.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: mon.a calling monitor election 2024-04-22T13:23:35.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: mon.c calling monitor election 2024-04-22T13:23:35.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: monmap epoch 3 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:35.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: min_mon_release 19 (squid) 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: election_strategy: 1 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: fsmap 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: mgrmap e13: x(active, since 58s) 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: overall HEALTH_OK 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:35.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: mon.a calling monitor election 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: mon.c calling monitor election 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: monmap epoch 3 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: min_mon_release 19 (squid) 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: election_strategy: 1 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:35.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: fsmap 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: mgrmap e13: x(active, since 58s) 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: overall HEALTH_OK 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:35.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.024+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/4035991387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f3c0057d0 msgr2=0x7f6f3c007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.024+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/4035991387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f3c0057d0 0x7f6f3c007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f54101380 tx=0x7f6f4400fe60 comp rx=0 tx=0).stop 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.024+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/4035991387 shutdown_connections 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.024+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/4035991387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f3c0057d0 0x7f6f3c007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.024+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/4035991387 >> 172.21.15.132:0/4035991387 conn(0x7f6f540fcf80 msgr2=0x7f6f540ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.025+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/4035991387 shutdown_connections 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.025+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/4035991387 wait complete. 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.025+0000 7f6f5a2f3700 1 Processor -- start 2024-04-22T13:23:36.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.025+0000 7f6f5a2f3700 1 -- start start 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.025+0000 7f6f5a2f3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f5a2f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f5a2f3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f5a2f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6f541082f0 con 0x7f6f54196460 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f5a2f3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5406bb40 con 0x7f6f3c0057d0 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f5a2f3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f54108470 con 0x7f6f5419ad30 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.132:58982/0 (socket says 172.21.15.132:58982) 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 -- 172.21.15.132:0/2707609421 learned_addr learned my addr 172.21.15.132:0/2707609421 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f53fff700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:36.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f58890700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f53fff700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 msgr2=0x7f6f54195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f53fff700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 msgr2=0x7f6f54195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f53fff700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f53fff700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f58890700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 msgr2=0x7f6f5419b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f58890700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 msgr2=0x7f6f5419b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f58890700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 msgr2=0x7f6f5419b1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f58890700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 msgr2=0x7f6f54195f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 -- 172.21.15.132:0/2707609421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f541a2aa0 con 0x7f6f54196460 2024-04-22T13:23:36.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f537fe700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800b780 tx=0x7f6f4800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:36.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.026+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004230 con 0x7f6f54196460 2024-04-22T13:23:36.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.027+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f480043d0 con 0x7f6f54196460 2024-04-22T13:23:36.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.027+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f541a2cd0 con 0x7f6f54196460 2024-04-22T13:23:36.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.027+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011bd0 con 0x7f6f54196460 2024-04-22T13:23:36.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.027+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f541a3200 con 0x7f6f54196460 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.028+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6f4801a400 con 0x7f6f54196460 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.028+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f540640a0 con 0x7f6f54196460 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.028+0000 7f6f517fa700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 0x7f6f4003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.028+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6f4804c4b0 con 0x7f6f54196460 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.028+0000 7f6f53fff700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 0x7f6f4003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:36.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.029+0000 7f6f53fff700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 0x7f6f4003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400fa10 tx=0x7f6f44005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:36.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.033+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f48014160 con 0x7f6f54196460 2024-04-22T13:23:36.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.333+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6f5402e140 con 0x7f6f54196460 2024-04-22T13:23:36.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.334+0000 7f6f517fa700 1 -- 172.21.15.132:0/2707609421 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f6f48018510 con 0x7f6f54196460 2024-04-22T13:23:36.331 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-22T13:23:36.331 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":3,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","modified":"2024-04-22T13:23:30.518726Z","created":"2024-04-22T13:21:40.526906Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T13:23:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.336+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 msgr2=0x7f6f4003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.336+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 0x7f6f4003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f4400fa10 tx=0x7f6f44005a90 comp rx=0 tx=0).stop 2024-04-22T13:23:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.336+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 msgr2=0x7f6f5419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.336+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800b780 tx=0x7f6f4800bb40 comp rx=0 tx=0).stop 2024-04-22T13:23:36.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.336+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 shutdown_connections 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6f40039ae0 0x7f6f4003bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f5419ad30 0x7f6f5419b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f3c0057d0 0x7f6f54195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 --2- 172.21.15.132:0/2707609421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f54196460 0x7f6f5419a7f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 >> 172.21.15.132:0/2707609421 conn(0x7f6f540fcf80 msgr2=0x7f6f54105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 shutdown_connections 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:36.337+0000 7f6f5a2f3700 1 -- 172.21.15.132:0/2707609421 wait complete. 2024-04-22T13:23:36.334 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 3 2024-04-22T13:23:36.637 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:36.637 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:36.637 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:36 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/2707609421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:36.637 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:36.939 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T13:23:36.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T13:23:36.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:36.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:36.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:36 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/2707609421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:36.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: mon.a calling monitor election 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: mon.c calling monitor election 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: monmap epoch 3 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: min_mon_release 19 (squid) 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: election_strategy: 1 2024-04-22T13:23:37.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: fsmap 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: mgrmap e13: x(active, since 58s) 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: overall HEALTH_OK 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/2707609421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T13:23:37.923 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:38.342 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mon.b calling monitor election 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mon.b calling monitor election 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mon.a calling monitor election 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mon.c calling monitor election 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: monmap epoch 3 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: min_mon_release 19 (squid) 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: election_strategy: 1 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: fsmap 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:38.343 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: mgrmap e13: x(active, since 60s) 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: overall HEALTH_OK 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:38.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:38 smithi087 ceph-mon[27999]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mon.b calling monitor election 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mon.b calling monitor election 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mon.a calling monitor election 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mon.c calling monitor election 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: monmap epoch 3 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: min_mon_release 19 (squid) 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: election_strategy: 1 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:38.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: fsmap 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: mgrmap e13: x(active, since 60s) 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: overall HEALTH_OK 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:38.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:38 smithi120 ceph-mon[31101]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:38.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mon.b calling monitor election 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mon.b calling monitor election 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mon.a calling monitor election 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mon.c calling monitor election 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: monmap epoch 3 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: last_changed 2024-04-22T13:23:30.518726+0000 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: created 2024-04-22T13:21:40.526906+0000 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: min_mon_release 19 (squid) 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: election_strategy: 1 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: fsmap 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T13:23:38.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: mgrmap e13: x(active, since 60s) 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: overall HEALTH_OK 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:23:38.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:38 smithi132 ceph-mon[31531]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 -- 172.21.15.87:0/3082014152 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d8108430 msgr2=0x7fc6d8108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/3082014152 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d8108430 0x7fc6d8108810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009910 tx=0x7fc6c8009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 -- 172.21.15.87:0/3082014152 shutdown_connections 2024-04-22T13:23:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/3082014152 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d8108430 0x7fc6d8108810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:38.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 -- 172.21.15.87:0/3082014152 >> 172.21.15.87:0/3082014152 conn(0x7fc6d80788d0 msgr2=0x7fc6d8078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:38.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 -- 172.21.15.87:0/3082014152 shutdown_connections 2024-04-22T13:23:38.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.928+0000 7fc6ddce9700 1 -- 172.21.15.87:0/3082014152 wait complete. 2024-04-22T13:23:38.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 Processor -- start 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 -- start start 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 0x7fc6d8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 0x7fc6d819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d810c920 con 0x7fc6d819ce80 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d810c7a0 con 0x7fc6d81985b0 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.929+0000 7fc6ddce9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d810caa0 con 0x7fc6d8108430 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51664/0 (socket says 172.21.15.87:51664) 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 -- 172.21.15.87:0/663982239 learned_addr learned my addr 172.21.15.87:0/663982239 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d77fe700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 0x7fc6d8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:38.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d6ffd700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 0x7fc6d819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 -- 172.21.15.87:0/663982239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 msgr2=0x7fc6d8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 0x7fc6d8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 -- 172.21.15.87:0/663982239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 msgr2=0x7fc6d819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 0x7fc6d819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 -- 172.21.15.87:0/663982239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d81a4bf0 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d77fe700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 0x7fc6d8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d6ffd700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 0x7fc6d819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d7fff700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000ba50 tx=0x7fc6d000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0019070 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d81a4e20 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6d0014440 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0007590 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.930+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81a52f0 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.931+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc6d001c070 con 0x7fc6d819ce80 2024-04-22T13:23:38.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.931+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d80640a0 con 0x7fc6d819ce80 2024-04-22T13:23:38.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.931+0000 7fc6d4ff9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 0x7fc6c003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:38.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.931+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc6d004bf10 con 0x7fc6d819ce80 2024-04-22T13:23:38.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.932+0000 7fc6d77fe700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 0x7fc6c003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:38.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.932+0000 7fc6d77fe700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 0x7fc6c003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8006010 tx=0x7fc6c801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:38.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:38.938+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6d0014910 con 0x7fc6d819ce80 2024-04-22T13:23:39.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.144+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc6d81a5750 con 0x7fc6d819ce80 2024-04-22T13:23:39.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.144+0000 7fc6d4ff9700 1 -- 172.21.15.87:0/663982239 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fc6d000e090 con 0x7fc6d819ce80 2024-04-22T13:23:39.142 INFO:teuthology.orchestra.run.smithi087.stdout:# minimal ceph.conf for 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:39.143 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-04-22T13:23:39.143 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:23:39.143 INFO:teuthology.orchestra.run.smithi087.stdout: mon_host = [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-04-22T13:23:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 msgr2=0x7fc6c003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 0x7fc6c003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8006010 tx=0x7fc6c801a040 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 msgr2=0x7fc6d819d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000ba50 tx=0x7fc6d000be10 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 shutdown_connections 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6c0039ae0 0x7fc6c003bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6d819ce80 0x7fc6d819d330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6d8108430 0x7fc6d8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 --2- 172.21.15.87:0/663982239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6d81985b0 0x7fc6d819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 >> 172.21.15.87:0/663982239 conn(0x7fc6d80788d0 msgr2=0x7fc6d80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 shutdown_connections 2024-04-22T13:23:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:39.147+0000 7fc6ddce9700 1 -- 172.21.15.87:0/663982239 wait complete. 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:39.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:39.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: Reconfiguring daemon mon.a on smithi087 2024-04-22T13:23:39.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:39.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:39 smithi087 ceph-mon[27999]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: Reconfiguring daemon mon.a on smithi087 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:39.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:39 smithi120 ceph-mon[31101]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: Reconfiguring daemon mon.a on smithi087 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:23:39.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:39 smithi132 ceph-mon[31531]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:39.875 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T13:23:39.876 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:23:39.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T13:23:39.941 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:23:39.941 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:40.004 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:23:40.005 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T13:23:40.036 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:23:40.036 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:40.102 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:23:40.102 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T13:23:40.134 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:23:40.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:23:40.149 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/663982239' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:40.202 INFO:tasks.cephadm:Adding mgr.x on smithi087 2024-04-22T13:23:40.202 INFO:tasks.cephadm:Adding mgr.y on smithi120 2024-04-22T13:23:40.202 INFO:tasks.cephadm:Adding mgr.z on smithi132 2024-04-22T13:23:40.202 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi087=x;smithi120=y;smithi132=z' 2024-04-22T13:23:40.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/663982239' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:40.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/663982239' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:40.480 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:23:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:41.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:41.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:41.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:41.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:41.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:41.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:41.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:41.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:41.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:41.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:41.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:42.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- 172.21.15.132:0/2285166176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c1014a0 msgr2=0x7f956c1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:42.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 --2- 172.21.15.132:0/2285166176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c1014a0 0x7f956c1018c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f955c009910 tx=0x7f955c009c20 comp rx=0 tx=0).stop 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- 172.21.15.132:0/2285166176 shutdown_connections 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 --2- 172.21.15.132:0/2285166176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c1014a0 0x7f956c1018c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- 172.21.15.132:0/2285166176 >> 172.21.15.132:0/2285166176 conn(0x7f956c0fca40 msgr2=0x7f956c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- 172.21.15.132:0/2285166176 shutdown_connections 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- 172.21.15.132:0/2285166176 wait complete. 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 Processor -- start 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.090+0000 7f9573632700 1 -- start start 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 0x7f956c19c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f956c1975d0 0x7f956c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1073e0 con 0x7f956c197150 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f956c107260 con 0x7f956c1975d0 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f956c107560 con 0x7f956c1014a0 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:42.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57480/0 (socket says 172.21.15.132:57480) 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 -- 172.21.15.132:0/2902522092 learned_addr learned my addr 172.21.15.132:0/2902522092 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9570bcd700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 0x7f956c19c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 -- 172.21.15.132:0/2902522092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f956c1975d0 msgr2=0x7f956c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f956c1975d0 0x7f956c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 -- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 msgr2=0x7f956c19c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 0x7f956c19c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 -- 172.21.15.132:0/2902522092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c19f030 con 0x7f956c1014a0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9570bcd700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 0x7f956c19c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95713ce700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f955c005e00 tx=0x7f955c004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c004300 con 0x7f956c1014a0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c19f2c0 con 0x7f956c1014a0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c0044a0 con 0x7f956c1014a0 2024-04-22T13:23:42.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.091+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c003b50 con 0x7f956c1014a0 2024-04-22T13:23:42.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.092+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c19f870 con 0x7f956c1014a0 2024-04-22T13:23:42.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.093+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f956c0640a0 con 0x7f956c1014a0 2024-04-22T13:23:42.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.093+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f955c003cf0 con 0x7f956c1014a0 2024-04-22T13:23:42.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.093+0000 7f95627fc700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 0x7f955803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:42.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.093+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f955c04dae0 con 0x7f956c1014a0 2024-04-22T13:23:42.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.093+0000 7f9570bcd700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 0x7f955803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:42.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.094+0000 7f9570bcd700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 0x7f955803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9568009ec0 tx=0x7f9568009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:42.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.098+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f955c013070 con 0x7f956c1014a0 2024-04-22T13:23:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:42 smithi132 ceph-mon[31531]: Reconfiguring mon.b (monmap changed)... 2024-04-22T13:23:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:42 smithi132 ceph-mon[31531]: Reconfiguring daemon mon.b on smithi120 2024-04-22T13:23:42.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.309+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi087=x;smithi120=y;smithi132=z", "target": ["mon-mgr", ""]}) -- 0x7f956c062f10 con 0x7f9558039ae0 2024-04-22T13:23:42.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.316+0000 7f95627fc700 1 -- 172.21.15.132:0/2902522092 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f956c062f10 con 0x7f9558039ae0 2024-04-22T13:23:42.312 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mgr update... 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 msgr2=0x7f955803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 0x7f955803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9568009ec0 tx=0x7f9568009210 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 msgr2=0x7f956c196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f955c005e00 tx=0x7f955c004f80 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 shutdown_connections 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9558039ae0 0x7f955803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f956c1975d0 0x7f956c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f956c1014a0 0x7f956c196c10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 --2- 172.21.15.132:0/2902522092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f956c197150 0x7f956c19c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 >> 172.21.15.132:0/2902522092 conn(0x7f956c0fca40 msgr2=0x7f956c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 shutdown_connections 2024-04-22T13:23:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:23:42.318+0000 7f9573632700 1 -- 172.21.15.132:0/2902522092 wait complete. 2024-04-22T13:23:42.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:42 smithi120 ceph-mon[31101]: Reconfiguring mon.b (monmap changed)... 2024-04-22T13:23:42.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:42 smithi120 ceph-mon[31101]: Reconfiguring daemon mon.b on smithi120 2024-04-22T13:23:42.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:42 smithi087 ceph-mon[27999]: Reconfiguring mon.b (monmap changed)... 2024-04-22T13:23:42.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:42 smithi087 ceph-mon[27999]: Reconfiguring daemon mon.b on smithi120 2024-04-22T13:23:42.923 DEBUG:teuthology.orchestra.run.smithi120:mgr.y> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T13:23:42.926 DEBUG:teuthology.orchestra.run.smithi132:mgr.z> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.z.service 2024-04-22T13:23:42.928 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T13:23:42.928 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:23:42.928 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T13:23:42.947 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T13:23:42.947 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-22 13:22:47.073078847 +0000 2024-04-22T13:23:43.005 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-22 13:17:02.455019742 +0000 2024-04-22T13:23:43.006 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-22 13:17:02.456019719 +0000 2024-04-22T13:23:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-22 13:17:02.455019742 +0000 2024-04-22T13:23:43.006 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T13:23:43.073 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-22T13:23:43.073 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-22T13:23:43.073 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000231837 s, 2.2 MB/s 2024-04-22T13:23:43.074 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T13:23:43.133 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-22 13:22:47.100078238 +0000 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-22 13:17:02.808011411 +0000 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-22 13:17:02.808011411 +0000 2024-04-22T13:23:43.191 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-22 13:17:02.808011411 +0000 2024-04-22T13:23:43.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T13:23:43.257 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-22T13:23:43.257 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-22T13:23:43.257 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000246915 s, 2.1 MB/s 2024-04-22T13:23:43.258 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T13:23:43.319 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-04-22T13:23:43.375 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-22 13:22:47.129077583 +0000 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-22 13:17:03.150003339 +0000 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-22 13:17:03.150003339 +0000 2024-04-22T13:23:43.376 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-22 13:17:03.150003339 +0000 2024-04-22T13:23:43.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T13:23:43.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi087=x;smithi120=y;smithi132=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:43.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: Saving service mgr spec with placement smithi087=x;smithi120=y;smithi132=z;count:3 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: Reconfiguring mon.c (monmap changed)... 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: Reconfiguring daemon mon.c on smithi132 2024-04-22T13:23:43.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:43 smithi087 ceph-mon[27999]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:43.439 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-22T13:23:43.440 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-22T13:23:43.440 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000193205 s, 2.7 MB/s 2024-04-22T13:23:43.441 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T13:23:43.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi087=x;smithi120=y;smithi132=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:43.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: Saving service mgr spec with placement smithi087=x;smithi120=y;smithi132=z;count:3 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: Reconfiguring mon.c (monmap changed)... 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: Reconfiguring daemon mon.c on smithi132 2024-04-22T13:23:43.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:43 smithi132 ceph-mon[31531]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:43.498 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-04-22T13:23:43.553 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-22 13:22:47.153077042 +0000 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-22 13:17:03.486995386 +0000 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-22 13:17:03.486995386 +0000 2024-04-22T13:23:43.554 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-22 13:17:03.486995386 +0000 2024-04-22T13:23:43.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T13:23:43.619 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-04-22T13:23:43.619 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-04-22T13:23:43.619 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00023974 s, 2.1 MB/s 2024-04-22T13:23:43.619 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi087=x;smithi120=y;smithi132=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: Saving service mgr spec with placement smithi087=x;smithi120=y;smithi132=z;count:3 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: Reconfiguring mon.c (monmap changed)... 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: Reconfiguring daemon mon.c on smithi132 2024-04-22T13:23:43.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:43 smithi120 ceph-mon[31101]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:43.675 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-04-22T13:23:43.675 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T13:23:43.694 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T13:23:43.694 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-04-22 13:23:29.948328238 +0000 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-04-22 13:18:05.838609579 +0000 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-04-22 13:18:05.838609579 +0000 2024-04-22T13:23:43.752 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-04-22 13:18:05.838609579 +0000 2024-04-22T13:23:43.753 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T13:23:43.818 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-04-22T13:23:43.818 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-04-22T13:23:43.819 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000299922 s, 1.7 MB/s 2024-04-22T13:23:43.820 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T13:23:43.877 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-04-22T13:23:43.934 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T13:23:43.934 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-04-22 13:23:29.948328238 +0000 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-04-22 13:18:06.161601491 +0000 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-04-22 13:18:06.161601491 +0000 2024-04-22T13:23:43.935 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-04-22 13:18:06.161601491 +0000 2024-04-22T13:23:43.935 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T13:23:43.998 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-04-22T13:23:43.999 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-04-22T13:23:43.999 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000226865 s, 2.3 MB/s 2024-04-22T13:23:44.000 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T13:23:44.057 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-04-22 13:23:29.949328214 +0000 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-04-22 13:18:06.520592502 +0000 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-04-22 13:18:06.520592502 +0000 2024-04-22T13:23:44.113 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-04-22 13:18:06.520592502 +0000 2024-04-22T13:23:44.114 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T13:23:44.177 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-04-22T13:23:44.177 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-04-22T13:23:44.177 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000198101 s, 2.6 MB/s 2024-04-22T13:23:44.178 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T13:23:44.235 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-04-22T13:23:44.289 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T13:23:44.289 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:44.289 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T13:23:44.289 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:44.289 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:44.290 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-04-22 13:23:29.949328214 +0000 2024-04-22T13:23:44.290 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-04-22 13:18:06.857584064 +0000 2024-04-22T13:23:44.290 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-04-22 13:18:06.857584064 +0000 2024-04-22T13:23:44.290 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-04-22 13:18:06.857584064 +0000 2024-04-22T13:23:44.290 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T13:23:44.351 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-04-22T13:23:44.352 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-04-22T13:23:44.352 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000200004 s, 2.6 MB/s 2024-04-22T13:23:44.353 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T13:23:44.409 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-22T13:23:44.409 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T13:23:44.426 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T13:23:44.426 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-22 13:23:20.300141941 +0000 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-22 13:17:29.164044612 +0000 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-22 13:17:29.164044612 +0000 2024-04-22T13:23:44.483 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-04-22 13:17:29.164044612 +0000 2024-04-22T13:23:44.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T13:23:44.550 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-22T13:23:44.550 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-22T13:23:44.550 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000283466 s, 1.8 MB/s 2024-04-22T13:23:44.552 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T13:23:44.609 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-04-22T13:23:44.665 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T13:23:44.665 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:44.665 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-22T13:23:44.665 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:44.666 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:44.666 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-22 13:23:20.300141941 +0000 2024-04-22T13:23:44.666 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-22 13:17:29.502035141 +0000 2024-04-22T13:23:44.666 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-22 13:17:29.502035141 +0000 2024-04-22T13:23:44.666 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-04-22 13:17:29.502035141 +0000 2024-04-22T13:23:44.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T13:23:44.728 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-22T13:23:44.728 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-22T13:23:44.728 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000252747 s, 2.0 MB/s 2024-04-22T13:23:44.729 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T13:23:44.785 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-22 13:23:20.301141915 +0000 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-22 13:17:29.840025671 +0000 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-22 13:17:29.840025671 +0000 2024-04-22T13:23:44.840 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-04-22 13:17:29.840025671 +0000 2024-04-22T13:23:44.841 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T13:23:44.902 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-22T13:23:44.903 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-22T13:23:44.903 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000195394 s, 2.6 MB/s 2024-04-22T13:23:44.904 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T13:23:44.960 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T13:23:45.015 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-22 13:23:20.301141915 +0000 2024-04-22T13:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-22 13:17:30.188015921 +0000 2024-04-22T13:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-22 13:17:30.188015921 +0000 2024-04-22T13:23:45.016 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-04-22 13:17:30.188015921 +0000 2024-04-22T13:23:45.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T13:23:45.077 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-22T13:23:45.078 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-22T13:23:45.078 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000178385 s, 2.9 MB/s 2024-04-22T13:23:45.079 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T13:23:45.134 INFO:tasks.cephadm:Deploying osd.0 on smithi087 with /dev/vg_nvme/lv_4... 2024-04-22T13:23:45.134 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T13:23:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: Deploying daemon mgr.z on smithi132 2024-04-22T13:23:45.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:45 smithi132 ceph-mon[31531]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:45.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: Deploying daemon mgr.z on smithi132 2024-04-22T13:23:45.422 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:45 smithi120 ceph-mon[31101]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: Deploying daemon mgr.z on smithi132 2024-04-22T13:23:45.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:45 smithi087 ceph-mon[27999]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:46.714 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:46 smithi132 systemd[1]: Starting Ceph mgr.z for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:23:47.024 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:46 smithi132 podman[33367]: 2024-04-22 13:23:46.672595386 +0000 UTC m=+0.017759592 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:23:47.024 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:46 smithi132 podman[33367]: 2024-04-22 13:23:46.789796562 +0000 UTC m=+0.134960744 container create 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T13:23:47.024 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 podman[33367]: 2024-04-22 13:23:47.00052526 +0000 UTC m=+0.345689454 container init 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T13:23:47.024 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 podman[33367]: 2024-04-22 13:23:47.003024854 +0000 UTC m=+0.348189037 container start 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T13:23:47.024 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mgr[33386]: -- 172.21.15.132:0/1777682802 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5644cdec85a0 con 0x5644cde9cc00 2024-04-22T13:23:47.333 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T13:23:47.334 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T13:23:47.334 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-04-22T13:23:47.334 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-04-22T13:23:47.334 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011542 s, 908 MB/s 2024-04-22T13:23:47.334 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-04-22T13:23:47.337 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 bash[33367]: 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e 2024-04-22T13:23:47.337 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 systemd[1]: Started Ceph mgr.z for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:23:47.337 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:47.156+0000 7f9c01bd9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:23:47.337 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:47.341+0000 7f9c01bd9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:23:47.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:47.442+0000 7f9c01bd9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:23:47.925 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:47.925 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:47.925 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:47.925 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:47.925 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:47.926 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:47.926 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:47.926 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:47.952 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_4 2024-04-22T13:23:47.999 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:48.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:48.155 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:47.749+0000 7f9c01bd9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:23:48.155 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:47.832+0000 7f9c01bd9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:23:48.155 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:48.156 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:48.187 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:23:48.964 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:48 smithi087 ceph-mon[27999]: Deploying daemon mgr.y on smithi120 2024-04-22T13:23:48.999 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:48 smithi120 ceph-mon[31101]: Deploying daemon mgr.y on smithi120 2024-04-22T13:23:49.066 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:48 smithi132 ceph-mon[31531]: Deploying daemon mgr.y on smithi120 2024-04-22T13:23:49.393 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:49.069+0000 7f9c01bd9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:23:49.393 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:49.241+0000 7f9c01bd9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:23:49.700 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:49.419+0000 7f9c01bd9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:23:49.700 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:49.703+0000 7f9c01bd9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- 172.21.15.87:0/548811897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8102ad0 msgr2=0x7f1ad8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- 172.21.15.87:0/548811897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8102ef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40099c0 tx=0x7f1ad4009cd0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- 172.21.15.87:0/548811897 shutdown_connections 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- 172.21.15.87:0/548811897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8104670 0x7f1ad810af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- 172.21.15.87:0/548811897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8102ef0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- 172.21.15.87:0/548811897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- 172.21.15.87:0/548811897 >> 172.21.15.87:0/548811897 conn(0x7f1ad80fe090 msgr2=0x7f1ad81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- 172.21.15.87:0/548811897 shutdown_connections 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- 172.21.15.87:0/548811897 wait complete. 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 Processor -- start 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- start start 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 0x7f1ad81a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810e5b0 con 0x7f1ad8104670 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810e430 con 0x7f1ad8102ad0 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.744+0000 7f1ae0770700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810e730 con 0x7f1ad8103cd0 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1aded0d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 0x7f1ad81a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1aded0d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 0x7f1ad81a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48072/0 (socket says 172.21.15.87:48072) 2024-04-22T13:23:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1aded0d700 1 -- 172.21.15.87:0/3068146030 learned_addr learned my addr 172.21.15.87:0/3068146030 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1ade50c700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 -- 172.21.15.87:0/3068146030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8102ad0 msgr2=0x7f1ad8198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 -- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 msgr2=0x7f1ad81a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 0x7f1ad81a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 -- 172.21.15.87:0/3068146030 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad81a0860 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1addd0b700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8199150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9f0 tx=0x7f1ac800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8004220 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad81a0b50 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac80043c0 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8005020 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.745+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad81a1090 con 0x7f1ad8103cd0 2024-04-22T13:23:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.746+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1ac80051c0 con 0x7f1ad8103cd0 2024-04-22T13:23:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.746+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad8069730 con 0x7f1ad8103cd0 2024-04-22T13:23:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.746+0000 7f1acf7fe700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 0x7f1ac403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:23:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.747+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1ac804ca10 con 0x7f1ad8103cd0 2024-04-22T13:23:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.747+0000 7f1ade50c700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 0x7f1ac403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:23:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.747+0000 7f1ade50c700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 0x7f1ac403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ad819a0f0 tx=0x7f1ad4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:23:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.756+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ac800a250 con 0x7f1ad8103cd0 2024-04-22T13:23:49.762 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:49 smithi120 ceph-mon[31101]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-mon[31531]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:49.964 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:49.807+0000 7f9c01bd9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:23:49.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:49.972+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1ad8062e80 con 0x7f1ac4039890 2024-04-22T13:23:49.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:49 smithi087 ceph-mon[27999]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:50.059 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:49.954+0000 7fa711645200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:23:50.060 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:50.062+0000 7fa711645200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:23:50.463 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:50.015+0000 7f9c01bd9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:23:50.463 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:50.216+0000 7f9c01bd9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:23:50.670 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:50.382+0000 7fa711645200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:23:50.670 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:50.476+0000 7fa711645200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:50.857+0000 7f9c01bd9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:23:50.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:51.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:23:51.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:23:51.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:23:51.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.587 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:51.294+0000 7f9c01bd9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:23:51.587 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:51.392+0000 7f9c01bd9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:23:51.587 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:51.590+0000 7f9c01bd9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.929 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:51.930 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:51.930 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.930 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:51.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.045 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:51.709+0000 7fa711645200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:51.879+0000 7fa711645200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:52.048+0000 7fa711645200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.046 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:52.047 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:52.441 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.115+0000 7f9c01bd9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:23:52.441 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.206+0000 7f9c01bd9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:23:52.623 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:52.334+0000 7fa711645200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:23:52.624 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:52.425+0000 7fa711645200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:23:52.624 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:52.626+0000 7fa711645200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:23:52.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.444+0000 7f9c01bd9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:23:52.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.576+0000 7f9c01bd9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:23:52.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.668+0000 7f9c01bd9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:23:52.920 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:52.817+0000 7fa711645200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: Reconfiguring daemon mgr.x on smithi087 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]: dispatch 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]': finished 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T13:23:52.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:52 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: Reconfiguring daemon mgr.x on smithi087 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]: dispatch 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]': finished 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:52.783+0000 7f9c01bd9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:23:53.087 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:53.090+0000 7f9c01bd9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: Reconfiguring daemon mgr.x on smithi087 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]: dispatch 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2547294829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48d1248e-5bb0-4829-a035-88f1d8b049d4"}]': finished 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T13:23:53.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:23:53.670 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:53.408+0000 7fa711645200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:23:53.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:53 smithi132 ceph-mon[31531]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:53.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:53 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/87288885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:23:53.964 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:53.687+0000 7f9c01bd9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:23:53.964 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:23:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:23:53.899+0000 7f9c01bd9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:23:54.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:53 smithi120 ceph-mon[31101]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:54.113 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:53 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/87288885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:23:54.113 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:53.823+0000 7fa711645200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:23:54.113 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:53.917+0000 7fa711645200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:23:54.113 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:54.116+0000 7fa711645200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:23:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:53 smithi087 ceph-mon[27999]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:53 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/87288885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:23:54.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:54.818+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f1ac80055b0 con 0x7f1ad8103cd0 2024-04-22T13:23:54.920 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:54.656+0000 7fa711645200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:54.747+0000 7fa711645200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-mon[31101]: Standby manager daemon z started 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:23:54.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:54 smithi132 ceph-mon[31531]: Standby manager daemon z started 2024-04-22T13:23:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:54 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:23:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:54 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:54 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:23:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:54 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:54 smithi087 ceph-mon[27999]: Standby manager daemon z started 2024-04-22T13:23:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:54 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:23:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:54 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:54 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:23:55.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:54 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/2901529020' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:55.324 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:54.987+0000 7fa711645200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:23:55.324 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:55.118+0000 7fa711645200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:23:55.324 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:55.212+0000 7fa711645200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:23:55.324 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:55.327+0000 7fa711645200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:23:55.920 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:55.626+0000 7fa711645200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:23:55.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-mon[31101]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:55.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-mon[31101]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-22T13:23:55.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:55 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:23:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:55 smithi132 ceph-mon[31531]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:55 smithi132 ceph-mon[31531]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-22T13:23:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:55 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:23:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:55 smithi087 ceph-mon[27999]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:55 smithi087 ceph-mon[27999]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-22T13:23:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:55 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:23:56.670 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:56.222+0000 7fa711645200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:23:56.671 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:23:56.433+0000 7fa711645200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:23:56.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:23:56.829+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mon.1 v2:172.21.15.132:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f1ac8003680 con 0x7f1ad8103cd0 2024-04-22T13:23:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-mon[31101]: Standby manager daemon y started 2024-04-22T13:23:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:23:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:23:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:56 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:57.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:56 smithi087 ceph-mon[27999]: Standby manager daemon y started 2024-04-22T13:23:57.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:56 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:23:57.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:56 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:57.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:56 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:23:57.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:56 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:56 smithi132 ceph-mon[31531]: Standby manager daemon y started 2024-04-22T13:23:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:56 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:23:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:56 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:23:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:56 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:23:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:56 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/117241552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:23:58.148 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:57 smithi087 ceph-mon[27999]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:58.148 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:57 smithi087 ceph-mon[27999]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-04-22T13:23:58.148 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:23:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:57 smithi120 ceph-mon[31101]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:57 smithi120 ceph-mon[31101]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-04-22T13:23:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:23:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:57 smithi132 ceph-mon[31531]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:57 smithi132 ceph-mon[31531]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-04-22T13:23:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:23:59 smithi087 ceph-mon[27999]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:59.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:59.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:59.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:59.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:23:59 smithi132 ceph-mon[31531]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:23:59.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:23:59.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:23:59.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:23:59.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:23:59.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:23:59 smithi120 ceph-mon[31101]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:00.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T13:24:00.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T13:24:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T13:24:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:01 smithi087 ceph-mon[27999]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:01 smithi087 ceph-mon[27999]: Deploying daemon osd.0 on smithi087 2024-04-22T13:24:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:01 smithi120 ceph-mon[31101]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:01 smithi120 ceph-mon[31101]: Deploying daemon osd.0 on smithi087 2024-04-22T13:24:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:01 smithi132 ceph-mon[31531]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:01 smithi132 ceph-mon[31531]: Deploying daemon osd.0 on smithi087 2024-04-22T13:24:03.921 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:03 smithi087 ceph-mon[27999]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:03 smithi120 ceph-mon[31101]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:03 smithi132 ceph-mon[31531]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:05.942 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:05 smithi087 ceph-mon[27999]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:05 smithi120 ceph-mon[31101]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:05 smithi132 ceph-mon[31531]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:07 smithi132 ceph-mon[31531]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:07.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:07.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:07 smithi087 ceph-mon[27999]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:07.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:07.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:07.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:07 smithi120 ceph-mon[31101]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:09.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.477+0000 7f1acf7fe700 1 -- 172.21.15.87:0/3068146030 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1ad8062e80 con 0x7f1ac4039890 2024-04-22T13:24:09.475 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 0 on host 'smithi087' 2024-04-22T13:24:09.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 msgr2=0x7f1ac403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:09.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 0x7f1ac403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ad819a0f0 tx=0x7f1ad4005fb0 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 msgr2=0x7f1ad8199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8199150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9f0 tx=0x7f1ac800ed00 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 shutdown_connections 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1ac4039890 0x7f1ac403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ad8104670 0x7f1ad81a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad8102ad0 0x7f1ad8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 --2- 172.21.15.87:0/3068146030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad8103cd0 0x7f1ad8199150 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 >> 172.21.15.87:0/3068146030 conn(0x7f1ad80fe090 msgr2=0x7f1ad8107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 shutdown_connections 2024-04-22T13:24:09.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:24:09.480+0000 7f1ae0770700 1 -- 172.21.15.87:0/3068146030 wait complete. 2024-04-22T13:24:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:09 smithi087 ceph-mon[27999]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.121 DEBUG:teuthology.orchestra.run.smithi087:osd.0> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T13:24:10.123 INFO:tasks.cephadm:Deploying osd.1 on smithi120 with /dev/vg_nvme/lv_4... 2024-04-22T13:24:10.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T13:24:10.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:09 smithi120 ceph-mon[31101]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:10.149 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.150 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:09 smithi132 ceph-mon[31531]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.367 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:24:10.807 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:10 smithi120 ceph-mon[31101]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T13:24:10.808 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:10 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.808 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:10 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.976 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:10 smithi087 ceph-mon[27999]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T13:24:10.976 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:10 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:10.976 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:10 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:11.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:10 smithi132 ceph-mon[31531]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T13:24:11.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:10 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:11.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:10 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:11 smithi120 ceph-mon[31101]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:11 smithi120 ceph-mon[31101]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T13:24:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:11 smithi120 ceph-mon[31101]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T13:24:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:11 smithi120 ceph-mon[31101]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:24:11.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:12.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:11 smithi087 ceph-mon[27999]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:12.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:11 smithi087 ceph-mon[27999]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T13:24:12.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:11 smithi087 ceph-mon[27999]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T13:24:12.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:11 smithi087 ceph-mon[27999]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:24:12.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:11 smithi132 ceph-mon[31531]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:11 smithi132 ceph-mon[31531]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T13:24:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:11 smithi132 ceph-mon[31531]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T13:24:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:11 smithi132 ceph-mon[31531]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:24:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111051 s, 944 MB/s 2024-04-22T13:24:12.236 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-04-22T13:24:12.371 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0[40130]: 2024-04-22T13:24:12.210+0000 7f8b5a40a700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T13:24:12.371 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0[40130]: 2024-04-22T13:24:12.214+0000 7f8b521f5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:24:12.819 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:12 smithi120 ceph-mon[31101]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:24:12.819 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:12 smithi120 ceph-mon[31101]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T13:24:12.819 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:12 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:12.819 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:12 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:12.819 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:12 smithi120 ceph-mon[31101]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' 2024-04-22T13:24:12.846 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_4 2024-04-22T13:24:13.014 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-mon[27999]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:24:13.014 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-mon[27999]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T13:24:13.014 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:13.014 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:13.014 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:12 smithi087 ceph-mon[27999]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' 2024-04-22T13:24:13.120 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:24:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:12 smithi132 ceph-mon[31531]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:24:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:12 smithi132 ceph-mon[31531]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T13:24:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:12 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:12 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:12 smithi132 ceph-mon[31531]: from='osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709]' entity='osd.0' 2024-04-22T13:24:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:24:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:24:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:13.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] boot 2024-04-22T13:24:13.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T13:24:13.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] boot 2024-04-22T13:24:14.153 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T13:24:14.154 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: osd.0 [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] boot 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T13:24:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.647+0000 7fda48c13700 1 -- 172.21.15.120:0/3088967237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda44103550 msgr2=0x7fda44109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.647+0000 7fda48c13700 1 --2- 172.21.15.120:0/3088967237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda44103550 0x7fda44109e00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00b0d0 tx=0x7fda3c00b3e0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 -- 172.21.15.120:0/3088967237 shutdown_connections 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 --2- 172.21.15.120:0/3088967237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda44103550 0x7fda44109e00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 --2- 172.21.15.120:0/3088967237 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda441019b0 0x7fda44101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 --2- 172.21.15.120:0/3088967237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44102bb0 0x7fda44103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 -- 172.21.15.120:0/3088967237 >> 172.21.15.120:0/3088967237 conn(0x7fda440fcf70 msgr2=0x7fda440ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 -- 172.21.15.120:0/3088967237 shutdown_connections 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 -- 172.21.15.120:0/3088967237 wait complete. 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 Processor -- start 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.648+0000 7fda48c13700 1 -- start start 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:14.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda44102bb0 0x7fda44103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:14.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 0x7fda441044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:14.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410d450 con 0x7fda44102bb0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410d2d0 con 0x7fda441019b0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda4410d5d0 con 0x7fda44103550 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44882/0 (socket says 172.21.15.120:44882) 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 -- 172.21.15.120:0/110870335 learned_addr learned my addr 172.21.15.120:0/110870335 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda42d9d700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 0x7fda441044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda41d9b700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda44102bb0 0x7fda44103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 -- 172.21.15.120:0/110870335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 msgr2=0x7fda441044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 0x7fda441044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 -- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda44102bb0 msgr2=0x7fda44103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda44102bb0 0x7fda44103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 -- 172.21.15.120:0/110870335 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda44104da0 con 0x7fda441019b0 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda42d9d700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 0x7fda441044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:24:14.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda4259c700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c970 tx=0x7fda3800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38004220 con 0x7fda441019b0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda441a2d20 con 0x7fda441019b0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380043c0 con 0x7fda441019b0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.649+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38003c20 con 0x7fda441019b0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.650+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda441a3180 con 0x7fda441019b0 2024-04-22T13:24:14.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.650+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda44069730 con 0x7fda441019b0 2024-04-22T13:24:14.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.651+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fda380076d0 con 0x7fda441019b0 2024-04-22T13:24:14.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.652+0000 7fda337fe700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 0x7fda2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:14.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.652+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7fda380ce440 con 0x7fda441019b0 2024-04-22T13:24:14.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.652+0000 7fda41d9b700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 0x7fda2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:14.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.653+0000 7fda41d9b700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 0x7fda2c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda34009c60 tx=0x7fda34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:24:14.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.657+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda38014030 con 0x7fda441019b0 2024-04-22T13:24:14.859 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:14.862+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fda440ff140 con 0x7fda2c0a3bb0 2024-04-22T13:24:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi087 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 17088M 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi087 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 17088M 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:15.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi087 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 17088M 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:16.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:16.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T13:24:16.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:16.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:16.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:16.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T13:24:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:16.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:17.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:17.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/805073098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:17.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:17.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]': finished 2024-04-22T13:24:17.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T13:24:17.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:17 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/805073098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]': finished 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T13:24:18.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:17 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/805073098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]: dispatch 2024-04-22T13:24:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b96dba19-35ae-4b99-9a8d-43a623679f0e"}]': finished 2024-04-22T13:24:18.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T13:24:18.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:17 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:19.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:18 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/2196228346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:19.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:18 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/2196228346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:19.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:18 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/2196228346' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:20.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:19 smithi120 ceph-mon[31101]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:20.178 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:19 smithi087 ceph-mon[27999]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:19 smithi132 ceph-mon[31531]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:22.155 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:21 smithi120 ceph-mon[31101]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:21 smithi132 ceph-mon[31531]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:21 smithi087 ceph-mon[27999]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:23 smithi120 ceph-mon[31101]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:23 smithi132 ceph-mon[31531]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:23 smithi087 ceph-mon[27999]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:25.106 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:24 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T13:24:25.106 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:24 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:24 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T13:24:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:24 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:25.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T13:24:25.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:26.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:25 smithi120 ceph-mon[31101]: Deploying daemon osd.1 on smithi120 2024-04-22T13:24:26.020 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:25 smithi120 ceph-mon[31101]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:25 smithi132 ceph-mon[31531]: Deploying daemon osd.1 on smithi120 2024-04-22T13:24:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:25 smithi132 ceph-mon[31531]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:25 smithi087 ceph-mon[27999]: Deploying daemon osd.1 on smithi120 2024-04-22T13:24:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:25 smithi087 ceph-mon[27999]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:28.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:27 smithi120 ceph-mon[31101]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:28.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:27 smithi132 ceph-mon[31531]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:28.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:27 smithi087 ceph-mon[27999]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:29 smithi120 ceph-mon[31101]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:29 smithi132 ceph-mon[31531]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:29 smithi087 ceph-mon[27999]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:32.033 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:31 smithi120 ceph-mon[31101]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:32.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:31 smithi132 ceph-mon[31531]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:32.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:31 smithi087 ceph-mon[27999]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:33.020 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 1 on host 'smithi120' 2024-04-22T13:24:33.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.023+0000 7fda337fe700 1 -- 172.21.15.120:0/110870335 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fda440ff140 con 0x7fda2c0a3bb0 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.021 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:32 smithi120 ceph-mon[31101]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.025+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 msgr2=0x7fda2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.025+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 0x7fda2c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda34009c60 tx=0x7fda34009260 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 msgr2=0x7fda441058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c970 tx=0x7fda3800cc80 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 shutdown_connections 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda2c0a3bb0 0x7fda2c0a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda44103550 0x7fda441044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda441019b0 0x7fda441058f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 --2- 172.21.15.120:0/110870335 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda44102bb0 0x7fda44103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 >> 172.21.15.120:0/110870335 conn(0x7fda440fcf70 msgr2=0x7fda440fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 shutdown_connections 2024-04-22T13:24:33.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:24:33.026+0000 7fda48c13700 1 -- 172.21.15.120:0/110870335 wait complete. 2024-04-22T13:24:33.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:33.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:33.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:32 smithi132 ceph-mon[31531]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:33.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:33.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:33.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:33.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:32 smithi087 ceph-mon[27999]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:33.618 DEBUG:teuthology.orchestra.run.smithi120:osd.1> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T13:24:33.620 INFO:tasks.cephadm:Deploying osd.2 on smithi132 with /dev/vg_nvme/lv_4... 2024-04-22T13:24:33.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T13:24:33.866 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:24:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:34.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:33 smithi120 ceph-mon[31101]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:34.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:33 smithi120 ceph-mon[31101]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:34.221 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.221 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.221 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:34.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:33 smithi132 ceph-mon[31531]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:34.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:33 smithi132 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:33 smithi087 ceph-mon[27999]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:34.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:33 smithi087 ceph-mon[27999]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T13:24:35.190 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T13:24:35.191 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.191 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T13:24:35.191 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:35.191 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.191 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:34 smithi132 ceph-mon[31531]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:35.235 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T13:24:35.235 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.236 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T13:24:35.236 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:35.236 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.236 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-mon[31101]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:35.236 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:24:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1[35704]: 2024-04-22T13:24:34.997+0000 7fa45d2d6700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T13:24:35.236 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:24:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1[35704]: 2024-04-22T13:24:35.000+0000 7fa4590c9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:24:35.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T13:24:35.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: from='osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T13:24:35.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:35.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:24:35.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:34 smithi087 ceph-mon[27999]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T13:24:35.743 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T13:24:35.743 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T13:24:35.743 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-22T13:24:35.744 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-22T13:24:35.744 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112879 s, 929 MB/s 2024-04-22T13:24:35.744 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 2024-04-22T13:24:35.744 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-22T13:24:36.074 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:35 smithi132 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:24:36.075 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:35 smithi132 ceph-mon[31531]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T13:24:36.075 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.075 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:35 smithi120 ceph-mon[31101]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:24:36.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:35 smithi120 ceph-mon[31101]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T13:24:36.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-04-22T13:24:36.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:35 smithi087 ceph-mon[27999]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:24:36.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:35 smithi087 ceph-mon[27999]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T13:24:36.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:36.603 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:24:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:24:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] boot 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi120 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 17088M 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:37.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:36 smithi132 ceph-mon[31531]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:37.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:24:37.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:24:37.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] boot 2024-04-22T13:24:37.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi120 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 17088M 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:36 smithi120 ceph-mon[31101]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:37.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:24:37.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:24:37.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: osd.1 [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] boot 2024-04-22T13:24:37.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T13:24:37.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi120 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 17088M 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:37.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:36 smithi087 ceph-mon[27999]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:38.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 -- 172.21.15.132:0/1799747213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 msgr2=0x7faea41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 --2- 172.21.15.132:0/1799747213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea41020c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae940099c0 tx=0x7fae94009cd0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 -- 172.21.15.132:0/1799747213 shutdown_connections 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 --2- 172.21.15.132:0/1799747213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faea4102600 0x7faea4109ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 --2- 172.21.15.132:0/1799747213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faea4102f20 0x7faea4103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 --2- 172.21.15.132:0/1799747213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea41020c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 -- 172.21.15.132:0/1799747213 >> 172.21.15.132:0/1799747213 conn(0x7faea40fc6d0 msgr2=0x7faea40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 -- 172.21.15.132:0/1799747213 shutdown_connections 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.016+0000 7faeac640700 1 -- 172.21.15.132:0/1799747213 wait complete. 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 Processor -- start 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 -- start start 2024-04-22T13:24:38.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faea4102600 0x7faea41991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faea4102f20 0x7faea41a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d4d0 con 0x7faea4102600 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d350 con 0x7faea4102f20 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeac640700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d650 con 0x7faea4101c60 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:35482/0 (socket says 172.21.15.132:35482) 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 -- 172.21.15.132:0/461378625 learned_addr learned my addr 172.21.15.132:0/461378625 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 -- 172.21.15.132:0/461378625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faea4102f20 msgr2=0x7faea41a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faea4102f20 0x7faea41a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 -- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faea4102600 msgr2=0x7faea41991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faea4102600 0x7faea41991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.017+0000 7faeaa3dc700 1 -- 172.21.15.132:0/461378625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea41a08f0 con 0x7faea4101c60 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7faeaa3dc700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00ada0 tx=0x7fae9c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:24:38.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c017020 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9c00c980 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c004420 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea41a0be0 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.018+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea41a10d0 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.019+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea41a12a0 con 0x7faea4101c60 2024-04-22T13:24:38.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.020+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fae9c00cb30 con 0x7faea4101c60 2024-04-22T13:24:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.021+0000 7fae9b7fe700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:24:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.021+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fae9c0ce610 con 0x7faea4101c60 2024-04-22T13:24:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.021+0000 7faea9bdb700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:24:38.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.021+0000 7faea9bdb700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 0x7fae900a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae940098e0 tx=0x7fae94009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:24:38.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.025+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae9c021020 con 0x7faea4101c60 2024-04-22T13:24:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:38 smithi132 ceph-mon[31531]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T13:24:38.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:38.231+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7faea40fe880 con 0x7fae900a3bb0 2024-04-22T13:24:38.388 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:38 smithi087 ceph-mon[27999]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T13:24:38.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:38 smithi120 ceph-mon[31101]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T13:24:39.154 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:39 smithi132 ceph-mon[31531]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:39.155 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:39.155 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:39.155 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:39.155 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:39 smithi132 ceph-mon[31531]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:39 smithi120 ceph-mon[31101]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:39 smithi120 ceph-mon[31101]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:39 smithi087 ceph-mon[27999]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:24:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:24:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:24:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:39 smithi087 ceph-mon[27999]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/4071055001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]': finished 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/2830165801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/4071055001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]': finished 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:42.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/2830165801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/4071055001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]: dispatch 2024-04-22T13:24:42.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02678f49-713f-40a2-8cc6-426c50c594a1"}]': finished 2024-04-22T13:24:42.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T13:24:42.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:42.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/2830165801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:24:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:43 smithi120 ceph-mon[31101]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:43 smithi132 ceph-mon[31531]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:43 smithi087 ceph-mon[27999]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:46.069 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:45 smithi132 ceph-mon[31531]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:45 smithi120 ceph-mon[31101]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:45 smithi087 ceph-mon[27999]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:47.946 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:47 smithi132 ceph-mon[31531]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:47 smithi120 ceph-mon[31101]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:47 smithi087 ceph-mon[27999]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:48.891 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:48 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T13:24:48.891 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:48 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:49.071 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:48 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T13:24:49.072 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:48 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:49.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:48 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T13:24:49.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:48 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:49.795 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:49 smithi132 ceph-mon[31531]: Deploying daemon osd.2 on smithi132 2024-04-22T13:24:49.795 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:49 smithi132 ceph-mon[31531]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:49.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:49 smithi087 ceph-mon[27999]: Deploying daemon osd.2 on smithi132 2024-04-22T13:24:49.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:49 smithi087 ceph-mon[27999]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:49 smithi120 ceph-mon[31101]: Deploying daemon osd.2 on smithi132 2024-04-22T13:24:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:49 smithi120 ceph-mon[31101]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:51 smithi132 ceph-mon[31531]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:51 smithi120 ceph-mon[31101]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:52.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:51 smithi087 ceph-mon[27999]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:54.021 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:53 smithi132 ceph-mon[31531]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:54.021 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.021 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.021 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:53 smithi120 ceph-mon[31101]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:53 smithi087 ceph-mon[27999]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:55.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:55 smithi132 ceph-mon[31531]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:55 smithi120 ceph-mon[31101]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:55 smithi087 ceph-mon[27999]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:56.745 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 2 on host 'smithi132' 2024-04-22T13:24:56.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.748+0000 7fae9b7fe700 1 -- 172.21.15.132:0/461378625 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faea40fe880 con 0x7fae900a3bb0 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.750+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 msgr2=0x7fae900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 0x7fae900a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae940098e0 tx=0x7fae94009f10 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 msgr2=0x7faea4198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00ada0 tx=0x7fae9c00c330 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 shutdown_connections 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fae900a3bb0 0x7fae900a6070 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faea4102f20 0x7faea41a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faea4101c60 0x7faea4198c70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 --2- 172.21.15.132:0/461378625 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faea4102600 0x7faea41991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:24:56.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 >> 172.21.15.132:0/461378625 conn(0x7faea40fc6d0 msgr2=0x7faea40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:24:56.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 shutdown_connections 2024-04-22T13:24:56.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:24:56.751+0000 7faeac640700 1 -- 172.21.15.132:0/461378625 wait complete. 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.374 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:57.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:57.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:57 smithi132 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:57.436 DEBUG:teuthology.orchestra.run.smithi132:osd.2> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T13:24:57.438 INFO:tasks.cephadm:Deploying osd.3 on smithi087 with /dev/vg_nvme/lv_3... 2024-04-22T13:24:57.438 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T13:24:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:57.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:57 smithi120 ceph-mon[31101]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:57.685 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:24:57.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:24:57.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:57.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:57 smithi087 ceph-mon[27999]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T13:24:59.008 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:58 smithi132 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T13:24:59.008 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:58 smithi132 ceph-mon[31531]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T13:24:59.008 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:58 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:59.008 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:58 smithi132 ceph-mon[31531]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.008 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:58 smithi132 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:58 smithi087 ceph-mon[27999]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T13:24:59.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:58 smithi087 ceph-mon[27999]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T13:24:59.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:58 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:59.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:58 smithi087 ceph-mon[27999]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:58 smithi087 ceph-mon[27999]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:58 smithi120 ceph-mon[31101]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T13:24:59.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:58 smithi120 ceph-mon[31101]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T13:24:59.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:58 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:24:59.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:58 smithi120 ceph-mon[31101]: from='osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:58 smithi120 ceph-mon[31101]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:24:59.464 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2[36682]: 2024-04-22T13:24:59.175+0000 7effb9a34700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T13:24:59.464 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2[36682]: 2024-04-22T13:24:59.180+0000 7effb181f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135655 s, 773 MB/s 2024-04-22T13:24:59.652 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-04-22T13:25:00.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:00.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:25:00.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T13:25:00.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:24:59 smithi087 ceph-mon[27999]: from='osd.2 ' entity='osd.2' 2024-04-22T13:25:00.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:00.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:25:00.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T13:25:00.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.054 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.054 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:24:59 smithi132 ceph-mon[31531]: from='osd.2 ' entity='osd.2' 2024-04-22T13:25:00.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:00.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:25:00.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T13:25:00.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:24:59 smithi120 ceph-mon[31101]: from='osd.2 ' entity='osd.2' 2024-04-22T13:25:00.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_3 2024-04-22T13:25:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] boot 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:00.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:00.946 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:25:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:25:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] boot 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:01.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:25:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: osd.2 [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] boot 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:01.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:01 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi132 2024-04-22T13:25:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:01 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 17088M 2024-04-22T13:25:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:01 smithi087 ceph-mon[27999]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:01 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:01 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi132 2024-04-22T13:25:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:01 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 17088M 2024-04-22T13:25:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:01 smithi120 ceph-mon[31101]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:01 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:01 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-04-22T13:25:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:01 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 17088M 2024-04-22T13:25:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:01 smithi132 ceph-mon[31531]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T13:25:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:01 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:02.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 -- 172.21.15.87:0/2564028841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc102bc0 msgr2=0x7fb4cc103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:02.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/2564028841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc103020 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0099c0 tx=0x7fb4bc009cd0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 -- 172.21.15.87:0/2564028841 shutdown_connections 2024-04-22T13:25:02.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/2564028841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103560 0x7fb4cc109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/2564028841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 0x7fb4cc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.266+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/2564028841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc103020 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- 172.21.15.87:0/2564028841 >> 172.21.15.87:0/2564028841 conn(0x7fb4cc0fcfa0 msgr2=0x7fb4cc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- 172.21.15.87:0/2564028841 shutdown_connections 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- 172.21.15.87:0/2564028841 wait complete. 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 Processor -- start 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- start start 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 0x7fb4cc198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc10d550 con 0x7fb4cc103560 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc10d3d0 con 0x7fb4cc102bc0 2024-04-22T13:25:02.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.267+0000 7fb4d37b4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc10d6d0 con 0x7fb4cc1019c0 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52516/0 (socket says 172.21.15.87:52516) 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 -- 172.21.15.87:0/4159726761 learned_addr learned my addr 172.21.15.87:0/4159726761 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1550700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 0x7fb4cc198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d0d4f700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 -- 172.21.15.87:0/4159726761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 msgr2=0x7fb4cc198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 0x7fb4cc198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 -- 172.21.15.87:0/4159726761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc102bc0 msgr2=0x7fb4cc199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 -- 172.21.15.87:0/4159726761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4cc1a0940 con 0x7fb4cc103560 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d1d51700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b970 tx=0x7fb4c400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4019070 con 0x7fb4cc103560 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4cc1a0c30 con 0x7fb4cc103560 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4c400cde0 con 0x7fb4cc103560 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4007370 con 0x7fb4cc103560 2024-04-22T13:25:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.268+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4cc1a1170 con 0x7fb4cc103560 2024-04-22T13:25:02.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.269+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4cc069730 con 0x7fb4cc103560 2024-04-22T13:25:02.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.270+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb4c401c070 con 0x7fb4cc103560 2024-04-22T13:25:02.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.271+0000 7fb4c27fc700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 0x7fb4b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:02.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.271+0000 7fb4d1550700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 0x7fb4b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:02.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.271+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fb4c40ceb30 con 0x7fb4cc103560 2024-04-22T13:25:02.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.271+0000 7fb4d1550700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 0x7fb4b80a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc19a1d0 tx=0x7fb4c8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:02.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.275+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4c4023070 con 0x7fb4cc103560 2024-04-22T13:25:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:02.483+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb4cc06b8c0 con 0x7fb4b80a3bb0 2024-04-22T13:25:02.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:02.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T13:25:02.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:02.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:02.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:02.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:03.024 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.024 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.024 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40439]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.024 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40439]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:03.025 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44472]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44472]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44472]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44472]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44475]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.060 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44475]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.061 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44475]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.061 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:25:02 smithi087 sudo[44475]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:03.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:03.364 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44533]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44533]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44533]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44533]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44536]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44536]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44536]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.365 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 sudo[44536]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39842]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39842]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39842]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39842]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39845]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.420 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:25:02 smithi120 sudo[39845]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39848]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39851]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39851]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 sudo[39851]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.463 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40442]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.463 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40442]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.463 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40442]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.463 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40442]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40445]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T13:25:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40445]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40445]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40445]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40448]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T13:25:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40448]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T13:25:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40448]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T13:25:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 sudo[40448]: pam_unix(sudo:session): session closed for user root 2024-04-22T13:25:04.135 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:04.135 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:04.135 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:25:04.135 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:03 smithi087 ceph-mon[27999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:03 smithi120 ceph-mon[31101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:25:04.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:03 smithi132 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T13:25:04.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:04.786+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4c4098940 con 0x7fb4cc103560 2024-04-22T13:25:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:04 smithi120 ceph-mon[31101]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T13:25:05.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:04 smithi132 ceph-mon[31531]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T13:25:05.277 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:04 smithi087 ceph-mon[27999]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T13:25:05.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:05.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: Cluster is now healthy 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]: dispatch 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]': finished 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T13:25:05.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: Cluster is now healthy 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]: dispatch 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]': finished 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T13:25:06.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T13:25:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: Cluster is now healthy 2024-04-22T13:25:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-22T13:25:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]: dispatch 2024-04-22T13:25:06.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2995042472' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5b62ee-b979-43d0-8f66-1c4183042bf1"}]': finished 2024-04-22T13:25:06.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T13:25:06.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:07.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:06 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3138809812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:06 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3138809812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3138809812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:08.139 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:07 smithi087 ceph-mon[27999]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:07 smithi120 ceph-mon[31101]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:07 smithi132 ceph-mon[31531]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:09 smithi087 ceph-mon[27999]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:09 smithi120 ceph-mon[31101]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:09 smithi132 ceph-mon[31531]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:11 smithi120 ceph-mon[31101]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:11 smithi132 ceph-mon[31531]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:11 smithi087 ceph-mon[27999]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:12.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:12 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T13:25:12.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:12 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:13.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:12 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T13:25:13.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:12 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:12 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T13:25:13.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:12 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:13 smithi087 ceph-mon[27999]: Deploying daemon osd.3 on smithi087 2024-04-22T13:25:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:13 smithi087 ceph-mon[27999]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:13 smithi120 ceph-mon[31101]: Deploying daemon osd.3 on smithi087 2024-04-22T13:25:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:13 smithi120 ceph-mon[31101]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:13 smithi132 ceph-mon[31531]: Deploying daemon osd.3 on smithi087 2024-04-22T13:25:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:13 smithi132 ceph-mon[31531]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:15 smithi120 ceph-mon[31101]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:15 smithi132 ceph-mon[31531]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:15 smithi087 ceph-mon[27999]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:18.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:17 smithi087 ceph-mon[27999]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:17 smithi120 ceph-mon[31101]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:18.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:17 smithi132 ceph-mon[31531]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:19 smithi087 ceph-mon[27999]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:19 smithi120 ceph-mon[31101]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:19 smithi132 ceph-mon[31531]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.593+0000 7fb4c27fc700 1 -- 172.21.15.87:0/4159726761 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb4cc06b8c0 con 0x7fb4b80a3bb0 2024-04-22T13:25:21.590 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 3 on host 'smithi087' 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 msgr2=0x7fb4b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 0x7fb4b80a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc19a1d0 tx=0x7fb4c8009190 comp rx=0 tx=0).stop 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 msgr2=0x7fb4cc1a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b970 tx=0x7fb4c400bc80 comp rx=0 tx=0).stop 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 shutdown_connections 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4b80a3bb0 0x7fb4b80a6070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4cc103560 0x7fb4cc1a0270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc1019c0 0x7fb4cc198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:21.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 --2- 172.21.15.87:0/4159726761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc102bc0 0x7fb4cc199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:21.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.595+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 >> 172.21.15.87:0/4159726761 conn(0x7fb4cc0fcfa0 msgr2=0x7fb4cc06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:25:21.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.596+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 shutdown_connections 2024-04-22T13:25:21.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:25:21.596+0000 7fb4d37b4700 1 -- 172.21.15.87:0/4159726761 wait complete. 2024-04-22T13:25:21.932 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:21 smithi087 ceph-mon[27999]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:21.932 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:21.932 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:21 smithi120 ceph-mon[31101]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:22.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:21 smithi132 ceph-mon[31531]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.587 DEBUG:teuthology.orchestra.run.smithi087:osd.3> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T13:25:22.589 INFO:tasks.cephadm:Deploying osd.4 on smithi120 with /dev/vg_nvme/lv_3... 2024-04-22T13:25:22.589 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T13:25:22.835 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:25:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:22 smithi120 ceph-mon[31101]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T13:25:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:22 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:22 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:22 smithi087 ceph-mon[27999]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T13:25:22.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:22 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:22.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:22 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:23.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:22 smithi132 ceph-mon[31531]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T13:25:23.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:22 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:23.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:22 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:23.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:23 smithi087 ceph-mon[27999]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:23.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:23 smithi087 ceph-mon[27999]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T13:25:23.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:23 smithi087 ceph-mon[27999]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T13:25:23.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:23 smithi087 ceph-mon[27999]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:25:23.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:23 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:23 smithi120 ceph-mon[31101]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:23 smithi120 ceph-mon[31101]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T13:25:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:23 smithi120 ceph-mon[31101]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T13:25:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:23 smithi120 ceph-mon[31101]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:25:24.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:23 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:23 smithi132 ceph-mon[31531]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:23 smithi132 ceph-mon[31531]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T13:25:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:23 smithi132 ceph-mon[31531]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T13:25:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:23 smithi132 ceph-mon[31531]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:25:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:23 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:24.487 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3[46676]: 2024-04-22T13:25:24.187+0000 7f7ed6ed1700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T13:25:24.487 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3[46676]: 2024-04-22T13:25:24.192+0000 7f7ececbc700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' 2024-04-22T13:25:25.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] boot 2024-04-22T13:25:25.078 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T13:25:25.078 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:24 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111587 s, 940 MB/s 2024-04-22T13:25:25.078 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-04-22T13:25:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:25:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T13:25:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' 2024-04-22T13:25:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] boot 2024-04-22T13:25:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T13:25:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:24 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:25:25.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: from='osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327]' entity='osd.3' 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: osd.3 [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] boot 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T13:25:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:25:25.728 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_3 2024-04-22T13:25:25.962 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:25:26.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:25 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:25:26.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:25 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:25:26.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:25 smithi087 ceph-mon[27999]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:26.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:25 smithi087 ceph-mon[27999]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T13:25:26.083 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:25 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:25:26.083 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:25 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:25:26.083 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:25 smithi120 ceph-mon[31101]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:26.084 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:25 smithi120 ceph-mon[31101]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T13:25:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:25 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:25:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:25 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:25:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:25 smithi132 ceph-mon[31531]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T13:25:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:25 smithi132 ceph-mon[31531]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 -- 172.21.15.120:0/1707457472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98781019a0 msgr2=0x7f9878101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 --2- 172.21.15.120:0/1707457472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98781019a0 0x7f9878101dc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f98680099c0 tx=0x7f9868009cd0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 -- 172.21.15.120:0/1707457472 shutdown_connections 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 --2- 172.21.15.120:0/1707457472 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878103540 0x7f9878109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 --2- 172.21.15.120:0/1707457472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98781019a0 0x7f9878101dc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 --2- 172.21.15.120:0/1707457472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9878102ba0 0x7f9878103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 -- 172.21.15.120:0/1707457472 >> 172.21.15.120:0/1707457472 conn(0x7f98780fcf40 msgr2=0x7f98780ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 -- 172.21.15.120:0/1707457472 shutdown_connections 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.393+0000 7f987c980700 1 -- 172.21.15.120:0/1707457472 wait complete. 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 Processor -- start 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 -- start start 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98781019a0 0x7f9878198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878102ba0 0x7f98781991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f987810d430 con 0x7f9878102ba0 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f987810d2b0 con 0x7f9878103540 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.394+0000 7f987c980700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f987810d5b0 con 0x7f98781019a0 2024-04-22T13:25:27.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:37558/0 (socket says 172.21.15.120:37558) 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 -- 172.21.15.120:0/1618486024 learned_addr learned my addr 172.21.15.120:0/1618486024 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9875d9b700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878102ba0 0x7f98781991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f987659c700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98781019a0 0x7f9878198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 -- 172.21.15.120:0/1618486024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98781019a0 msgr2=0x7f9878198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98781019a0 0x7f9878198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 -- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878102ba0 msgr2=0x7f98781991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878102ba0 0x7f98781991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 -- 172.21.15.120:0/1618486024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98781a0920 con 0x7f9878103540 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f9876d9d700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9870007ee0 tx=0x7f987000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870004bb0 con 0x7f9878103540 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9870004d50 con 0x7f9878103540 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98781a0c10 con 0x7f9878103540 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870005920 con 0x7f9878103540 2024-04-22T13:25:27.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.395+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98781a10d0 con 0x7f9878103540 2024-04-22T13:25:27.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.396+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9878069760 con 0x7f9878103540 2024-04-22T13:25:27.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.397+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f987001a020 con 0x7f9878103540 2024-04-22T13:25:27.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.398+0000 7f98677fe700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 0x7f98600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:27.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.398+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f98700cef40 con 0x7f9878103540 2024-04-22T13:25:27.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.398+0000 7f987659c700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 0x7f98600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:27.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.399+0000 7f987659c700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 0x7f98600a6100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9868006010 tx=0x7f9868003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:27.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.402+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9878069760 con 0x7f9878103540 2024-04-22T13:25:27.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:27.607+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f987806ae60 con 0x7f98600a3c40 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi087 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 8544M 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.604 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.605 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.605 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:27 smithi120 ceph-mon[31101]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:27.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T13:25:27.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi087 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 8544M 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:27 smithi132 ceph-mon[31531]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:27.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi087 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 8544M 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:27.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:27.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:27.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:27.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:27 smithi087 ceph-mon[27999]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:28.465 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:28 smithi120 ceph-mon[31101]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:28.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:28.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:28.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:28.466 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:28 smithi120 ceph-mon[31101]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T13:25:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:28 smithi132 ceph-mon[31531]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:28.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:28.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:28 smithi132 ceph-mon[31531]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T13:25:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:28 smithi087 ceph-mon[27999]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:28 smithi087 ceph-mon[27999]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T13:25:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:29 smithi120 ceph-mon[31101]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T13:25:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:29 smithi120 ceph-mon[31101]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:29.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:29 smithi132 ceph-mon[31531]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T13:25:29.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:29 smithi132 ceph-mon[31531]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:29 smithi087 ceph-mon[27999]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T13:25:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:29 smithi087 ceph-mon[27999]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:30.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:30 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/2061791876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:30 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:30 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]': finished 2024-04-22T13:25:30.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:30 smithi120 ceph-mon[31101]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T13:25:30.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:30.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:30 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/2061791876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:30 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:30 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]': finished 2024-04-22T13:25:30.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:30 smithi132 ceph-mon[31531]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T13:25:30.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:30 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:30 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/2061791876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:30 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]: dispatch 2024-04-22T13:25:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:30 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd7d7e47-595c-4f2b-846c-0030654a9954"}]': finished 2024-04-22T13:25:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:30 smithi087 ceph-mon[27999]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T13:25:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:30 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:31.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:31 smithi120 ceph-mon[31101]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:31.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:31 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/3004779435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:31 smithi132 ceph-mon[31531]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:31 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/3004779435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:31.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:31 smithi087 ceph-mon[27999]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:31.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:31 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/3004779435' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:34.008 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:33 smithi120 ceph-mon[31101]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:25:34.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:33 smithi132 ceph-mon[31531]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:25:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:33 smithi087 ceph-mon[27999]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:25:36.144 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:35 smithi120 ceph-mon[31101]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T13:25:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:35 smithi132 ceph-mon[31531]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T13:25:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:35 smithi087 ceph-mon[27999]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T13:25:37.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:37 smithi120 ceph-mon[31101]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:25:38.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:37 smithi087 ceph-mon[27999]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:25:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:37 smithi132 ceph-mon[31531]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:25:38.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T13:25:38.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:38.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:38 smithi120 ceph-mon[31101]: Deploying daemon osd.4 on smithi120 2024-04-22T13:25:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T13:25:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:38 smithi132 ceph-mon[31531]: Deploying daemon osd.4 on smithi120 2024-04-22T13:25:39.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T13:25:39.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:39.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:38 smithi087 ceph-mon[27999]: Deploying daemon osd.4 on smithi120 2024-04-22T13:25:40.102 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:39 smithi120 ceph-mon[31101]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:25:40.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:39 smithi132 ceph-mon[31531]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:25:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:39 smithi087 ceph-mon[27999]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:25:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:41 smithi120 ceph-mon[31101]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T13:25:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:41 smithi132 ceph-mon[31531]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T13:25:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:41 smithi087 ceph-mon[27999]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T13:25:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:43 smithi120 ceph-mon[31101]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T13:25:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:43 smithi132 ceph-mon[31531]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T13:25:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:43 smithi087 ceph-mon[27999]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T13:25:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:46.042 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:45 smithi120 ceph-mon[31101]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:46.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:45 smithi132 ceph-mon[31531]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:45 smithi087 ceph-mon[27999]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:47.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.221+0000 7f98677fe700 1 -- 172.21.15.120:0/1618486024 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f987806ae60 con 0x7f98600a3c40 2024-04-22T13:25:47.217 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 4 on host 'smithi120' 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 msgr2=0x7f98600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 0x7f98600a6100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9868006010 tx=0x7f9868003680 comp rx=0 tx=0).stop 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 msgr2=0x7f98781a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9870007ee0 tx=0x7f987000d3b0 comp rx=0 tx=0).stop 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 shutdown_connections 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98600a3c40 0x7f98600a6100 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9878103540 0x7f98781a0250 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:47.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98781019a0 0x7f9878198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:47.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 --2- 172.21.15.120:0/1618486024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9878102ba0 0x7f98781991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:47.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.223+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 >> 172.21.15.120:0/1618486024 conn(0x7f98780fcf40 msgr2=0x7f987806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:25:47.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.224+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 shutdown_connections 2024-04-22T13:25:47.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:25:47.224+0000 7f987c980700 1 -- 172.21.15.120:0/1618486024 wait complete. 2024-04-22T13:25:47.428 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.428 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.428 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:47.428 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:47.429 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.429 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:47.429 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.429 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.429 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:47.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:47.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:47.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:47.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:25:47.867 DEBUG:teuthology.orchestra.run.smithi120:osd.4> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T13:25:47.869 INFO:tasks.cephadm:Deploying osd.5 on smithi132 with /dev/vg_nvme/lv_3... 2024-04-22T13:25:47.870 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T13:25:48.115 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:25:48.555 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:48 smithi132 ceph-mon[31531]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:48.555 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:48 smithi132 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:48 smithi120 ceph-mon[31101]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:48 smithi120 ceph-mon[31101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:48 smithi087 ceph-mon[27999]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:48 smithi087 ceph-mon[27999]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T13:25:49.403 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T13:25:49.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T13:25:49.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:49.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:49 smithi132 ceph-mon[31531]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T13:25:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T13:25:49.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:49.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-mon[31101]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: from='osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:25:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:49 smithi087 ceph-mon[27999]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:49.973 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T13:25:49.974 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T13:25:49.974 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-22T13:25:49.974 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-22T13:25:49.974 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112464 s, 932 MB/s 2024-04-22T13:25:49.974 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-22T13:25:50.170 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:25:49.812+0000 7f1bf1645700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T13:25:50.170 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:25:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:25:49.819+0000 7f1be9430700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:25:50.514 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:50 smithi120 ceph-mon[31101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:25:50.514 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:50 smithi120 ceph-mon[31101]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T13:25:50.514 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.514 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:50 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.514 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:50 smithi120 ceph-mon[31101]: from='osd.4 ' entity='osd.4' 2024-04-22T13:25:50.592 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:50 smithi132 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:25:50.592 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:50 smithi132 ceph-mon[31531]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T13:25:50.592 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.592 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:50 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.592 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:50 smithi132 ceph-mon[31531]: from='osd.4 ' entity='osd.4' 2024-04-22T13:25:50.621 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-04-22T13:25:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:50 smithi087 ceph-mon[27999]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:25:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:50 smithi087 ceph-mon[27999]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T13:25:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:50 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:50 smithi087 ceph-mon[27999]: from='osd.4 ' entity='osd.4' 2024-04-22T13:25:50.873 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] boot 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.454 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:51.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:51.455 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.648 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] boot 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:51.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:25:51.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:25:51.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: osd.4 [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] boot 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:25:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:25:52.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 -- 172.21.15.132:0/2007902533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58102aa0 msgr2=0x7f4b58102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/2007902533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58102aa0 0x7f4b58102ec0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009990 tx=0x7f4b48009ca0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 -- 172.21.15.132:0/2007902533 shutdown_connections 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/2007902533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b5810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/2007902533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58102aa0 0x7f4b58102ec0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/2007902533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58103ca0 0x7f4b58104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 -- 172.21.15.132:0/2007902533 >> 172.21.15.132:0/2007902533 conn(0x7f4b580fe040 msgr2=0x7f4b58100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 -- 172.21.15.132:0/2007902533 shutdown_connections 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.484+0000 7f4b5e78f700 1 -- 172.21.15.132:0/2007902533 wait complete. 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 Processor -- start 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 -- start start 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 0x7f4b5819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:52.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b581a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5810e510 con 0x7f4b58104640 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5810e390 con 0x7f4b58102aa0 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.485+0000 7f4b5e78f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5810e690 con 0x7f4b58103ca0 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43946/0 (socket says 172.21.15.132:43946) 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 -- 172.21.15.132:0/106533457 learned_addr learned my addr 172.21.15.132:0/106533457 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b57fff700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 0x7f4b5819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b5cd2c700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b581a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 -- 172.21.15.132:0/106533457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 msgr2=0x7f4b5819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 0x7f4b5819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 -- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 msgr2=0x7f4b581a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:25:52.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b581a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 -- 172.21.15.132:0/106533457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b581a29e0 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b5cd2c700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b581a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b57fff700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 0x7f4b5819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b577fe700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00c9b0 tx=0x7f4b4c00cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c007b00 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b4c007ca0 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.486+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b581a2cd0 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.487+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c018730 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.487+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b581a31b0 con 0x7f4b58103ca0 2024-04-22T13:25:52.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.487+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b581a3470 con 0x7f4b58103ca0 2024-04-22T13:25:52.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.488+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b4c00f450 con 0x7f4b58103ca0 2024-04-22T13:25:52.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.489+0000 7f4b557fa700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 0x7f4b400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:25:52.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.489+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f4b4c0cf3a0 con 0x7f4b58103ca0 2024-04-22T13:25:52.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.489+0000 7f4b57fff700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 0x7f4b400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:25:52.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.490+0000 7f4b57fff700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 0x7f4b400a6180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b48003790 tx=0x7f4b48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:25:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.494+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b4c014030 con 0x7f4b58103ca0 2024-04-22T13:25:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:52 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi120 2024-04-22T13:25:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:52 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 8544M 2024-04-22T13:25:52.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:25:52.700+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4b581a3640 con 0x7f4b400a3cc0 2024-04-22T13:25:52.697 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:52 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi120 2024-04-22T13:25:52.697 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:52 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 8544M 2024-04-22T13:25:52.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:52 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi120 2024-04-22T13:25:52.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:52 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 8544M 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:53.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:25:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:25:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/4063164984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]': finished 2024-04-22T13:25:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T13:25:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:55 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/4063164984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]': finished 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T13:25:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:55 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/4063164984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]: dispatch 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c76c783-6c30-48d4-b20b-c4ef76b03f7f"}]': finished 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T13:25:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:55 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:25:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/3439144079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/3439144079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:57.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/3439144079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:25:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:57 smithi120 ceph-mon[31101]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:57 smithi132 ceph-mon[31531]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:57 smithi087 ceph-mon[27999]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:25:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:25:59 smithi132 ceph-mon[31531]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:00.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:25:59 smithi120 ceph-mon[31101]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:25:59 smithi087 ceph-mon[27999]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:01 smithi120 ceph-mon[31101]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:01 smithi132 ceph-mon[31531]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:02.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:01 smithi087 ceph-mon[27999]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:03.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T13:26:03.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:02 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:03.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T13:26:03.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:02 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:03.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T13:26:03.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:02 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:03.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:03 smithi132 ceph-mon[31531]: Deploying daemon osd.5 on smithi132 2024-04-22T13:26:03.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:03 smithi132 ceph-mon[31531]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:03 smithi120 ceph-mon[31101]: Deploying daemon osd.5 on smithi132 2024-04-22T13:26:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:03 smithi120 ceph-mon[31101]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:03 smithi087 ceph-mon[27999]: Deploying daemon osd.5 on smithi132 2024-04-22T13:26:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:03 smithi087 ceph-mon[27999]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:05.869 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:05 smithi132 ceph-mon[31531]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:05 smithi120 ceph-mon[31101]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:06.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:05 smithi087 ceph-mon[27999]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:07.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:07 smithi132 ceph-mon[31531]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:08.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:07 smithi087 ceph-mon[27999]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:07 smithi120 ceph-mon[31101]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:09 smithi132 ceph-mon[31531]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:09 smithi087 ceph-mon[27999]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:09 smithi120 ceph-mon[31101]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:11.932 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:11.932 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:11.933 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:11.933 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:11.933 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:11.933 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:12.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.272+0000 7f4b557fa700 1 -- 172.21.15.132:0/106533457 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b581a3640 con 0x7f4b400a3cc0 2024-04-22T13:26:12.268 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 5 on host 'smithi132' 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 msgr2=0x7f4b400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 0x7f4b400a6180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b48003790 tx=0x7f4b48005fd0 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 msgr2=0x7f4b5819b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00c9b0 tx=0x7f4b4c00cd70 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 shutdown_connections 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4b400a3cc0 0x7f4b400a6180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b58104640 0x7f4b581a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b58102aa0 0x7f4b5819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 --2- 172.21.15.132:0/106533457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b58103ca0 0x7f4b5819b2d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 >> 172.21.15.132:0/106533457 conn(0x7f4b580fe040 msgr2=0x7f4b58074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 shutdown_connections 2024-04-22T13:26:12.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:26:12.274+0000 7f4b5e78f700 1 -- 172.21.15.132:0/106533457 wait complete. 2024-04-22T13:26:12.995 DEBUG:teuthology.orchestra.run.smithi132:osd.5> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T13:26:12.998 INFO:tasks.cephadm:Deploying osd.6 on smithi087 with /dev/vg_nvme/lv_2... 2024-04-22T13:26:12.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T13:26:13.244 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.366 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:13 smithi087 ceph-mon[27999]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.549 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:13 smithi132 ceph-mon[31531]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T13:26:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:13 smithi120 ceph-mon[31101]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:14.522 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T13:26:14.522 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-mon[31531]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:14.522 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-mon[31531]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T13:26:14.522 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:14.522 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:14.522 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:26:14.263+0000 7f6de19ba700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T13:26:14.522 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:26:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:26:14.269+0000 7f6ddd7ad700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:26:14.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:14 smithi087 ceph-mon[27999]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T13:26:14.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:14 smithi087 ceph-mon[27999]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:14.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:14 smithi087 ceph-mon[27999]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T13:26:14.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:14.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:14 smithi087 ceph-mon[27999]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:14 smithi120 ceph-mon[31101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T13:26:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:14 smithi120 ceph-mon[31101]: from='osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:14 smithi120 ceph-mon[31101]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T13:26:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:14 smithi120 ceph-mon[31101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:26:15.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:15 smithi087 ceph-mon[27999]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:26:15.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:15 smithi087 ceph-mon[27999]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T13:26:15.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:15 smithi087 ceph-mon[27999]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:15.487 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T13:26:15.487 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T13:26:15.487 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-04-22T13:26:15.487 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-04-22T13:26:15.488 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117338 s, 894 MB/s 2024-04-22T13:26:15.488 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-04-22T13:26:15.660 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:15 smithi132 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:26:15.660 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:15 smithi132 ceph-mon[31531]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T13:26:15.660 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.660 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.660 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:15 smithi132 ceph-mon[31531]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:15 smithi120 ceph-mon[31101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:26:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:15 smithi120 ceph-mon[31101]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T13:26:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:15 smithi120 ceph-mon[31101]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-22T13:26:16.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_2 2024-04-22T13:26:16.427 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] boot 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:16.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] boot 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:16.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:16.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:26:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:26:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: osd.5 [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] boot 2024-04-22T13:26:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:16.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:17.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:17 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi132 2024-04-22T13:26:17.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:17 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 8544M 2024-04-22T13:26:17.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:17 smithi087 ceph-mon[27999]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T13:26:17.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:17 smithi087 ceph-mon[27999]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:17.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:17 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi132 2024-04-22T13:26:17.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:17 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 8544M 2024-04-22T13:26:17.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:17 smithi120 ceph-mon[31101]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T13:26:17.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:17 smithi120 ceph-mon[31101]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:17.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:17 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-04-22T13:26:17.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:17 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 8544M 2024-04-22T13:26:17.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:17 smithi132 ceph-mon[31531]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T13:26:17.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:17 smithi132 ceph-mon[31531]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 -- 172.21.15.87:0/3856935393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248102aa0 msgr2=0x7f3248102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 --2- 172.21.15.87:0/3856935393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248102aa0 0x7f3248102ec0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32440099c0 tx=0x7f3244009cd0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 -- 172.21.15.87:0/3856935393 shutdown_connections 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 --2- 172.21.15.87:0/3856935393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f324810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 --2- 172.21.15.87:0/3856935393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248102aa0 0x7f3248102ec0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.131+0000 7f32507c6700 1 --2- 172.21.15.87:0/3856935393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248103ca0 0x7f3248104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- 172.21.15.87:0/3856935393 >> 172.21.15.87:0/3856935393 conn(0x7f32480fe040 msgr2=0x7f3248100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:26:18.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- 172.21.15.87:0/3856935393 shutdown_connections 2024-04-22T13:26:18.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- 172.21.15.87:0/3856935393 wait complete. 2024-04-22T13:26:18.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 Processor -- start 2024-04-22T13:26:18.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- start start 2024-04-22T13:26:18.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 0x7f3248198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:18.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f32481a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f324810e5b0 con 0x7f3248103ca0 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f324810e430 con 0x7f3248104640 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f32507c6700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f324810e730 con 0x7f3248102aa0 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.132+0000 7f324dd61700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324e562700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 0x7f3248198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52852/0 (socket says 172.21.15.87:52852) 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 -- 172.21.15.87:0/3869725730 learned_addr learned my addr 172.21.15.87:0/3869725730 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324ed63700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f32481a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 -- 172.21.15.87:0/3869725730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 msgr2=0x7f3248198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 0x7f3248198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 -- 172.21.15.87:0/3869725730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 msgr2=0x7f32481a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f32481a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 -- 172.21.15.87:0/3869725730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32481a0900 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324e562700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 0x7f3248198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324ed63700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f32481a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f324dd61700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f323800d980 tx=0x7f323800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238017070 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3238005020 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32481a0bf0 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238009b50 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.133+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32481a10b0 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.134+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3248069730 con 0x7f3248103ca0 2024-04-22T13:26:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.136+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f32380051d0 con 0x7f3248103ca0 2024-04-22T13:26:18.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.137+0000 7f323f7fe700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 0x7f32340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:18.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.137+0000 7f324e562700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 0x7f32340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:18.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.137+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f32380cf810 con 0x7f3248103ca0 2024-04-22T13:26:18.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.137+0000 7f324e562700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 0x7f32340a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f32440038c0 tx=0x7f3244000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:26:18.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.140+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3238020020 con 0x7f3248103ca0 2024-04-22T13:26:18.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:18.350+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3248062e80 con 0x7f32340a3c40 2024-04-22T13:26:18.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:18 smithi087 ceph-mon[27999]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T13:26:18.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:18 smithi120 ceph-mon[31101]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T13:26:18.671 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:18 smithi132 ceph-mon[31531]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T13:26:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T13:26:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:19.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:19.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:19.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:19.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:19.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:19 smithi087 ceph-mon[27999]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T13:26:19.571 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:19.572 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:19 smithi120 ceph-mon[31101]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:19 smithi132 ceph-mon[31531]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T13:26:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:21 smithi120 ceph-mon[31101]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:21 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]: dispatch 2024-04-22T13:26:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:21 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]': finished 2024-04-22T13:26:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:21 smithi120 ceph-mon[31101]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T13:26:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:21 smithi132 ceph-mon[31531]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:21 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]: dispatch 2024-04-22T13:26:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:21 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]': finished 2024-04-22T13:26:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:21 smithi132 ceph-mon[31531]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T13:26:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:21 smithi087 ceph-mon[27999]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:21 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]: dispatch 2024-04-22T13:26:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:21 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2481903813' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d20c5d7-b08d-4a48-8694-1d2faf0a1651"}]': finished 2024-04-22T13:26:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:21 smithi087 ceph-mon[27999]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T13:26:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:23.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2135348971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:23.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2135348971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:23.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2135348971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:23 smithi120 ceph-mon[31101]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:26:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:23 smithi132 ceph-mon[31531]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:26:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:23 smithi087 ceph-mon[27999]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:26:25.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:25 smithi087 ceph-mon[27999]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T13:26:26.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:25 smithi120 ceph-mon[31101]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T13:26:26.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:25 smithi132 ceph-mon[31531]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T13:26:27.851 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:27 smithi087 ceph-mon[27999]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:26:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:27 smithi120 ceph-mon[31101]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:26:28.174 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:27 smithi132 ceph-mon[31531]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:26:29.859 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:29 smithi087 ceph-mon[27999]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:26:29.859 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T13:26:29.859 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:29 smithi120 ceph-mon[31101]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:26:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T13:26:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:29 smithi132 ceph-mon[31531]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:26:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T13:26:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:30.900 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:30 smithi087 ceph-mon[27999]: Deploying daemon osd.6 on smithi087 2024-04-22T13:26:31.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:30 smithi120 ceph-mon[31101]: Deploying daemon osd.6 on smithi087 2024-04-22T13:26:31.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:30 smithi132 ceph-mon[31531]: Deploying daemon osd.6 on smithi087 2024-04-22T13:26:31.948 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:31 smithi087 ceph-mon[27999]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:26:32.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:31 smithi120 ceph-mon[31101]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:26:32.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:31 smithi132 ceph-mon[31531]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:26:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:33 smithi120 ceph-mon[31101]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:34.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:33 smithi132 ceph-mon[31531]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:34.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:33 smithi087 ceph-mon[27999]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:35 smithi120 ceph-mon[31101]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:35 smithi132 ceph-mon[31531]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:35 smithi087 ceph-mon[27999]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:37.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:37 smithi087 ceph-mon[27999]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:37 smithi120 ceph-mon[31101]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:37 smithi132 ceph-mon[31531]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:38.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.811+0000 7f323f7fe700 1 -- 172.21.15.87:0/3869725730 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3248062e80 con 0x7f32340a3c40 2024-04-22T13:26:38.808 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 6 on host 'smithi087' 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 msgr2=0x7f32340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 0x7f32340a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f32440038c0 tx=0x7f3244000bc0 comp rx=0 tx=0).stop 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 msgr2=0x7f32481991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f323800d980 tx=0x7f323800dc90 comp rx=0 tx=0).stop 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 shutdown_connections 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f32340a3c40 0x7f32340a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3248104640 0x7f32481a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3248102aa0 0x7f3248198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:38.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 --2- 172.21.15.87:0/3869725730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3248103ca0 0x7f32481991c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:38.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.813+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 >> 172.21.15.87:0/3869725730 conn(0x7f32480fe040 msgr2=0x7f32481070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:26:38.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.814+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 shutdown_connections 2024-04-22T13:26:38.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:26:38.814+0000 7f32507c6700 1 -- 172.21.15.87:0/3869725730 wait complete. 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:38.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:38 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:39.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:39.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:39.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T13:26:39.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:38 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T13:26:39.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:39.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:38 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:26:39.440 DEBUG:teuthology.orchestra.run.smithi087:osd.6> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T13:26:39.443 INFO:tasks.cephadm:Deploying osd.7 on smithi120 with /dev/vg_nvme/lv_2... 2024-04-22T13:26:39.443 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T13:26:39.688 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:26:40.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:39 smithi120 ceph-mon[31101]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:40.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:39 smithi120 ceph-mon[31101]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T13:26:40.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:39 smithi120 ceph-mon[31101]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T13:26:40.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:39 smithi120 ceph-mon[31101]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:26:40.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:40.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:39 smithi132 ceph-mon[31531]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:40.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:39 smithi132 ceph-mon[31531]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T13:26:40.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:39 smithi132 ceph-mon[31531]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T13:26:40.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:39 smithi132 ceph-mon[31531]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:26:40.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-mon[27999]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-mon[27999]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T13:26:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-mon[27999]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T13:26:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-mon[27999]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:26:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:40.238 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:26:39.873+0000 7fa094506700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T13:26:40.238 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:26:39 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:26:39.884+0000 7fa0902f9700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:26:41.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:40 smithi087 ceph-mon[27999]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:26:41.096 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:40 smithi087 ceph-mon[27999]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T13:26:41.096 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.096 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:40 smithi120 ceph-mon[31101]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:26:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:40 smithi120 ceph-mon[31101]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T13:26:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:40 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:40 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:40 smithi132 ceph-mon[31531]: from='osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:26:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:40 smithi132 ceph-mon[31531]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T13:26:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:40 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:40 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011501 s, 912 MB/s 2024-04-22T13:26:41.895 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-04-22T13:26:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:26:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:26:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] boot 2024-04-22T13:26:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T13:26:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:42.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:26:42.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:26:42.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:42.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] boot 2024-04-22T13:26:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T13:26:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: osd.6 [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] boot 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T13:26:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:26:42.538 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_2 2024-04-22T13:26:42.779 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:26:42.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T13:26:42.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:42.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:42.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:43.164 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:26:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:26:44.166 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:43 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi087 2024-04-22T13:26:44.167 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:43 smithi120 ceph-mon[31101]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:44.167 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:43 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 5696M 2024-04-22T13:26:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:43 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi087 2024-04-22T13:26:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:43 smithi132 ceph-mon[31531]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:43 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 5696M 2024-04-22T13:26:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:43 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi087 2024-04-22T13:26:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:43 smithi087 ceph-mon[27999]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:43 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 5696M 2024-04-22T13:26:44.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 -- 172.21.15.120:0/3659479349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c1019a0 msgr2=0x7f786c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3659479349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c1019a0 0x7f786c101dc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78600099c0 tx=0x7f7860009cd0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 -- 172.21.15.120:0/3659479349 shutdown_connections 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3659479349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c103540 0x7f786c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3659479349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c1019a0 0x7f786c101dc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3659479349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c102ba0 0x7f786c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 -- 172.21.15.120:0/3659479349 >> 172.21.15.120:0/3659479349 conn(0x7f786c0fcf60 msgr2=0x7f786c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 -- 172.21.15.120:0/3659479349 shutdown_connections 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.400+0000 7f7872d9f700 1 -- 172.21.15.120:0/3659479349 wait complete. 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 Processor -- start 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 -- start start 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c1019a0 0x7f786c105930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 0x7f786c103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:44.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f786c10d410 con 0x7f786c102ba0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f786c10d290 con 0x7f786c103540 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7872d9f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f786c10d590 con 0x7f786c1019a0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:34624/0 (socket says 172.21.15.120:34624) 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 -- 172.21.15.120:0/3297056891 learned_addr learned my addr 172.21.15.120:0/3297056891 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f7870b3b700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c1019a0 0x7f786c105930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f786bfff700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 0x7f786c103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 -- 172.21.15.120:0/3297056891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c1019a0 msgr2=0x7f786c105930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c1019a0 0x7f786c105930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 -- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 msgr2=0x7f786c103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 0x7f786c103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 -- 172.21.15.120:0/3297056891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c104a00 con 0x7f786c103540 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f786bfff700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 0x7f786c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.401+0000 7f787133c700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7864007d70 tx=0x7f786400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.402+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864016020 con 0x7f786c103540 2024-04-22T13:26:44.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.402+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f786400ad40 con 0x7f786c103540 2024-04-22T13:26:44.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.402+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640115e0 con 0x7f786c103540 2024-04-22T13:26:44.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.402+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c104cf0 con 0x7f786c103540 2024-04-22T13:26:44.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.402+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1a75d0 con 0x7f786c103540 2024-04-22T13:26:44.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.403+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c069730 con 0x7f786c103540 2024-04-22T13:26:44.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.404+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7864019020 con 0x7f786c103540 2024-04-22T13:26:44.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.404+0000 7f7869ffb700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 0x7f785c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:26:44.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.405+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f78640cf6b0 con 0x7f786c103540 2024-04-22T13:26:44.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.405+0000 7f7870b3b700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 0x7f785c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:26:44.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.405+0000 7f7870b3b700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 0x7f785c0a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f786000b580 tx=0x7f7860005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:26:44.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.409+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7864020020 con 0x7f786c103540 2024-04-22T13:26:44.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:26:44.617+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f786c0ff0a0 con 0x7f785c0a3cc0 2024-04-22T13:26:45.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:44 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:45.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:44 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:45.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:44 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:44 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:44 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:44 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:45.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:44 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:26:45.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:44 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:26:45.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:44 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:45 smithi120 ceph-mon[31101]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:45 smithi120 ceph-mon[31101]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:46.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:45 smithi132 ceph-mon[31531]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:46.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:45 smithi132 ceph-mon[31531]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:45 smithi087 ceph-mon[27999]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:26:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:45 smithi087 ceph-mon[27999]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/3437365299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]': finished 2024-04-22T13:26:48.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T13:26:48.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:26:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/3437365299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]': finished 2024-04-22T13:26:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T13:26:48.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/3437365299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]: dispatch 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "086e7b21-8587-47e5-a1c3-4cb514d43975"}]': finished 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T13:26:48.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:26:49.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/271658731' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:49.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/271658731' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:49.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/271658731' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:26:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:49 smithi120 ceph-mon[31101]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:49 smithi132 ceph-mon[31531]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:49 smithi087 ceph-mon[27999]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:50 smithi120 ceph-mon[31101]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:51.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:50 smithi132 ceph-mon[31531]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:51.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:50 smithi087 ceph-mon[27999]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:53 smithi120 ceph-mon[31101]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:53 smithi132 ceph-mon[31531]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:53 smithi087 ceph-mon[27999]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:55.820 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:55 smithi120 ceph-mon[31101]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:55.820 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:55 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T13:26:55.820 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:55 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:55 smithi132 ceph-mon[31531]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:55 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T13:26:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:55 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:55 smithi087 ceph-mon[27999]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:55 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T13:26:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:55 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:26:57.035 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:56 smithi120 ceph-mon[31101]: Deploying daemon osd.7 on smithi120 2024-04-22T13:26:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:56 smithi132 ceph-mon[31531]: Deploying daemon osd.7 on smithi120 2024-04-22T13:26:57.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:56 smithi087 ceph-mon[27999]: Deploying daemon osd.7 on smithi120 2024-04-22T13:26:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:57 smithi120 ceph-mon[31101]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:57 smithi132 ceph-mon[31531]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:26:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:57 smithi087 ceph-mon[27999]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:00.089 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:26:59 smithi120 ceph-mon[31101]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:26:59 smithi132 ceph-mon[31531]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:26:59 smithi087 ceph-mon[27999]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:01 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:01 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:01 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:01 smithi120 ceph-mon[31101]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:01 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:01 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:01 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:01 smithi132 ceph-mon[31531]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:01 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:01 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:01 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:01 smithi087 ceph-mon[27999]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:04.150 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:04.151 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.151 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.151 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:04.151 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:04.151 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:04.531 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 7 on host 'smithi120' 2024-04-22T13:27:04.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.535+0000 7f7869ffb700 1 -- 172.21.15.120:0/3297056891 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f786c0ff0a0 con 0x7f785c0a3cc0 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 msgr2=0x7f785c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 0x7f785c0a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f786000b580 tx=0x7f7860005fb0 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 msgr2=0x7f786c1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7864007d70 tx=0x7f786400a340 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 shutdown_connections 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f785c0a3cc0 0x7f785c0a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f786c103540 0x7f786c1044c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f786c1019a0 0x7f786c105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 --2- 172.21.15.120:0/3297056891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f786c102ba0 0x7f786c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 >> 172.21.15.120:0/3297056891 conn(0x7f786c0fcf60 msgr2=0x7f786c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 shutdown_connections 2024-04-22T13:27:04.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:27:04.537+0000 7f7872d9f700 1 -- 172.21.15.120:0/3297056891 wait complete. 2024-04-22T13:27:05.240 DEBUG:teuthology.orchestra.run.smithi120:osd.7> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T13:27:05.242 INFO:tasks.cephadm:Deploying osd.8 on smithi132 with /dev/vg_nvme/lv_2... 2024-04-22T13:27:05.243 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T13:27:05.490 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:05 smithi132 ceph-mon[31531]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:05 smithi087 ceph-mon[27999]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:05.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:05.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:05.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.767 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T13:27:05.768 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:05 smithi120 ceph-mon[31101]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:06 smithi132 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T13:27:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:06 smithi132 ceph-mon[31531]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T13:27:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:06 smithi132 ceph-mon[31531]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:06 smithi132 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:06.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-mon[31101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T13:27:06.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-mon[31101]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T13:27:06.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-mon[31101]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:06.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:06.791 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-mon[31101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:06 smithi087 ceph-mon[27999]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T13:27:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:06 smithi087 ceph-mon[27999]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T13:27:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:06 smithi087 ceph-mon[27999]: from='osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:06 smithi087 ceph-mon[27999]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:27:07.100 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:27:06.956+0000 7f886d147700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T13:27:07.100 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:27:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:27:06.964+0000 7f8864f32700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: from='osd.7 ' entity='osd.7' 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] boot 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T13:27:07.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114894 s, 913 MB/s 2024-04-22T13:27:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-22T13:27:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:27:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T13:27:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:07.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: from='osd.7 ' entity='osd.7' 2024-04-22T13:27:07.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] boot 2024-04-22T13:27:07.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T13:27:07.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: from='osd.7 ' entity='osd.7' 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: osd.7 [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] boot 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T13:27:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:27:08.425 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-04-22T13:27:08.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:27:08.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:08 smithi120 ceph-mon[31101]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T13:27:08.672 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:08.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:08 smithi132 ceph-mon[31531]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T13:27:08.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:27:08.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:08.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:08 smithi087 ceph-mon[27999]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T13:27:09.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:09 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi120 2024-04-22T13:27:09.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:09 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 5696M 2024-04-22T13:27:09.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:09 smithi132 ceph-mon[31531]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:09.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:09 smithi132 ceph-mon[31531]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T13:27:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:09 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi120 2024-04-22T13:27:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:09 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 5696M 2024-04-22T13:27:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:09 smithi120 ceph-mon[31101]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:09 smithi120 ceph-mon[31101]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T13:27:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:09 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi120 2024-04-22T13:27:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:09 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 5696M 2024-04-22T13:27:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:09 smithi087 ceph-mon[27999]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:09 smithi087 ceph-mon[27999]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1224373367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 msgr2=0x7f3b94109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1224373367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b94109e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00b0a0 tx=0x7f3b8c00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1224373367 shutdown_connections 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1224373367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b94109e00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1224373367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b941019b0 0x7f3b94101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1224373367 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b94102bb0 0x7f3b94103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.415+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1224373367 >> 172.21.15.132:0/1224373367 conn(0x7f3b940fcf70 msgr2=0x7f3b940ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1224373367 shutdown_connections 2024-04-22T13:27:10.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1224373367 wait complete. 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 Processor -- start 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- start start 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b941019b0 0x7f3b94105940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 0x7f3b94103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9410d4a0 con 0x7f3b941019b0 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9410d320 con 0x7f3b94102bb0 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.416+0000 7f3b9b76e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9410d620 con 0x7f3b94103550 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52988/0 (socket says 172.21.15.132:52988) 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 -- 172.21.15.132:0/1462492316 learned_addr learned my addr 172.21.15.132:0/1462492316 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b98d09700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 0x7f3b94103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:10.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b9950a700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b941019b0 0x7f3b94105940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 -- 172.21.15.132:0/1462492316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 msgr2=0x7f3b94103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 0x7f3b94103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 -- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b941019b0 msgr2=0x7f3b94105940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b941019b0 0x7f3b94105940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 -- 172.21.15.132:0/1462492316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b94104d80 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b98d09700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 0x7f3b94103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b99d0b700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00b070 tx=0x7f3b8c0094f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c021070 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c007c80 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c01d450 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b941a2d40 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.417+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b941a31a0 con 0x7f3b94103550 2024-04-22T13:27:10.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.418+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b94069730 con 0x7f3b94103550 2024-04-22T13:27:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.419+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3b8c003a00 con 0x7f3b94103550 2024-04-22T13:27:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.420+0000 7f3b8a7fc700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 0x7f3b800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.420+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f3b8c00f020 con 0x7f3b94103550 2024-04-22T13:27:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.420+0000 7f3b9950a700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 0x7f3b800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:10.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.420+0000 7f3b9950a700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 0x7f3b800a6180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3b90005950 tx=0x7f3b9000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:27:10.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.424+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b8c02a020 con 0x7f3b94103550 2024-04-22T13:27:10.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:10.634+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3b940ff140 con 0x7f3b800a3cc0 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:11 smithi120 ceph-mon[31101]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:11 smithi132 ceph-mon[31531]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:11 smithi087 ceph-mon[27999]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:12.845 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:12 smithi132 ceph-mon[31531]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T13:27:12.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:12 smithi120 ceph-mon[31101]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T13:27:12.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:12 smithi087 ceph-mon[27999]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/2936737867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]': finished 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T13:27:13.853 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/2936737867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]': finished 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T13:27:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/2936737867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]: dispatch 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d2acda1-2a71-452c-acc3-5fbaf491f53c"}]': finished 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T13:27:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:14.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:14 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/3683003084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:14.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:14 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/3683003084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:14.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/3683003084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:15 smithi120 ceph-mon[31101]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:15.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:15 smithi132 ceph-mon[31531]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:15 smithi087 ceph-mon[27999]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:18.081 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:17 smithi132 ceph-mon[31531]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:17 smithi120 ceph-mon[31101]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:17 smithi087 ceph-mon[27999]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:19.895 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:19 smithi132 ceph-mon[31531]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:19.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:19 smithi087 ceph-mon[27999]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:20.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:19 smithi120 ceph-mon[31101]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:21.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:21 smithi132 ceph-mon[31531]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:21.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T13:27:21.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:22.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:21 smithi120 ceph-mon[31101]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:22.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T13:27:22.168 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:21 smithi087 ceph-mon[27999]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T13:27:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:22.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:22 smithi132 ceph-mon[31531]: Deploying daemon osd.8 on smithi132 2024-04-22T13:27:23.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:22 smithi120 ceph-mon[31101]: Deploying daemon osd.8 on smithi132 2024-04-22T13:27:23.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:22 smithi087 ceph-mon[27999]: Deploying daemon osd.8 on smithi132 2024-04-22T13:27:24.023 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:23 smithi132 ceph-mon[31531]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:23 smithi120 ceph-mon[31101]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:23 smithi087 ceph-mon[27999]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:26.166 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:25 smithi132 ceph-mon[31531]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:26.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:25 smithi120 ceph-mon[31101]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:25 smithi087 ceph-mon[27999]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:28.127 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:27 smithi132 ceph-mon[31531]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:28.127 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.127 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.127 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:27 smithi120 ceph-mon[31101]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:28.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:27 smithi087 ceph-mon[27999]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:28.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:28.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:30.646 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 8 on host 'smithi132' 2024-04-22T13:27:30.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.650+0000 7f3b8a7fc700 1 -- 172.21.15.132:0/1462492316 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3b940ff140 con 0x7f3b800a3cc0 2024-04-22T13:27:30.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 msgr2=0x7f3b800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:30.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 0x7f3b800a6180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3b90005950 tx=0x7f3b9000a300 comp rx=0 tx=0).stop 2024-04-22T13:27:30.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 msgr2=0x7f3b941044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00b070 tx=0x7f3b8c0094f0 comp rx=0 tx=0).stop 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 shutdown_connections 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3b800a3cc0 0x7f3b800a6180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b94103550 0x7f3b941044d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b941019b0 0x7f3b94105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 --2- 172.21.15.132:0/1462492316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b94102bb0 0x7f3b94103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 >> 172.21.15.132:0/1462492316 conn(0x7f3b940fcf70 msgr2=0x7f3b940fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 shutdown_connections 2024-04-22T13:27:30.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:27:30.653+0000 7f3b9b76e700 1 -- 172.21.15.132:0/1462492316 wait complete. 2024-04-22T13:27:31.379 DEBUG:teuthology.orchestra.run.smithi132:osd.8> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.8.service 2024-04-22T13:27:31.381 INFO:tasks.cephadm:Deploying osd.9 on smithi087 with /dev/vg_nvme/lv_1... 2024-04-22T13:27:31.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T13:27:31.630 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:27:31.689 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.689 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.689 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:31.689 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.689 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.690 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:31 smithi132 ceph-mon[31531]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:31 smithi087 ceph-mon[27999]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:31.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:31.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:31.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T13:27:31.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:31 smithi120 ceph-mon[31101]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:32.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:32 smithi087 ceph-mon[27999]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T13:27:32.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:32 smithi087 ceph-mon[27999]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:32.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:32 smithi087 ceph-mon[27999]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T13:27:32.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:32.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:32 smithi087 ceph-mon[27999]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:32.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:32 smithi132 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T13:27:32.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:32 smithi132 ceph-mon[31531]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:32.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:32 smithi132 ceph-mon[31531]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T13:27:32.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:32.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:32 smithi132 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:32 smithi120 ceph-mon[31101]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T13:27:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:32 smithi120 ceph-mon[31101]: from='osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:32 smithi120 ceph-mon[31101]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T13:27:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:32 smithi120 ceph-mon[31101]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:27:33.213 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:27:33.090+0000 7f66d772a700 -1 osd.8 0 waiting for initial osdmap 2024-04-22T13:27:33.213 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:27:33.101+0000 7f66cf515700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:33 smithi087 ceph-mon[27999]: from='osd.8 ' entity='osd.8' 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:33 smithi120 ceph-mon[31101]: from='osd.8 ' entity='osd.8' 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115131 s, 911 MB/s 2024-04-22T13:27:33.927 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-04-22T13:27:33.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:27:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T13:27:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-22T13:27:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:33 smithi132 ceph-mon[31531]: from='osd.8 ' entity='osd.8' 2024-04-22T13:27:34.585 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_1 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] boot 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T13:27:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:34.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:34 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.827 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:27:34.904 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:27:34.904 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] boot 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:34.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:34 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: osd.8 [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] boot 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:27:34.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:34.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:34.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:34.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:34 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:35.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:35 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi132 2024-04-22T13:27:35.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:35 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 5696M 2024-04-22T13:27:35.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:35 smithi120 ceph-mon[31101]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:35.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:35 smithi120 ceph-mon[31101]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T13:27:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:35 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-04-22T13:27:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:35 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 5696M 2024-04-22T13:27:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:35 smithi132 ceph-mon[31531]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:35 smithi132 ceph-mon[31531]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T13:27:35.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:35 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi132 2024-04-22T13:27:35.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:35 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 5696M 2024-04-22T13:27:35.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:35 smithi087 ceph-mon[27999]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:35.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:35 smithi087 ceph-mon[27999]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3988318252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc103550 msgr2=0x7f86dc109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3988318252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc103550 0x7f86dc109e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f86d400b0d0 tx=0x7f86d400b3e0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3988318252 shutdown_connections 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3988318252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc103550 0x7f86dc109e00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3988318252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc1019b0 0x7f86dc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3988318252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc102bb0 0x7f86dc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3988318252 >> 172.21.15.87:0/3988318252 conn(0x7f86dc0fcf70 msgr2=0x7f86dc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:27:36.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3988318252 shutdown_connections 2024-04-22T13:27:36.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.583+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3988318252 wait complete. 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.584+0000 7f86e0aa5700 1 Processor -- start 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.584+0000 7f86e0aa5700 1 -- start start 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc102bb0 0x7f86dc103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc103550 0x7f86dc1044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f86dc10d450 con 0x7f86dc1019b0 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f86dc10d2d0 con 0x7f86dc103550 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86e0aa5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f86dc10d5d0 con 0x7f86dc102bb0 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57660/0 (socket says 172.21.15.87:57660) 2024-04-22T13:27:36.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 -- 172.21.15.87:0/3332842574 learned_addr learned my addr 172.21.15.87:0/3332842574 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86d9d9b700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc102bb0 0x7f86dc103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86dad9d700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc103550 0x7f86dc1044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 -- 172.21.15.87:0/3332842574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc102bb0 msgr2=0x7f86dc103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc102bb0 0x7f86dc103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 -- 172.21.15.87:0/3332842574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc103550 msgr2=0x7f86dc1044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc103550 0x7f86dc1044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 -- 172.21.15.87:0/3332842574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86dc104d80 con 0x7f86dc1019b0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.585+0000 7f86da59c700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86d000d980 tx=0x7f86d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.586+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0017070 con 0x7f86dc1019b0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.586+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d000fd40 con 0x7f86dc1019b0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.586+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86dc1a2ce0 con 0x7f86dc1019b0 2024-04-22T13:27:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.586+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d000e3b0 con 0x7f86dc1019b0 2024-04-22T13:27:36.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.586+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86dc1a31a0 con 0x7f86dc1019b0 2024-04-22T13:27:36.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.587+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86dc069730 con 0x7f86dc1019b0 2024-04-22T13:27:36.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.588+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f86d000f890 con 0x7f86dc1019b0 2024-04-22T13:27:36.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.589+0000 7f86cb7fe700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 0x7f86c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:27:36.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.589+0000 7f86d9d9b700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 0x7f86c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:27:36.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.589+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f86d00cf4b0 con 0x7f86dc1019b0 2024-04-22T13:27:36.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.589+0000 7f86d9d9b700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 0x7f86c40a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00ac50 tx=0x7f86cc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:27:36.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.592+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f86d009c840 con 0x7f86dc1019b0 2024-04-22T13:27:36.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:36.802+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f86dc0ff160 con 0x7f86c40a3c40 2024-04-22T13:27:37.560 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T13:27:37.560 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:37.560 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:37.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:37.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:37.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:27:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:38.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:38 smithi087 ceph-mon[27999]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T13:27:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:38 smithi120 ceph-mon[31101]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T13:27:38.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:38 smithi132 ceph-mon[31531]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T13:27:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:39 smithi087 ceph-mon[27999]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T13:27:39.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:39 smithi120 ceph-mon[31101]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T13:27:39.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:39 smithi132 ceph-mon[31531]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T13:27:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]: dispatch 2024-04-22T13:27:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]': finished 2024-04-22T13:27:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:40 smithi120 ceph-mon[31101]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T13:27:40.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:40 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:40.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]: dispatch 2024-04-22T13:27:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]': finished 2024-04-22T13:27:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:40 smithi132 ceph-mon[31531]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T13:27:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:40 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]: dispatch 2024-04-22T13:27:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/471879014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07f9c4fd-b4f0-48db-b5de-05dd6c55b890"}]': finished 2024-04-22T13:27:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:40 smithi087 ceph-mon[27999]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T13:27:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:41.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1039852870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:41.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:41 smithi120 ceph-mon[31101]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:27:41.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1039852870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:41.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:41 smithi132 ceph-mon[31531]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:27:41.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1039852870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:27:41.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:41 smithi087 ceph-mon[27999]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T13:27:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:43 smithi120 ceph-mon[31101]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T13:27:44.188 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:43 smithi087 ceph-mon[27999]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T13:27:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:43 smithi132 ceph-mon[31531]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T13:27:46.148 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:45 smithi087 ceph-mon[27999]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:27:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:45 smithi120 ceph-mon[31101]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:27:46.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:45 smithi132 ceph-mon[31531]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T13:27:47.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:47 smithi087 ceph-mon[27999]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:27:47.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T13:27:47.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:48.030 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:47 smithi132 ceph-mon[31531]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:27:48.030 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T13:27:48.030 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:47 smithi120 ceph-mon[31101]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T13:27:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T13:27:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:49.071 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:48 smithi120 ceph-mon[31101]: Deploying daemon osd.9 on smithi087 2024-04-22T13:27:49.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:48 smithi132 ceph-mon[31531]: Deploying daemon osd.9 on smithi087 2024-04-22T13:27:49.235 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:48 smithi087 ceph-mon[27999]: Deploying daemon osd.9 on smithi087 2024-04-22T13:27:49.895 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:49 smithi087 ceph-mon[27999]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:50.083 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:49 smithi120 ceph-mon[31101]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:49 smithi132 ceph-mon[31531]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:51.898 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:51 smithi087 ceph-mon[27999]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:51 smithi120 ceph-mon[31101]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:52.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:51 smithi132 ceph-mon[31531]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:53 smithi120 ceph-mon[31101]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:53 smithi132 ceph-mon[31531]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:54.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:53 smithi087 ceph-mon[27999]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:54 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:54 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:54 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:54 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:54 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:54 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:55.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:54 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:54 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:55.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:54 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:56.168 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:55 smithi087 ceph-mon[27999]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:55 smithi120 ceph-mon[31101]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:55 smithi132 ceph-mon[31531]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:57.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.866+0000 7f86cb7fe700 1 -- 172.21.15.87:0/3332842574 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f86dc0ff160 con 0x7f86c40a3c40 2024-04-22T13:27:57.863 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 9 on host 'smithi087' 2024-04-22T13:27:57.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 msgr2=0x7f86c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 0x7f86c40a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00ac50 tx=0x7f86cc00a380 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 msgr2=0x7f86dc105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86d000d980 tx=0x7f86d000dc90 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 shutdown_connections 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86c40a3c40 0x7f86c40a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86dc103550 0x7f86dc1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86dc1019b0 0x7f86dc105940 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 --2- 172.21.15.87:0/3332842574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86dc102bb0 0x7f86dc103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 >> 172.21.15.87:0/3332842574 conn(0x7f86dc0fcf70 msgr2=0x7f86dc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.868+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 shutdown_connections 2024-04-22T13:27:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:27:57.869+0000 7f86e0aa5700 1 -- 172.21.15.87:0/3332842574 wait complete. 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T13:27:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:57 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:57 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:57 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:27:58.672 DEBUG:teuthology.orchestra.run.smithi087:osd.9> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.9.service 2024-04-22T13:27:58.674 INFO:tasks.cephadm:Deploying osd.10 on smithi120 with /dev/vg_nvme/lv_1... 2024-04-22T13:27:58.675 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T13:27:58.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:58 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:58.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:58 smithi087 ceph-mon[27999]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T13:27:58.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:58 smithi087 ceph-mon[27999]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T13:27:58.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:58 smithi087 ceph-mon[27999]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:27:58.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:58 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:58.919 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:27:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:58 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:58 smithi120 ceph-mon[31101]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T13:27:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:58 smithi120 ceph-mon[31101]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T13:27:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:58 smithi120 ceph-mon[31101]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:27:59.040 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:58 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:58 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:27:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:58 smithi132 ceph-mon[31531]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T13:27:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:58 smithi132 ceph-mon[31531]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T13:27:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:58 smithi132 ceph-mon[31531]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-04-22T13:27:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:58 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:27:59.737 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:27:59.453+0000 7fc3d2739700 -1 osd.9 0 waiting for initial osdmap 2024-04-22T13:27:59.737 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:27:59.463+0000 7fc3ca524700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:27:59 smithi120 ceph-mon[31101]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:27:59 smithi132 ceph-mon[31531]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:27:59 smithi087 ceph-mon[27999]: from='osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844]' entity='osd.9' 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] boot 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T13:28:00.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:00 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:00.940 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T13:28:00.940 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T13:28:00.941 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-04-22T13:28:00.941 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-04-22T13:28:00.941 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111693 s, 939 MB/s 2024-04-22T13:28:00.941 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] boot 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T13:28:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:00 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: osd.9 [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] boot 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T13:28:01.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:00 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:28:01.586 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_1 2024-04-22T13:28:01.833 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:28:01.851 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:01 smithi120 ceph-mon[31101]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:01.851 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:01 smithi120 ceph-mon[31101]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T13:28:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:01 smithi087 ceph-mon[27999]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:01 smithi087 ceph-mon[27999]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T13:28:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:01 smithi132 ceph-mon[31531]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-22T13:28:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:01 smithi132 ceph-mon[31531]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T13:28:03.368 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi087 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 4272M 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.369 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:03 smithi120 ceph-mon[31101]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi087 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 4272M 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:03 smithi132 ceph-mon[31531]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi087 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 4272M 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:03 smithi087 ceph-mon[27999]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 -- 172.21.15.120:0/3982487586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf901019a0 msgr2=0x7fdf90101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 --2- 172.21.15.120:0/3982487586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf901019a0 0x7fdf90101dc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf840099c0 tx=0x7fdf84009cd0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 -- 172.21.15.120:0/3982487586 shutdown_connections 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 --2- 172.21.15.120:0/3982487586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf90103540 0x7fdf90109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 --2- 172.21.15.120:0/3982487586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf901019a0 0x7fdf90101dc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 --2- 172.21.15.120:0/3982487586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf90102ba0 0x7fdf90103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 -- 172.21.15.120:0/3982487586 >> 172.21.15.120:0/3982487586 conn(0x7fdf900fcf40 msgr2=0x7fdf900ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 -- 172.21.15.120:0/3982487586 shutdown_connections 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.609+0000 7fdf96f29700 1 -- 172.21.15.120:0/3982487586 wait complete. 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 Processor -- start 2024-04-22T13:28:03.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 -- start start 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 0x7fdf90105ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf90102ba0 0x7fdf90104170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010d430 con 0x7fdf901019a0 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010d2b0 con 0x7fdf90103540 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf96f29700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010d5b0 con 0x7fdf90102ba0 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:47996/0 (socket says 172.21.15.120:47996) 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 -- 172.21.15.120:0/4207652302 learned_addr learned my addr 172.21.15.120:0/4207652302 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf94cc5700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 0x7fdf90105ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:03.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 -- 172.21.15.120:0/4207652302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf90102ba0 msgr2=0x7fdf90104170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf90102ba0 0x7fdf90104170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 -- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 msgr2=0x7fdf90105ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 0x7fdf90105ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 -- 172.21.15.120:0/4207652302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf90104f80 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf94cc5700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 0x7fdf90105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf954c6700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf88007ee0 tx=0x7fdf8800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800dbd0 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.610+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8800dd70 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.611+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf901a2dc0 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.611+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88005a30 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.611+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901a3270 con 0x7fdf90103540 2024-04-22T13:28:03.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.612+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf901a34a0 con 0x7fdf90103540 2024-04-22T13:28:03.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.613+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf8801a020 con 0x7fdf90103540 2024-04-22T13:28:03.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.613+0000 7fdf8dffb700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 0x7fdf800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:03.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.614+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fdf880cf320 con 0x7fdf90103540 2024-04-22T13:28:03.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.614+0000 7fdf94cc5700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 0x7fdf800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:03.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.614+0000 7fdf94cc5700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 0x7fdf800a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf84000c00 tx=0x7fdf84005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:03.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.618+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf88010070 con 0x7fdf90103540 2024-04-22T13:28:03.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:03.824+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdf9006ae30 con 0x7fdf800a3cc0 2024-04-22T13:28:04.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:04 smithi132 ceph-mon[31531]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:04.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:04 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:04.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:04 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:04.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:04 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:04.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:04 smithi087 ceph-mon[27999]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:04.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:04 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:04.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:04 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:04.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:04 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:04.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:04 smithi120 ceph-mon[31101]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:04.540 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:04 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:04.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:04 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:04.541 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:04 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:05.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:05 smithi120 ceph-mon[31101]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:05.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:05 smithi132 ceph-mon[31531]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:05.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:05 smithi087 ceph-mon[27999]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/1406295908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:06 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:06 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]': finished 2024-04-22T13:28:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:06 smithi087 ceph-mon[27999]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T13:28:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:07.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:06 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/1406295908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:07.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:06 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:07.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:06 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]': finished 2024-04-22T13:28:07.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:06 smithi120 ceph-mon[31101]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T13:28:07.061 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:06 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/1406295908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:06 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]: dispatch 2024-04-22T13:28:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:06 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a596e14d-82e2-4b4c-8af0-21f73df9b9d2"}]': finished 2024-04-22T13:28:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:06 smithi132 ceph-mon[31531]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T13:28:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:07 smithi087 ceph-mon[27999]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:07 smithi087 ceph-mon[27999]: from='client.? 172.21.15.120:0/3148403340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:07 smithi120 ceph-mon[31101]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:07 smithi120 ceph-mon[31101]: from='client.? 172.21.15.120:0/3148403340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:07 smithi132 ceph-mon[31531]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:07 smithi132 ceph-mon[31531]: from='client.? 172.21.15.120:0/3148403340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:09 smithi087 ceph-mon[27999]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:09 smithi120 ceph-mon[31101]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:09 smithi132 ceph-mon[31531]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:11.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:11 smithi120 ceph-mon[31101]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:11 smithi132 ceph-mon[31531]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:11 smithi087 ceph-mon[27999]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:14.000 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:13 smithi120 ceph-mon[31101]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:14.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:13 smithi132 ceph-mon[31531]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:14.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:13 smithi087 ceph-mon[27999]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:14.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T13:28:14.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:14 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T13:28:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:14 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:15.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T13:28:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:14 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:16.076 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:15 smithi120 ceph-mon[31101]: Deploying daemon osd.10 on smithi120 2024-04-22T13:28:16.077 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:15 smithi120 ceph-mon[31101]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:15 smithi132 ceph-mon[31531]: Deploying daemon osd.10 on smithi120 2024-04-22T13:28:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:15 smithi132 ceph-mon[31531]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:16.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:15 smithi087 ceph-mon[27999]: Deploying daemon osd.10 on smithi120 2024-04-22T13:28:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:15 smithi087 ceph-mon[27999]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:17.856 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:17 smithi120 ceph-mon[31101]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:18.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:17 smithi132 ceph-mon[31531]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:17 smithi087 ceph-mon[27999]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:20.052 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:19 smithi120 ceph-mon[31101]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:20.052 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.052 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.052 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:19 smithi132 ceph-mon[31531]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:20.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:19 smithi087 ceph-mon[27999]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:20.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:20.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:21 smithi120 ceph-mon[31101]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:22.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:21 smithi132 ceph-mon[31531]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:22.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:21 smithi087 ceph-mon[27999]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:23.235 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 10 on host 'smithi120' 2024-04-22T13:28:23.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.238+0000 7fdf8dffb700 1 -- 172.21.15.120:0/4207652302 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdf9006ae30 con 0x7fdf800a3cc0 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.241+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 msgr2=0x7fdf800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.241+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 0x7fdf800a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf84000c00 tx=0x7fdf84005fd0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.241+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 msgr2=0x7fdf901046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.241+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf88007ee0 tx=0x7fdf8800d3b0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 shutdown_connections 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf800a3cc0 0x7fdf800a6180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf90103540 0x7fdf901046b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf901019a0 0x7fdf90105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 --2- 172.21.15.120:0/4207652302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf90102ba0 0x7fdf90104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:23.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 >> 172.21.15.120:0/4207652302 conn(0x7fdf900fcf40 msgr2=0x7fdf9006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:23.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 shutdown_connections 2024-04-22T13:28:23.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-04-22T13:28:23.242+0000 7fdf96f29700 1 -- 172.21.15.120:0/4207652302 wait complete. 2024-04-22T13:28:23.888 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:23 smithi120 ceph-mon[31101]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:23.888 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:23 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:23.888 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:23 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:23.888 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:23 smithi120 ceph-mon[31101]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:23.888 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:23 smithi120 ceph-mon[31101]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:23.915 DEBUG:teuthology.orchestra.run.smithi120:osd.10> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.10.service 2024-04-22T13:28:23.917 INFO:tasks.cephadm:Deploying osd.11 on smithi132 with /dev/vg_nvme/lv_1... 2024-04-22T13:28:23.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T13:28:24.163 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:28:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:23 smithi132 ceph-mon[31531]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:23 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:23 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:23 smithi132 ceph-mon[31531]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:24.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:23 smithi132 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:23 smithi087 ceph-mon[27999]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:23 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:23 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:23 smithi087 ceph-mon[27999]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:24.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:23 smithi087 ceph-mon[27999]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-mon[31101]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:25.421 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:28:25.223+0000 7fdd5b70c700 -1 osd.10 0 waiting for initial osdmap 2024-04-22T13:28:25.421 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:28:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:28:25.235+0000 7fdd574ff700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:25 smithi132 ceph-mon[31531]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-04-22T13:28:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:25 smithi087 ceph-mon[27999]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114273 s, 918 MB/s 2024-04-22T13:28:26.369 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-22T13:28:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:26 smithi087 ceph-mon[27999]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:28:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:26 smithi087 ceph-mon[27999]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T13:28:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:26 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:26 smithi120 ceph-mon[31101]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:28:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:26 smithi120 ceph-mon[31101]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T13:28:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:26 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:26 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:26.705 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:26 smithi132 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-04-22T13:28:26.705 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:26 smithi132 ceph-mon[31531]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T13:28:26.705 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:26.706 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:26 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:27.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_1 2024-04-22T13:28:27.293 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] boot 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:27.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:27 smithi132 ceph-mon[31531]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] boot 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:27 smithi120 ceph-mon[31101]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: osd.10 [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] boot 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:27.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:27 smithi087 ceph-mon[27999]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T13:28:28.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi120 2024-04-22T13:28:28.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 4272M 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:28 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi120 2024-04-22T13:28:28.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 4272M 2024-04-22T13:28:28.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:28 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi120 2024-04-22T13:28:28.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.990 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 4272M 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:28.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:28 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.100+0000 7f5473ef9700 1 -- 172.21.15.132:0/4034810195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 msgr2=0x7f546c103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.100+0000 7f5473ef9700 1 --2- 172.21.15.132:0/4034810195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c103020 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f545c009990 tx=0x7f545c009ca0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.100+0000 7f5473ef9700 1 -- 172.21.15.132:0/4034810195 shutdown_connections 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.100+0000 7f5473ef9700 1 --2- 172.21.15.132:0/4034810195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c103560 0x7f546c109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.100+0000 7f5473ef9700 1 --2- 172.21.15.132:0/4034810195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c1019c0 0x7f546c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 --2- 172.21.15.132:0/4034810195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c103020 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- 172.21.15.132:0/4034810195 >> 172.21.15.132:0/4034810195 conn(0x7f546c0fcfa0 msgr2=0x7f546c0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- 172.21.15.132:0/4034810195 shutdown_connections 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- 172.21.15.132:0/4034810195 wait complete. 2024-04-22T13:28:29.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 Processor -- start 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- start start 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c1019c0 0x7f546c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c103560 0x7f546c1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f546c10d4b0 con 0x7f546c103560 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f546c10d330 con 0x7f546c1019c0 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5473ef9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f546c10d630 con 0x7f546c102bc0 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5471494700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.101+0000 7f5471494700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59188/0 (socket says 172.21.15.132:59188) 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 -- 172.21.15.132:0/213766093 learned_addr learned my addr 172.21.15.132:0/213766093 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471c95700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c1019c0 0x7f546c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5472496700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c103560 0x7f546c1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 -- 172.21.15.132:0/213766093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c1019c0 msgr2=0x7f546c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:29.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c1019c0 0x7f546c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 -- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c103560 msgr2=0x7f546c1a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c103560 0x7f546c1a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 -- 172.21.15.132:0/213766093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c1a08e0 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5471494700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f545c003790 tx=0x7f545c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c027070 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f545c00bd90 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c022dd0 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546c1a0b70 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.102+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546c1a0fc0 con 0x7f546c102bc0 2024-04-22T13:28:29.102 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.103+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f546c1a1190 con 0x7f546c102bc0 2024-04-22T13:28:29.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.104+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f545c02a070 con 0x7f546c102bc0 2024-04-22T13:28:29.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.105+0000 7f5462ffd700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 0x7f54580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:29.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.105+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f545c0d0fc0 con 0x7f546c102bc0 2024-04-22T13:28:29.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.105+0000 7f5471c95700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 0x7f54580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:29.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.105+0000 7f5471c95700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 0x7f54580a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f546c199b40 tx=0x7f546800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:29.105 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.108+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f545c031020 con 0x7f546c102bc0 2024-04-22T13:28:29.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:29.318+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f546c1a1360 con 0x7f54580a3c40 2024-04-22T13:28:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:29 smithi132 ceph-mon[31531]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:29 smithi132 ceph-mon[31531]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T13:28:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:29 smithi120 ceph-mon[31101]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:29 smithi120 ceph-mon[31101]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T13:28:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:29 smithi087 ceph-mon[27999]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:29 smithi087 ceph-mon[27999]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T13:28:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T13:28:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T13:28:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:30.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:30 smithi120 ceph-mon[31101]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:30.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:30 smithi120 ceph-mon[31101]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T13:28:30.940 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:30 smithi132 ceph-mon[31531]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:30.940 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:30 smithi132 ceph-mon[31531]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T13:28:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:30 smithi087 ceph-mon[27999]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:28:30.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:30 smithi087 ceph-mon[27999]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T13:28:31.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:31 smithi120 ceph-mon[31101]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:31 smithi132 ceph-mon[31531]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:31.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:31 smithi087 ceph-mon[27999]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:32 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/141326217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:32 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:32 smithi120 ceph-mon[31101]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]': finished 2024-04-22T13:28:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:32 smithi120 ceph-mon[31101]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T13:28:32.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:32 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:32.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:32 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/141326217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:32 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:32 smithi132 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]': finished 2024-04-22T13:28:32.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:32 smithi132 ceph-mon[31531]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T13:28:32.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:32 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:32.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:32 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/141326217' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:32 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]: dispatch 2024-04-22T13:28:32.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:32 smithi087 ceph-mon[27999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8cbc34d-da2e-48c0-a354-f30062b39ade"}]': finished 2024-04-22T13:28:32.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:32 smithi087 ceph-mon[27999]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T13:28:32.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:32 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:33 smithi120 ceph-mon[31101]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T13:28:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:33 smithi120 ceph-mon[31101]: from='client.? 172.21.15.132:0/1912837933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:33 smithi132 ceph-mon[31531]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T13:28:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:33 smithi132 ceph-mon[31531]: from='client.? 172.21.15.132:0/1912837933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:33.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:33 smithi087 ceph-mon[27999]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T13:28:33.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:33 smithi087 ceph-mon[27999]: from='client.? 172.21.15.132:0/1912837933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T13:28:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:35 smithi120 ceph-mon[31101]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:28:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:35 smithi132 ceph-mon[31531]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:28:36.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:35 smithi087 ceph-mon[27999]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T13:28:37.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:37 smithi132 ceph-mon[31531]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T13:28:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:37 smithi120 ceph-mon[31101]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T13:28:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:37 smithi087 ceph-mon[27999]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T13:28:40.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:39 smithi120 ceph-mon[31101]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-22T13:28:40.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:39 smithi132 ceph-mon[31531]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-22T13:28:40.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:39 smithi087 ceph-mon[27999]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-22T13:28:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:40 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T13:28:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:40 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:40 smithi120 ceph-mon[31101]: Deploying daemon osd.11 on smithi132 2024-04-22T13:28:41.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:40 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T13:28:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:40 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:41.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:40 smithi132 ceph-mon[31531]: Deploying daemon osd.11 on smithi132 2024-04-22T13:28:41.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T13:28:41.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:40 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:41.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:40 smithi087 ceph-mon[27999]: Deploying daemon osd.11 on smithi132 2024-04-22T13:28:42.097 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:41 smithi132 ceph-mon[31531]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:28:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:41 smithi120 ceph-mon[31101]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:28:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:41 smithi087 ceph-mon[27999]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T13:28:44.110 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:43 smithi132 ceph-mon[31531]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T13:28:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:43 smithi120 ceph-mon[31101]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T13:28:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:43 smithi087 ceph-mon[27999]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T13:28:45.902 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:45 smithi132 ceph-mon[31531]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:45 smithi120 ceph-mon[31101]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:45 smithi087 ceph-mon[27999]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:47 smithi120 ceph-mon[31101]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:47.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:47.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:47 smithi132 ceph-mon[31531]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:47.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:47.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:47.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:47 smithi087 ceph-mon[27999]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:49.683 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 11 on host 'smithi132' 2024-04-22T13:28:49.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.686+0000 7f5462ffd700 1 -- 172.21.15.132:0/213766093 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f546c1a1360 con 0x7f54580a3c40 2024-04-22T13:28:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 msgr2=0x7f54580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 0x7f54580a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f546c199b40 tx=0x7f546800a380 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 msgr2=0x7f546c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f545c003790 tx=0x7f545c009580 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 shutdown_connections 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f54580a3c40 0x7f54580a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f546c103560 0x7f546c1a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546c1019c0 0x7f546c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 --2- 172.21.15.132:0/213766093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f546c102bc0 0x7f546c1991a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 >> 172.21.15.132:0/213766093 conn(0x7f546c0fcfa0 msgr2=0x7f546c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 shutdown_connections 2024-04-22T13:28:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:28:49.689+0000 7f5473ef9700 1 -- 172.21.15.132:0/213766093 wait complete. 2024-04-22T13:28:49.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:49 smithi132 ceph-mon[31531]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:49 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:49 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:49 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:49 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:49 smithi120 ceph-mon[31101]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:50.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:49 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.123 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:49 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:49 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:49 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:49 smithi087 ceph-mon[27999]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:49 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:50.414 DEBUG:teuthology.orchestra.run.smithi132:osd.11> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.11.service 2024-04-22T13:28:50.417 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-22T13:28:50.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T13:28:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:28:50.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:50 smithi132 ceph-mon[31531]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:50.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:50 smithi132 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:51.055 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:50 smithi087 ceph-mon[27999]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:51.056 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:50 smithi087 ceph-mon[27999]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:50 smithi120 ceph-mon[31101]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:51.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:50 smithi120 ceph-mon[31101]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:51.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:51.926 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:28:51.805+0000 7ff206032700 -1 osd.11 0 waiting for initial osdmap 2024-04-22T13:28:51.926 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:28:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:28:51.822+0000 7ff201e25700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T13:28:52.125 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:52.125 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T13:28:52.125 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T13:28:52.125 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:52.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:52.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:51 smithi087 ceph-mon[27999]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: from='osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:51 smithi120 ceph-mon[31101]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-22T13:28:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 -- 172.21.15.87:0/3518276873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658104650 msgr2=0x7f265810af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 --2- 172.21.15.87:0/3518276873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658104650 0x7f265810af00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f26480099c0 tx=0x7f2648009cd0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 -- 172.21.15.87:0/3518276873 shutdown_connections 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 --2- 172.21.15.87:0/3518276873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658104650 0x7f265810af00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 --2- 172.21.15.87:0/3518276873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658102ab0 0x7f2658102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 --2- 172.21.15.87:0/3518276873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658103cb0 0x7f2658104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 -- 172.21.15.87:0/3518276873 >> 172.21.15.87:0/3518276873 conn(0x7f26580fe050 msgr2=0x7f2658100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 -- 172.21.15.87:0/3518276873 shutdown_connections 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 -- 172.21.15.87:0/3518276873 wait complete. 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.526+0000 7f265d19c700 1 Processor -- start 2024-04-22T13:28:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 -- start start 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658102ab0 0x7f26580767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 0x7f2658075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f265810e5e0 con 0x7f2658103cb0 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f265810e460 con 0x7f2658104650 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265d19c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f265810e760 con 0x7f2658102ab0 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40638/0 (socket says 172.21.15.87:40638) 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 -- 172.21.15.87:0/3995219743 learned_addr learned my addr 172.21.15.87:0/3995219743 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f2656d9d700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658102ab0 0x7f26580767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265759e700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 0x7f2658075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 -- 172.21.15.87:0/3995219743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658102ab0 msgr2=0x7f26580767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658102ab0 0x7f26580767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 -- 172.21.15.87:0/3995219743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 msgr2=0x7f2658075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 0x7f2658075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265659c700 1 -- 172.21.15.87:0/3995219743 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2658075c20 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.527+0000 7f265759e700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 0x7f2658075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f265659c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f264c00d980 tx=0x7f264c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c017070 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26581a2d50 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f264c005020 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c009b50 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.528+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26581a31b0 con 0x7f2658103cb0 2024-04-22T13:28:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.529+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2658069730 con 0x7f2658103cb0 2024-04-22T13:28:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.530+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f264c0051d0 con 0x7f2658103cb0 2024-04-22T13:28:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.531+0000 7f263ffff700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 0x7f26400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.531+0000 7f2656d9d700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 0x7f26400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.531+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f264c0cfd60 con 0x7f2658103cb0 2024-04-22T13:28:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.531+0000 7f2656d9d700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 0x7f26400a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2658103b10 tx=0x7f265000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:52.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.536+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f264c020020 con 0x7f2658103cb0 2024-04-22T13:28:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.733+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f265802e190 con 0x7f2658103cb0 2024-04-22T13:28:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.734+0000 7f263ffff700 1 -- 172.21.15.87:0/3995219743 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f264c09cd00 con 0x7f2658103cb0 2024-04-22T13:28:52.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 msgr2=0x7f26400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 0x7f26400a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2658103b10 tx=0x7f265000d040 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 msgr2=0x7f2658074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f264c00d980 tx=0x7f264c00dc90 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 shutdown_connections 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26400a3c40 0x7f26400a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2658104650 0x7f2658075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2658102ab0 0x7f26580767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 --2- 172.21.15.87:0/3995219743 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658103cb0 0x7f2658074e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 >> 172.21.15.87:0/3995219743 conn(0x7f26580fe050 msgr2=0x7f2658106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:52.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 shutdown_connections 2024-04-22T13:28:52.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:52.736+0000 7f265d19c700 1 -- 172.21.15.87:0/3995219743 wait complete. 2024-04-22T13:28:52.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:52 smithi087 ceph-mon[27999]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:28:52.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:52 smithi087 ceph-mon[27999]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T13:28:52.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:52.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:52 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:52.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:52 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3995219743' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:53.072 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:52 smithi132 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:28:53.072 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:52 smithi132 ceph-mon[31531]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T13:28:53.072 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:52 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:53.072 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:52 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:53.072 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:52 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3995219743' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:52 smithi120 ceph-mon[31101]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-22T13:28:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:52 smithi120 ceph-mon[31101]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T13:28:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:52 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:52 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:52 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3995219743' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:53.457 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1713792506,"num_in_osds":12,"osd_in_since":1713792512,"num_remapped_pgs":0} 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: purged_snaps scrub starts 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: purged_snaps scrub ok 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] boot 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:53 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: purged_snaps scrub starts 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: purged_snaps scrub ok 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] boot 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T13:28:54.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:53 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: purged_snaps scrub starts 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: purged_snaps scrub ok 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: osd.11 [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] boot 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:28:54.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:53 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:28:54.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T13:28:54.701 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:28:55.097 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:54 smithi087 ceph-mon[27999]: Detected new or changed devices on smithi132 2024-04-22T13:28:55.098 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:54 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 4272M 2024-04-22T13:28:55.098 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:54 smithi087 ceph-mon[27999]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T13:28:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:54 smithi120 ceph-mon[31101]: Detected new or changed devices on smithi132 2024-04-22T13:28:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:54 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 4272M 2024-04-22T13:28:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:54 smithi120 ceph-mon[31101]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T13:28:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:54 smithi132 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-04-22T13:28:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:54 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 4272M 2024-04-22T13:28:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:54 smithi132 ceph-mon[31531]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T13:28:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:55 smithi120 ceph-mon[31101]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:56.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:55 smithi087 ceph-mon[27999]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:55 smithi132 ceph-mon[31531]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 -- 172.21.15.87:0/1686245877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4103530 msgr2=0x7f0ff4109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 --2- 172.21.15.87:0/1686245877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4103530 0x7f0ff4109de0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40099c0 tx=0x7f0fe4009cd0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 -- 172.21.15.87:0/1686245877 shutdown_connections 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 --2- 172.21.15.87:0/1686245877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4103530 0x7f0ff4109de0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 --2- 172.21.15.87:0/1686245877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4101990 0x7f0ff4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 --2- 172.21.15.87:0/1686245877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4102b90 0x7f0ff4102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 -- 172.21.15.87:0/1686245877 >> 172.21.15.87:0/1686245877 conn(0x7f0ff40fcf30 msgr2=0x7f0ff40ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 -- 172.21.15.87:0/1686245877 shutdown_connections 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.387+0000 7f0ffa177700 1 -- 172.21.15.87:0/1686245877 wait complete. 2024-04-22T13:28:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 Processor -- start 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 -- start start 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4101990 0x7f0ff406bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 0x7f0ff419ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410d520 con 0x7f0ff4102b90 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410d3a0 con 0x7f0ff4103530 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ffa177700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410d6a0 con 0x7f0ff4101990 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40654/0 (socket says 172.21.15.87:40654) 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 -- 172.21.15.87:0/2489530819 learned_addr learned my addr 172.21.15.87:0/2489530819 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff37fe700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4101990 0x7f0ff406bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff3fff700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 0x7f0ff419ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 -- 172.21.15.87:0/2489530819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4101990 msgr2=0x7f0ff406bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4101990 0x7f0ff406bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 -- 172.21.15.87:0/2489530819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 msgr2=0x7f0ff419ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 0x7f0ff419ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff2ffd700 1 -- 172.21.15.87:0/2489530819 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff41a08a0 con 0x7f0ff4102b90 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.388+0000 7f0ff3fff700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 0x7f0ff419ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ff2ffd700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800ba50 tx=0x7f0fe800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8019070 con 0x7f0ff4102b90 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fe8014440 con 0x7f0ff4102b90 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff41a0b90 con 0x7f0ff4102b90 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe80075d0 con 0x7f0ff4102b90 2024-04-22T13:28:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.389+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff41a1120 con 0x7f0ff4102b90 2024-04-22T13:28:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.390+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff4069730 con 0x7f0ff4102b90 2024-04-22T13:28:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.391+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0fe801c070 con 0x7f0ff4102b90 2024-04-22T13:28:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.391+0000 7f0ff0ff9700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 0x7f0fdc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.392+0000 7f0ff37fe700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 0x7f0fdc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:56.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.392+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0fe80cf820 con 0x7f0ff4102b90 2024-04-22T13:28:56.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.392+0000 7f0ff37fe700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 0x7f0fdc0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0fec006fd0 tx=0x7f0fec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.395+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fe809c6a0 con 0x7f0ff4102b90 2024-04-22T13:28:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.597+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0ff402e190 con 0x7f0ff4102b90 2024-04-22T13:28:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.597+0000 7f0ff0ff9700 1 -- 172.21.15.87:0/2489530819 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f0fe809bfa0 con 0x7f0ff4102b90 2024-04-22T13:28:56.593 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:28:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 msgr2=0x7f0fdc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 0x7f0fdc0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0fec006fd0 tx=0x7f0fec008040 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 msgr2=0x7f0ff419d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800ba50 tx=0x7f0fe800be10 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 shutdown_connections 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0fdc0a3cc0 0x7f0fdc0a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff4103530 0x7f0ff419ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ff4101990 0x7f0ff406bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 --2- 172.21.15.87:0/2489530819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ff4102b90 0x7f0ff419d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.599+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 >> 172.21.15.87:0/2489530819 conn(0x7f0ff40fcf30 msgr2=0x7f0ff4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.600+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 shutdown_connections 2024-04-22T13:28:56.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:56.600+0000 7f0ffa177700 1 -- 172.21.15.87:0/2489530819 wait complete. 2024-04-22T13:28:56.912 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2489530819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2489530819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2489530819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T13:28:57.385 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713792532,"num_in_osds":12,"osd_in_since":1713792512,"num_remapped_pgs":0} 2024-04-22T13:28:57.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T13:28:57.620 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:28:58.010 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:57 smithi087 ceph-mon[27999]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:58.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:57 smithi120 ceph-mon[31101]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:58.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:57 smithi132 ceph-mon[31531]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.430+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3704688696 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b4107e10 msgr2=0x7fa3b410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.430+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3704688696 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b4107e10 0x7fa3b410f990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00b0d0 tx=0x7fa3ac00b3e0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3704688696 shutdown_connections 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3704688696 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b4107e10 0x7fa3b410f990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3704688696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b41011f0 0x7fa3b41015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3704688696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3704688696 >> 172.21.15.87:0/3704688696 conn(0x7fa3b4077f40 msgr2=0x7fa3b40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3704688696 shutdown_connections 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3704688696 wait complete. 2024-04-22T13:28:59.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 Processor -- start 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.431+0000 7fa3b8a2c700 1 -- start start 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41026b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b4107e10 0x7fa3b4102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b4112cb0 con 0x7fa3b41011f0 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b4112b30 con 0x7fa3b4107e10 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b4112e30 con 0x7fa3b4101ba0 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40666/0 (socket says 172.21.15.87:40666) 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 -- 172.21.15.87:0/3187166918 learned_addr learned my addr 172.21.15.87:0/3187166918 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b2d9d700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b4107e10 0x7fa3b4102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b1d9b700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 -- 172.21.15.87:0/3187166918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 msgr2=0x7fa3b41026b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:59.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41026b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 -- 172.21.15.87:0/3187166918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b4107e10 msgr2=0x7fa3b4102bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b4107e10 0x7fa3b4102bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 -- 172.21.15.87:0/3187166918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b41a7660 con 0x7fa3b41011f0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b1d9b700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b259c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400d950 tx=0x7fa3a400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4016070 con 0x7fa3b41011f0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3a4009df0 con 0x7fa3b41011f0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b41a7950 con 0x7fa3b41011f0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.432+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400f3c0 con 0x7fa3b41011f0 2024-04-22T13:28:59.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.433+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b41a7e90 con 0x7fa3b41011f0 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.433+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b410b050 con 0x7fa3b41011f0 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.435+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa3a4019070 con 0x7fa3b41011f0 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.435+0000 7fa3a37fe700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 0x7fa39c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.435+0000 7fa3b1d9b700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 0x7fa39c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.435+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa3a40cfb10 con 0x7fa3b41011f0 2024-04-22T13:28:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.436+0000 7fa3b1d9b700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 0x7fa39c0a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4105580 tx=0x7fa3a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:28:59.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.439+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3a409c990 con 0x7fa3b41011f0 2024-04-22T13:28:59.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.637+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3b410aa30 con 0x7fa3b41011f0 2024-04-22T13:28:59.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.638+0000 7fa3a37fe700 1 -- 172.21.15.87:0/3187166918 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18475 (secure 0 0 0) 0x7fa3a409c290 con 0x7fa3b41011f0 2024-04-22T13:28:59.634 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:28:59.634 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":81,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:28:53.811375+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:28:59.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 msgr2=0x7fa39c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 0x7fa39c0a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4105580 tx=0x7fa3a8008040 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 msgr2=0x7fa3b4102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400d950 tx=0x7fa3a400dc60 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 shutdown_connections 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa39c0a3c40 0x7fa39c0a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b4107e10 0x7fa3b4102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa3b41011f0 0x7fa3b4102170 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 --2- 172.21.15.87:0/3187166918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b4101ba0 0x7fa3b41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 >> 172.21.15.87:0/3187166918 conn(0x7fa3b4077f40 msgr2=0x7fa3b40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 shutdown_connections 2024-04-22T13:28:59.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:28:59.640+0000 7fa3b8a2c700 1 -- 172.21.15.87:0/3187166918 wait complete. 2024-04-22T13:28:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:59 smithi132 ceph-mon[31531]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:28:59 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3187166918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:28:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:59 smithi087 ceph-mon[27999]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:28:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:28:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3187166918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:00.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:59 smithi120 ceph-mon[31101]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:00.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:28:59 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3187166918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:00.437 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:29:00.438 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T13:29:00.674 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:29:02.106 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:01 smithi087 ceph-mon[27999]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:01 smithi120 ceph-mon[31101]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:01 smithi132 ceph-mon[31531]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- 172.21.15.87:0/1083641273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234101960 msgr2=0x7f5234101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 --2- 172.21.15.87:0/1083641273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234101960 0x7f5234101de0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5224009990 tx=0x7f5224009ca0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- 172.21.15.87:0/1083641273 shutdown_connections 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 --2- 172.21.15.87:0/1083641273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 --2- 172.21.15.87:0/1083641273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234107960 0x7f5234107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 --2- 172.21.15.87:0/1083641273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234101960 0x7f5234101de0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- 172.21.15.87:0/1083641273 >> 172.21.15.87:0/1083641273 conn(0x7f52340789e0 msgr2=0x7f5234078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- 172.21.15.87:0/1083641273 shutdown_connections 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- 172.21.15.87:0/1083641273 wait complete. 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 Processor -- start 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 -- start start 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.543+0000 7f523bbad700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234101960 0x7f523419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5234111bf0 con 0x7f5234107960 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5234111a70 con 0x7f5234101960 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5234111d70 con 0x7f5234102320 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:02.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47554/0 (socket says 172.21.15.87:47554) 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 -- 172.21.15.87:0/1493293630 learned_addr learned my addr 172.21.15.87:0/1493293630 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f5239949700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234101960 0x7f523419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f5239148700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 -- 172.21.15.87:0/1493293630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 msgr2=0x7f523419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 -- 172.21.15.87:0/1493293630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234101960 msgr2=0x7f523419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234101960 0x7f523419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 -- 172.21.15.87:0/1493293630 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52341a5610 con 0x7f5234107960 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f5239148700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523a14a700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f522c00b970 tx=0x7f522c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c019070 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c00cde0 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52341a5840 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c007370 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.544+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52341a5da0 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.545+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5234069730 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.546+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f522c01c070 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.547+0000 7f522affd700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 0x7f52200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.547+0000 7f5239949700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 0x7f52200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.547+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f522c0cf390 con 0x7f5234107960 2024-04-22T13:29:02.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.547+0000 7f5239949700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 0x7f52200a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5230005950 tx=0x7f523000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:02.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.552+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f522c09c210 con 0x7f5234107960 2024-04-22T13:29:02.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.750+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f523402df80 con 0x7f5234107960 2024-04-22T13:29:02.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.751+0000 7f522affd700 1 -- 172.21.15.87:0/1493293630 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f522c09bb10 con 0x7f5234107960 2024-04-22T13:29:02.747 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:29:02.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 msgr2=0x7f52200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:02.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 0x7f52200a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5230005950 tx=0x7f523000a300 comp rx=0 tx=0).stop 2024-04-22T13:29:02.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 msgr2=0x7f52341a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:02.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f522c00b970 tx=0x7f522c00bc80 comp rx=0 tx=0).stop 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 shutdown_connections 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f52200a3cc0 0x7f52200a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5234107960 0x7f52341a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5234101960 0x7f523419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 --2- 172.21.15.87:0/1493293630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5234102320 0x7f523419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 >> 172.21.15.87:0/1493293630 conn(0x7f52340789e0 msgr2=0x7f5234104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 shutdown_connections 2024-04-22T13:29:02.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:02.753+0000 7f523bbad700 1 -- 172.21.15.87:0/1493293630 wait complete. 2024-04-22T13:29:03.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1493293630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:03.178 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1493293630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:03.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1493293630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:03.626 INFO:tasks.cephadm:Adding mds.a on smithi087 2024-04-22T13:29:03.626 INFO:tasks.cephadm:Adding mds.d on smithi087 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.g on smithi087 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.j on smithi087 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.b on smithi120 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.e on smithi120 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.h on smithi120 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.k on smithi120 2024-04-22T13:29:03.627 INFO:tasks.cephadm:Adding mds.c on smithi132 2024-04-22T13:29:03.628 INFO:tasks.cephadm:Adding mds.f on smithi132 2024-04-22T13:29:03.628 INFO:tasks.cephadm:Adding mds.i on smithi132 2024-04-22T13:29:03.628 INFO:tasks.cephadm:Adding mds.l on smithi132 2024-04-22T13:29:03.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l' 2024-04-22T13:29:03.876 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:29:03.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:03 smithi132 ceph-mon[31531]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:03 smithi120 ceph-mon[31101]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:04.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:03 smithi087 ceph-mon[27999]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.524+0000 7f1a3d219700 1 -- 172.21.15.132:0/1952230622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 msgr2=0x7f1a38101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.524+0000 7f1a3d219700 1 --2- 172.21.15.132:0/1952230622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a38101dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009990 tx=0x7f1a28009ca0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 -- 172.21.15.132:0/1952230622 shutdown_connections 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 --2- 172.21.15.132:0/1952230622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38103540 0x7f1a38109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 --2- 172.21.15.132:0/1952230622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a38101dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 --2- 172.21.15.132:0/1952230622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38102ba0 0x7f1a38103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 -- 172.21.15.132:0/1952230622 >> 172.21.15.132:0/1952230622 conn(0x7f1a380fcf40 msgr2=0x7f1a380ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 -- 172.21.15.132:0/1952230622 shutdown_connections 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 -- 172.21.15.132:0/1952230622 wait complete. 2024-04-22T13:29:05.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.525+0000 7f1a3d219700 1 Processor -- start 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 -- start start 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 0x7f1a3819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38103540 0x7f1a3819fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d490 con 0x7f1a38103540 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d310 con 0x7f1a38102ba0 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3d219700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d610 con 0x7f1a381019a0 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57768/0 (socket says 172.21.15.132:57768) 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 -- 172.21.15.132:0/3868134073 learned_addr learned my addr 172.21.15.132:0/3868134073 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3759e700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38103540 0x7f1a3819fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3659c700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 0x7f1a3819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 -- 172.21.15.132:0/3868134073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 msgr2=0x7f1a3819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:05.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 0x7f1a3819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 -- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38103540 msgr2=0x7f1a3819fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38103540 0x7f1a3819fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 -- 172.21.15.132:0/3868134073 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a381a08c0 con 0x7f1a381019a0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a3659c700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 0x7f1a3819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.526+0000 7f1a36d9d700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a28000c00 tx=0x7f1a28009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.527+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28027070 con 0x7f1a381019a0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.527+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a280041c0 con 0x7f1a381019a0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.527+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a381a0b50 con 0x7f1a381019a0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.527+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800f460 con 0x7f1a381019a0 2024-04-22T13:29:05.526 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.527+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381a0fa0 con 0x7f1a381019a0 2024-04-22T13:29:05.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.528+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3802df80 con 0x7f1a381019a0 2024-04-22T13:29:05.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.529+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1a2802a070 con 0x7f1a381019a0 2024-04-22T13:29:05.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.529+0000 7f1a1ffff700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 0x7f1a200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:05.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.530+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1a280d1620 con 0x7f1a381019a0 2024-04-22T13:29:05.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.530+0000 7f1a3659c700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 0x7f1a200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:05.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.530+0000 7f1a3659c700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 0x7f1a200a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005d90 tx=0x7f1a2c005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:05.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.536+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a28031070 con 0x7f1a381019a0 2024-04-22T13:29:05.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.741+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}) -- 0x7f1a38062f80 con 0x7f1a200a3c40 2024-04-22T13:29:05.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.750+0000 7f1a1ffff700 1 -- 172.21.15.132:0/3868134073 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f1a38062f80 con 0x7f1a200a3c40 2024-04-22T13:29:05.746 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mds.all update... 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 msgr2=0x7f1a200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 0x7f1a200a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005d90 tx=0x7f1a2c005d20 comp rx=0 tx=0).stop 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 msgr2=0x7f1a3806bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a28000c00 tx=0x7f1a28009580 comp rx=0 tx=0).stop 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 shutdown_connections 2024-04-22T13:29:05.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a200a3c40 0x7f1a200a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a38103540 0x7f1a3819fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a381019a0 0x7f1a3806bcf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 --2- 172.21.15.132:0/3868134073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a38102ba0 0x7f1a3819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 >> 172.21.15.132:0/3868134073 conn(0x7f1a380fcf40 msgr2=0x7f1a38105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 shutdown_connections 2024-04-22T13:29:05.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-04-22T13:29:05.751+0000 7f1a3d219700 1 -- 172.21.15.132:0/3868134073 wait complete. 2024-04-22T13:29:06.114 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:05 smithi132 ceph-mon[31531]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:06.114 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:06.115 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:06.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:05 smithi120 ceph-mon[31101]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:06.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:05 smithi087 ceph-mon[27999]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:06.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:06.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:06.449 DEBUG:teuthology.orchestra.run.smithi087:mds.a> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.a.service 2024-04-22T13:29:06.456 DEBUG:teuthology.orchestra.run.smithi087:mds.d> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.d.service 2024-04-22T13:29:06.499 DEBUG:teuthology.orchestra.run.smithi087:mds.g> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.g.service 2024-04-22T13:29:06.542 DEBUG:teuthology.orchestra.run.smithi087:mds.j> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.j.service 2024-04-22T13:29:06.584 DEBUG:teuthology.orchestra.run.smithi120:mds.b> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.b.service 2024-04-22T13:29:06.587 DEBUG:teuthology.orchestra.run.smithi120:mds.e> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.e.service 2024-04-22T13:29:06.630 DEBUG:teuthology.orchestra.run.smithi120:mds.h> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.h.service 2024-04-22T13:29:06.673 DEBUG:teuthology.orchestra.run.smithi120:mds.k> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.k.service 2024-04-22T13:29:06.716 DEBUG:teuthology.orchestra.run.smithi132:mds.c> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.c.service 2024-04-22T13:29:06.718 DEBUG:teuthology.orchestra.run.smithi132:mds.f> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.f.service 2024-04-22T13:29:06.762 DEBUG:teuthology.orchestra.run.smithi132:mds.i> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.i.service 2024-04-22T13:29:06.805 DEBUG:teuthology.orchestra.run.smithi132:mds.l> sudo journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.l.service 2024-04-22T13:29:06.848 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-22T13:29:06.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: Saving service mds.all spec with placement smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:06 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:06.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 -- 172.21.15.87:0/2458669933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3140102aa0 msgr2=0x7f3140104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:06.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 --2- 172.21.15.87:0/2458669933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3140102aa0 0x7f3140104e90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3138009ca0 tx=0x7f313802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 -- 172.21.15.87:0/2458669933 shutdown_connections 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 --2- 172.21.15.87:0/2458669933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140107d00 0x7f314010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 --2- 172.21.15.87:0/2458669933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31401053d0 0x7f31401077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 --2- 172.21.15.87:0/2458669933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3140102aa0 0x7f3140104e90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 -- 172.21.15.87:0/2458669933 >> 172.21.15.87:0/2458669933 conn(0x7f31400fc410 msgr2=0x7f31400fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 -- 172.21.15.87:0/2458669933 shutdown_connections 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 -- 172.21.15.87:0/2458669933 wait complete. 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.973+0000 7f3145598640 1 Processor -- start 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- start start 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140102aa0 0x7f31401981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 0x7f314019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f314010dae0 con 0x7f31401053d0 2024-04-22T13:29:06.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f314010d960 con 0x7f3140107d00 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f314010dc60 con 0x7f3140102aa0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47576/0 (socket says 172.21.15.87:47576) 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 -- 172.21.15.87:0/2973180631 learned_addr learned my addr 172.21.15.87:0/2973180631 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313effd640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140102aa0 0x7f31401981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313f7fe640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 0x7f314019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 -- 172.21.15.87:0/2973180631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140102aa0 msgr2=0x7f31401981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140102aa0 0x7f31401981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 -- 172.21.15.87:0/2973180631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 msgr2=0x7f314019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 0x7f314019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 -- 172.21.15.87:0/2973180631 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314019fde0 con 0x7f31401053d0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313f7fe640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 0x7f314019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f313e7fc640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3128005fd0 tx=0x7f312800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128017070 con 0x7f31401053d0 2024-04-22T13:29:06.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128009c30 con 0x7f31401053d0 2024-04-22T13:29:06.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31401a00d0 con 0x7f31401053d0 2024-04-22T13:29:06.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128003d30 con 0x7f31401053d0 2024-04-22T13:29:06.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.974+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31401a0620 con 0x7f31401053d0 2024-04-22T13:29:06.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.975+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31400fdf00 con 0x7f31401053d0 2024-04-22T13:29:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.976+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3128003ed0 con 0x7f31401053d0 2024-04-22T13:29:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.977+0000 7f3137fff640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 0x7f312c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.977+0000 7f313effd640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 0x7f312c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.977+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f31280cfc30 con 0x7f31401053d0 2024-04-22T13:29:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.977+0000 7f313effd640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 0x7f312c0a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3138005e30 tx=0x7f3138000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:06.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:06.981+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3128020030 con 0x7f31401053d0 2024-04-22T13:29:07.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.161+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f314002d810 con 0x7f31401053d0 2024-04-22T13:29:07.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.163+0000 7f3137fff640 1 -- 172.21.15.87:0/2973180631 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18475 (secure 0 0 0) 0x7f3128009de0 con 0x7f31401053d0 2024-04-22T13:29:07.159 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:07.159 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":81,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:28:53.811375+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:29:07.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 msgr2=0x7f312c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 0x7f312c0a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3138005e30 tx=0x7f3138000f40 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 msgr2=0x7f3140198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3128005fd0 tx=0x7f312800dde0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 shutdown_connections 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f312c0a3bb0 0x7f312c0a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3140107d00 0x7f314019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31401053d0 0x7f3140198700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 --2- 172.21.15.87:0/2973180631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3140102aa0 0x7f31401981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 >> 172.21.15.87:0/2973180631 conn(0x7f31400fc410 msgr2=0x7f3140103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 shutdown_connections 2024-04-22T13:29:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.165+0000 7f3145598640 1 -- 172.21.15.87:0/2973180631 wait complete. 2024-04-22T13:29:07.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: Saving service mds.all spec with placement smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:07.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:06 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:07.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:29:07.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: Saving service mds.all spec with placement smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:06 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 -- 172.21.15.87:0/1467748668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 msgr2=0x7f6510109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 --2- 172.21.15.87:0/1467748668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f6510109320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6500009ca0 tx=0x7f650002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 -- 172.21.15.87:0/1467748668 shutdown_connections 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 --2- 172.21.15.87:0/1467748668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f6510109320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 --2- 172.21.15.87:0/1467748668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510102040 0x7f65101024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 --2- 172.21.15.87:0/1467748668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510100e60 0x7f6510101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 -- 172.21.15.87:0/1467748668 >> 172.21.15.87:0/1467748668 conn(0x7f65100fc440 msgr2=0x7f65100fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 -- 172.21.15.87:0/1467748668 shutdown_connections 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.278+0000 7f6514f8f640 1 -- 172.21.15.87:0/1467748668 wait complete. 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 Processor -- start 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 -- start start 2024-04-22T13:29:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510100e60 0x7f6510198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102040 0x7f6510198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f651010c9f0 con 0x7f6510102a00 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f651010c870 con 0x7f6510102040 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f6514f8f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f651010cb70 con 0x7f6510100e60 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47596/0 (socket says 172.21.15.87:47596) 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 -- 172.21.15.87:0/3450998377 learned_addr learned my addr 172.21.15.87:0/3450998377 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650e59c640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510100e60 0x7f6510198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650dd9b640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102040 0x7f6510198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 -- 172.21.15.87:0/3450998377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510100e60 msgr2=0x7f6510198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510100e60 0x7f6510198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 -- 172.21.15.87:0/3450998377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102040 msgr2=0x7f6510198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102040 0x7f6510198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 -- 172.21.15.87:0/3450998377 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651019fd20 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f650ed9d640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f650002c700 tx=0x7f6500034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.279+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500039040 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.280+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6500044070 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.280+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651019ffb0 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.280+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65000042a0 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.280+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65101a0410 con 0x7f6510102a00 2024-04-22T13:29:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.280+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65101a0670 con 0x7f6510102a00 2024-04-22T13:29:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.282+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f65000325c0 con 0x7f6510102a00 2024-04-22T13:29:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.282+0000 7f64ff7fe640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 0x7f64f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.282+0000 7f650e59c640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 0x7f64f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.282+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f65000f2d60 con 0x7f6510102a00 2024-04-22T13:29:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.283+0000 7f650e59c640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 0x7f64f80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6508005fd0 tx=0x7f650800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.286+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65000bfd00 con 0x7f6510102a00 2024-04-22T13:29:07.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.463+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65101a0a00 con 0x7f6510102a00 2024-04-22T13:29:07.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.463+0000 7f64ff7fe640 1 -- 172.21.15.87:0/3450998377 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f65000bf600 con 0x7f6510102a00 2024-04-22T13:29:07.459 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 msgr2=0x7f64f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 0x7f64f80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6508005fd0 tx=0x7f650800b410 comp rx=0 tx=0).stop 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 msgr2=0x7f651019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f650002c700 tx=0x7f6500034760 comp rx=0 tx=0).stop 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 shutdown_connections 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64f80a3bb0 0x7f64f80a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6510102a00 0x7f651019f6b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102040 0x7f6510198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 --2- 172.21.15.87:0/3450998377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6510100e60 0x7f6510198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 >> 172.21.15.87:0/3450998377 conn(0x7f65100fc440 msgr2=0x7f651006a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:07.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 shutdown_connections 2024-04-22T13:29:07.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.465+0000 7f6514f8f640 1 -- 172.21.15.87:0/3450998377 wait complete. 2024-04-22T13:29:07.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:07.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 -- 172.21.15.87:0/276076505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 msgr2=0x7f28a0102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/276076505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a0102440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f288c009900 tx=0x7f288c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:07.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 -- 172.21.15.87:0/276076505 shutdown_connections 2024-04-22T13:29:07.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/276076505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 0x7f28a01092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/276076505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a0102440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/276076505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 0x7f28a0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 -- 172.21.15.87:0/276076505 >> 172.21.15.87:0/276076505 conn(0x7f28a00fc3e0 msgr2=0x7f28a00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 -- 172.21.15.87:0/276076505 shutdown_connections 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.573+0000 7f28a6ba4640 1 -- 172.21.15.87:0/276076505 wait complete. 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 Processor -- start 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 -- start start 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 0x7f28a0197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 0x7f28a019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f28a010ca10 con 0x7f28a0101fc0 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f28a010c890 con 0x7f28a0102980 2024-04-22T13:29:07.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a6ba4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f28a010cb90 con 0x7f28a0100e70 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47606/0 (socket says 172.21.15.87:47606) 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 -- 172.21.15.87:0/3721539978 learned_addr learned my addr 172.21.15.87:0/3721539978 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a4940640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 0x7f28a0197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f28a5141640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 0x7f28a019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 -- 172.21.15.87:0/3721539978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 msgr2=0x7f28a0197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 0x7f28a0197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 -- 172.21.15.87:0/3721539978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 msgr2=0x7f28a019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 0x7f28a019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 -- 172.21.15.87:0/3721539978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a019eca0 con 0x7f28a0101fc0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.574+0000 7f289ffff640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.575+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c043020 con 0x7f28a0101fc0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.575+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f288c0044c0 con 0x7f28a0101fc0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.575+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c003c90 con 0x7f28a0101fc0 2024-04-22T13:29:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.575+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a019ef30 con 0x7f28a0101fc0 2024-04-22T13:29:07.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.575+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a019f340 con 0x7f28a0101fc0 2024-04-22T13:29:07.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.576+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a0108180 con 0x7f28a0101fc0 2024-04-22T13:29:07.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.577+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f288c046020 con 0x7f28a0101fc0 2024-04-22T13:29:07.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.577+0000 7f289dffb640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 0x7f28900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:07.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.577+0000 7f28a4940640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 0x7f28900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:07.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.577+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f288c0f2100 con 0x7f28a0101fc0 2024-04-22T13:29:07.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.578+0000 7f28a4940640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 0x7f28900a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2894009bd0 tx=0x7f2894005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:07.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.582+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f288c04d020 con 0x7f28a0101fc0 2024-04-22T13:29:07.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.797+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28a002d810 con 0x7f28a0101fc0 2024-04-22T13:29:07.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.797+0000 7f289dffb640 1 -- 172.21.15.87:0/3721539978 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f288c0bef80 con 0x7f28a0101fc0 2024-04-22T13:29:07.793 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:07.794 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-22T13:29:07.794 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 1 2024-04-22T13:29:07.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 msgr2=0x7f28900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 0x7f28900a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2894009bd0 tx=0x7f2894005f90 comp rx=0 tx=0).stop 2024-04-22T13:29:07.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 msgr2=0x7f28a01975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:07.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f288c000c00 tx=0x7f288c0040c0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 shutdown_connections 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f28900a3bb0 0x7f28900a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28a0102980 0x7f28a019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28a0101fc0 0x7f28a01975c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 --2- 172.21.15.87:0/3721539978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28a0100e70 0x7f28a0197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 >> 172.21.15.87:0/3721539978 conn(0x7f28a00fc3e0 msgr2=0x7f28a0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 shutdown_connections 2024-04-22T13:29:07.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:07.799+0000 7f28a6ba4640 1 -- 172.21.15.87:0/3721539978 wait complete. 2024-04-22T13:29:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:07 smithi132 ceph-mon[31531]: Deploying daemon mds.l on smithi132 2024-04-22T13:29:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:07 smithi132 ceph-mon[31531]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:07 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2973180631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:07 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3450998377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:07 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3721539978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:07 smithi120 ceph-mon[31101]: Deploying daemon mds.l on smithi132 2024-04-22T13:29:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:07 smithi120 ceph-mon[31101]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:07 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2973180631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:07 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3450998377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:07 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3721539978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:07 smithi087 ceph-mon[27999]: Deploying daemon mds.l on smithi132 2024-04-22T13:29:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:07 smithi087 ceph-mon[27999]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:07 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2973180631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:07 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3450998377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:07 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3721539978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:08.963 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:08 smithi132 systemd[1]: Starting Ceph mds.l for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:09.332 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 podman[61617]: 2024-04-22 13:29:08.965253856 +0000 UTC m=+0.018020542 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:09.332 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 podman[61617]: 2024-04-22 13:29:09.098830738 +0000 UTC m=+0.151597460 container create 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 podman[61617]: 2024-04-22 13:29:09.451673101 +0000 UTC m=+0.504439777 container init 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 podman[61617]: 2024-04-22 13:29:09.454249462 +0000 UTC m=+0.507016145 container start 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mds[61636]: -- 172.21.15.132:0/591029564 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fc14c9a40 con 0x557fc149cc00 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l[61632]: starting mds.l at 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 bash[61617]: 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 2024-04-22T13:29:09.713 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:29:09 smithi132 systemd[1]: Started Ceph mds.l for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:09.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 -- 172.21.15.87:0/371629339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 msgr2=0x7f720010c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 --2- 172.21.15.87:0/371629339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f720010c2d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009ca0 tx=0x7f71f802c180 comp rx=0 tx=0).stop 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 -- 172.21.15.87:0/371629339 shutdown_connections 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 --2- 172.21.15.87:0/371629339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f720010c2d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 --2- 172.21.15.87:0/371629339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 0x7f7200074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 --2- 172.21.15.87:0/371629339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 0x7f7200074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 -- 172.21.15.87:0/371629339 >> 172.21.15.87:0/371629339 conn(0x7f72000fd500 msgr2=0x7f72000ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 -- 172.21.15.87:0/371629339 shutdown_connections 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.922+0000 7f720793d640 1 -- 172.21.15.87:0/371629339 wait complete. 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 Processor -- start 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 -- start start 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 0x7f720019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 0x7f720019a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f720010fce0 con 0x7f7200109e40 2024-04-22T13:29:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f720010fb60 con 0x7f7200075f10 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f720793d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f720010fe60 con 0x7f7200074930 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49798/0 (socket says 172.21.15.87:49798) 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 -- 172.21.15.87:0/1323932983 learned_addr learned my addr 172.21.15.87:0/1323932983 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7204ed8640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 0x7f720019a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f72056d9640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 0x7f720019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 -- 172.21.15.87:0/1323932983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 msgr2=0x7f720019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 0x7f720019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 -- 172.21.15.87:0/1323932983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 msgr2=0x7f720019a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 0x7f720019a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.923+0000 7f7205eda640 1 -- 172.21.15.87:0/1323932983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72001a1f70 con 0x7f7200109e40 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f7205eda640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71f8005e30 tx=0x7f71f8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f8039040 con 0x7f7200109e40 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f8044070 con 0x7f7200109e40 2024-04-22T13:29:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72001a2200 con 0x7f7200109e40 2024-04-22T13:29:09.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f80328a0 con 0x7f7200109e40 2024-04-22T13:29:09.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.924+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72001a26f0 con 0x7f7200109e40 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.925+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7200108c00 con 0x7f7200109e40 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.926+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f71f8032b00 con 0x7f7200109e40 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.927+0000 7f71f67fc640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 0x7f71ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.927+0000 7f72056d9640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 0x7f71ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.927+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f71f80f2630 con 0x7f7200109e40 2024-04-22T13:29:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.927+0000 7f72056d9640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 0x7f71ec0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71f0005fd0 tx=0x7f71f000b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:09.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:09.931+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71f80bf410 con 0x7f7200109e40 2024-04-22T13:29:10.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.150+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7200062b50 con 0x7f7200109e40 2024-04-22T13:29:10.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.151+0000 7f71f67fc640 1 -- 172.21.15.87:0/1323932983 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f71f80bed10 con 0x7f7200109e40 2024-04-22T13:29:10.147 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:10.147 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T13:29:10.147 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 2 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 msgr2=0x7f71ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 0x7f71ec0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71f0005fd0 tx=0x7f71f000b590 comp rx=0 tx=0).stop 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 msgr2=0x7f72001a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71f8005e30 tx=0x7f71f8003c20 comp rx=0 tx=0).stop 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 shutdown_connections 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71ec0a3c30 0x7f71ec0a60f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7200109e40 0x7f72001a1900 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7200075f10 0x7f720019a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:10.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 --2- 172.21.15.87:0/1323932983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7200074930 0x7f720019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:10.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 >> 172.21.15.87:0/1323932983 conn(0x7f72000fd500 msgr2=0x7f7200105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:10.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 shutdown_connections 2024-04-22T13:29:10.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:10.153+0000 7f720793d640 1 -- 172.21.15.87:0/1323932983 wait complete. 2024-04-22T13:29:10.159 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:10.159 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6832/1335303979,v1:172.21.15.132:6833/1335303979] up:boot 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: fsmap 1 up:standby 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:10.160 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:09 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6832/1335303979,v1:172.21.15.132:6833/1335303979] up:boot 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: fsmap 1 up:standby 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:09 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:10.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6832/1335303979,v1:172.21.15.132:6833/1335303979] up:boot 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: fsmap 1 up:standby 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:09 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:11.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:10 smithi120 ceph-mon[31101]: Deploying daemon mds.k on smithi120 2024-04-22T13:29:11.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:10 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1323932983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:11.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:10 smithi132 ceph-mon[31531]: Deploying daemon mds.k on smithi120 2024-04-22T13:29:11.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:10 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1323932983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:11.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:10 smithi087 ceph-mon[27999]: Deploying daemon mds.k on smithi120 2024-04-22T13:29:11.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:10 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1323932983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:11 smithi120 ceph-mon[31101]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:12.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:12.190 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:11 smithi087 ceph-mon[27999]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:12.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:11 smithi132 ceph-mon[31531]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- 172.21.15.87:0/1919488049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 msgr2=0x7f1258101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- 172.21.15.87:0/1919488049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258101ca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f124c00aa10 tx=0x7f124c02cf70 comp rx=0 tx=0).stop 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- 172.21.15.87:0/1919488049 shutdown_connections 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- 172.21.15.87:0/1919488049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12581021e0 0x7f1258108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- 172.21.15.87:0/1919488049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258101ca0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- 172.21.15.87:0/1919488049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1258100630 0x7f1258100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- 172.21.15.87:0/1919488049 >> 172.21.15.87:0/1919488049 conn(0x7f12580fbba0 msgr2=0x7f12580fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- 172.21.15.87:0/1919488049 shutdown_connections 2024-04-22T13:29:12.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- 172.21.15.87:0/1919488049 wait complete. 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 Processor -- start 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- start start 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1258100630 0x7f12581980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12581021e0 0x7f125819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f125810b8f0 con 0x7f1258101820 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.279+0000 7f125ee93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f125810b770 con 0x7f12581021e0 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f125ee93640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f125810ba70 con 0x7f1258100630 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49818/0 (socket says 172.21.15.87:49818) 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 -- 172.21.15.87:0/912516213 learned_addr learned my addr 172.21.15.87:0/912516213 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:12.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f125cc2f640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1258100630 0x7f12581980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f125d430640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12581021e0 0x7f125819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 -- 172.21.15.87:0/912516213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1258100630 msgr2=0x7f12581980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1258100630 0x7f12581980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 -- 172.21.15.87:0/912516213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12581021e0 msgr2=0x7f125819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12581021e0 0x7f125819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 -- 172.21.15.87:0/912516213 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125819fd10 con 0x7f1258101820 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1257fff640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f124c00a340 tx=0x7f124c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c035e00 con 0x7f1258101820 2024-04-22T13:29:12.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f124c043070 con 0x7f1258101820 2024-04-22T13:29:12.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125819ffa0 con 0x7f1258101820 2024-04-22T13:29:12.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.280+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c0038e0 con 0x7f1258101820 2024-04-22T13:29:12.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.281+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12581a0400 con 0x7f1258101820 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.282+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1258107950 con 0x7f1258101820 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.282+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f124c004ef0 con 0x7f1258101820 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.283+0000 7f1255ffb640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 0x7f12400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.283+0000 7f125cc2f640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 0x7f12400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.283+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f124c0f23c0 con 0x7f1258101820 2024-04-22T13:29:12.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.284+0000 7f125cc2f640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 0x7f12400a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1258101680 tx=0x7f1248005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:12.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.288+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f124c0bf240 con 0x7f1258101820 2024-04-22T13:29:12.318 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:29:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k[60493]: starting mds.k at 2024-04-22T13:29:12.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.522+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f125802d660 con 0x7f1258101820 2024-04-22T13:29:12.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.523+0000 7f1255ffb640 1 -- 172.21.15.87:0/912516213 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f124c0beb40 con 0x7f1258101820 2024-04-22T13:29:12.519 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:12.519 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T13:29:12.519 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 2 2024-04-22T13:29:12.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 msgr2=0x7f12400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 0x7f12400a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1258101680 tx=0x7f1248005f90 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 msgr2=0x7f1258198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f124c00a340 tx=0x7f124c004970 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 shutdown_connections 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12400a3bb0 0x7f12400a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12581021e0 0x7f125819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1258101820 0x7f1258198630 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 --2- 172.21.15.87:0/912516213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1258100630 0x7f12581980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 >> 172.21.15.87:0/912516213 conn(0x7f12580fbba0 msgr2=0x7f1258104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 shutdown_connections 2024-04-22T13:29:12.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:12.525+0000 7f125ee93640 1 -- 172.21.15.87:0/912516213 wait complete. 2024-04-22T13:29:12.670 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:29:12 smithi120 bash[60478]: ea2124c2d7946bfeea7baf4a12d09454d9d0bd6c1a2ba66f107f332820d5cb2b 2024-04-22T13:29:12.670 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:29:12 smithi120 systemd[1]: Started Ceph mds.k for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:13.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: Deploying daemon mds.j on smithi087 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/912516213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6832/820427937,v1:172.21.15.120:6833/820427937] up:boot 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: fsmap 2 up:standby 2024-04-22T13:29:13.637 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:13 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: Deploying daemon mds.j on smithi087 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/912516213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6832/820427937,v1:172.21.15.120:6833/820427937] up:boot 2024-04-22T13:29:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: fsmap 2 up:standby 2024-04-22T13:29:13.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:13 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:29:13.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:13.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:13.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: Deploying daemon mds.j on smithi087 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/912516213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6832/820427937,v1:172.21.15.120:6833/820427937] up:boot 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: fsmap 2 up:standby 2024-04-22T13:29:13.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:13 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:29:14.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- 172.21.15.87:0/2293379922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103190 msgr2=0x7f37b8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- 172.21.15.87:0/2293379922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103190 0x7f37b8103610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009900 tx=0x7f37a802bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- 172.21.15.87:0/2293379922 shutdown_connections 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- 172.21.15.87:0/2293379922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8103b50 0x7f37b810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- 172.21.15.87:0/2293379922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103190 0x7f37b8103610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- 172.21.15.87:0/2293379922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8101fa0 0x7f37b81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- 172.21.15.87:0/2293379922 >> 172.21.15.87:0/2293379922 conn(0x7f37b80fd530 msgr2=0x7f37b80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- 172.21.15.87:0/2293379922 shutdown_connections 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- 172.21.15.87:0/2293379922 wait complete. 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 Processor -- start 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- start start 2024-04-22T13:29:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8101fa0 0x7f37b8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8103190 0x7f37b8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37b810db30 con 0x7f37b8103b50 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f37b810d9b0 con 0x7f37b8101fa0 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.730+0000 7f37c071e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f37b810dcb0 con 0x7f37b8103190 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49834/0 (socket says 172.21.15.87:49834) 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 -- 172.21.15.87:0/3412765814 learned_addr learned my addr 172.21.15.87:0/3412765814 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37be4ba640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8101fa0 0x7f37b8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37bdcb9640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8103190 0x7f37b8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:14.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 -- 172.21.15.87:0/3412765814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8103190 msgr2=0x7f37b8074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8103190 0x7f37b8074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 -- 172.21.15.87:0/3412765814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8101fa0 msgr2=0x7f37b8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8101fa0 0x7f37b8075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 -- 172.21.15.87:0/3412765814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b8075110 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37becbb640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f37b000cad0 tx=0x7f37b000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b000ae20 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b81a22e0 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37b0005020 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b000f6b0 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.731+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b81a2800 con 0x7f37b8103b50 2024-04-22T13:29:14.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.732+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b8068de0 con 0x7f37b8103b50 2024-04-22T13:29:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.734+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f37b00051d0 con 0x7f37b8103b50 2024-04-22T13:29:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.735+0000 7f37af7fe640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 0x7f37a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.735+0000 7f37be4ba640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 0x7f37a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.735+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f37b00d0340 con 0x7f37b8103b50 2024-04-22T13:29:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.735+0000 7f37be4ba640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 0x7f37a40a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f37b40058c0 tx=0x7f37b400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:14.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.739+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37b0014030 con 0x7f37b8103b50 2024-04-22T13:29:14.736 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:14 smithi087 systemd[1]: Starting Ceph mds.j for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:14.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.964+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37b8062b50 con 0x7f37b8103b50 2024-04-22T13:29:14.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.965+0000 7f37af7fe640 1 -- 172.21.15.87:0/3412765814 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f37b009cb70 con 0x7f37b8103b50 2024-04-22T13:29:14.961 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:14.961 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T13:29:14.961 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 3 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 msgr2=0x7f37a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 0x7f37a40a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f37b40058c0 tx=0x7f37b400a2d0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 msgr2=0x7f37b8074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f37b000cad0 tx=0x7f37b000cf90 comp rx=0 tx=0).stop 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 shutdown_connections 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f37a40a3bb0 0x7f37a40a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37b8103b50 0x7f37b8074880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b8103190 0x7f37b8074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 --2- 172.21.15.87:0/3412765814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b8101fa0 0x7f37b8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:14.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 >> 172.21.15.87:0/3412765814 conn(0x7f37b80fd530 msgr2=0x7f37b8106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:14.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 shutdown_connections 2024-04-22T13:29:14.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:14.967+0000 7f37c071e640 1 -- 172.21.15.87:0/3412765814 wait complete. 2024-04-22T13:29:15.147 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:14 smithi087 podman[65785]: 2024-04-22 13:29:14.79441592 +0000 UTC m=+0.018261793 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:15.147 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:14 smithi087 podman[65785]: 2024-04-22 13:29:14.915902809 +0000 UTC m=+0.139748672 container create 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:15.147 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 podman[65785]: 2024-04-22 13:29:15.151977799 +0000 UTC m=+0.375823666 container init 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T13:29:15.487 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 podman[65785]: 2024-04-22 13:29:15.154903633 +0000 UTC m=+0.378749497 container start 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T13:29:15.487 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mds[65806]: -- 172.21.15.87:0/1428568406 <== mon.0 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55577295ba40 con 0x55577292f000 2024-04-22T13:29:15.487 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j[65802]: starting mds.j at 2024-04-22T13:29:15.487 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 bash[65785]: 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa 2024-04-22T13:29:15.487 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:29:15 smithi087 systemd[1]: Started Ceph mds.j for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3412765814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:16.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:16.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:15 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3412765814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:15 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3412765814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:15 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:16.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:17.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 -- 172.21.15.87:0/2918664822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1581029b0 msgr2=0x7fd1581092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 --2- 172.21.15.87:0/2918664822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1581029b0 0x7fd1581092d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd150009ca0 tx=0x7fd15002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 -- 172.21.15.87:0/2918664822 shutdown_connections 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 --2- 172.21.15.87:0/2918664822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1581029b0 0x7fd1581092d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 --2- 172.21.15.87:0/2918664822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158101ff0 0x7fd158102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 --2- 172.21.15.87:0/2918664822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd158100ea0 0x7fd1581012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 -- 172.21.15.87:0/2918664822 >> 172.21.15.87:0/2918664822 conn(0x7fd1580fc410 msgr2=0x7fd1580fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 -- 172.21.15.87:0/2918664822 shutdown_connections 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.091+0000 7fd15fb75640 1 -- 172.21.15.87:0/2918664822 wait complete. 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 Processor -- start 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 -- start start 2024-04-22T13:29:17.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 0x7fd158198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 0x7fd15819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810c9f0 con 0x7fd158101ff0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810c870 con 0x7fd1581029b0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15fb75640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810cb70 con 0x7fd158100ea0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49866/0 (socket says 172.21.15.87:49866) 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 -- 172.21.15.87:0/1970062956 learned_addr learned my addr 172.21.15.87:0/1970062956 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15e112640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 0x7fd15819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d911640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 0x7fd158198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 -- 172.21.15.87:0/1970062956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 msgr2=0x7fd158198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 0x7fd158198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 -- 172.21.15.87:0/1970062956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 msgr2=0x7fd15819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:17.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.092+0000 7fd15d110640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 0x7fd15819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15d110640 1 -- 172.21.15.87:0/1970062956 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15819fd60 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15e112640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 0x7fd15819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15d911640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 0x7fd158198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15d110640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d950 tx=0x7fd14800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148017070 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15819fff0 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd148009c30 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14800f620 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.093+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1581a0510 con 0x7fd158101ff0 2024-04-22T13:29:17.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.094+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1581a0740 con 0x7fd158101ff0 2024-04-22T13:29:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.095+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd14801a070 con 0x7fd158101ff0 2024-04-22T13:29:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.095+0000 7fd14effd640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 0x7fd1440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.096+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd1480cfa90 con 0x7fd158101ff0 2024-04-22T13:29:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.096+0000 7fd15d911640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 0x7fd1440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:17.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.096+0000 7fd15d911640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 0x7fd1440a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1540058c0 tx=0x7fd154005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:17.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.100+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd148009de0 con 0x7fd158101ff0 2024-04-22T13:29:17.097 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:16 smithi087 ceph-mon[27999]: Deploying daemon mds.d on smithi087 2024-04-22T13:29:17.097 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:16 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.87:6834/915813894,v1:172.21.15.87:6835/915813894] up:boot 2024-04-22T13:29:17.097 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:16 smithi087 ceph-mon[27999]: fsmap 3 up:standby 2024-04-22T13:29:17.097 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:16 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:29:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:16 smithi120 ceph-mon[31101]: Deploying daemon mds.d on smithi087 2024-04-22T13:29:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:16 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.87:6834/915813894,v1:172.21.15.87:6835/915813894] up:boot 2024-04-22T13:29:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:16 smithi120 ceph-mon[31101]: fsmap 3 up:standby 2024-04-22T13:29:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:16 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:29:17.174 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:16 smithi132 ceph-mon[31531]: Deploying daemon mds.d on smithi087 2024-04-22T13:29:17.175 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:16 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.87:6834/915813894,v1:172.21.15.87:6835/915813894] up:boot 2024-04-22T13:29:17.175 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:16 smithi132 ceph-mon[31531]: fsmap 3 up:standby 2024-04-22T13:29:17.175 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:16 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:29:17.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.344+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1581a0a30 con 0x7fd158101ff0 2024-04-22T13:29:17.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.345+0000 7fd14effd640 1 -- 172.21.15.87:0/1970062956 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fd14809c910 con 0x7fd158101ff0 2024-04-22T13:29:17.341 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:17.341 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T13:29:17.341 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 4 2024-04-22T13:29:17.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 msgr2=0x7fd1440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 0x7fd1440a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1540058c0 tx=0x7fd154005850 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 msgr2=0x7fd158198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d950 tx=0x7fd14800de10 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 shutdown_connections 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1440a3bb0 0x7fd1440a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1581029b0 0x7fd15819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd158101ff0 0x7fd158198680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 --2- 172.21.15.87:0/1970062956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd158100ea0 0x7fd158198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 >> 172.21.15.87:0/1970062956 conn(0x7fd1580fc410 msgr2=0x7fd15806a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 shutdown_connections 2024-04-22T13:29:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:17.348+0000 7fd15fb75640 1 -- 172.21.15.87:0/1970062956 wait complete. 2024-04-22T13:29:17.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:17 smithi087 systemd[1]: Starting Ceph mds.d for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:18.117 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:17 smithi087 ceph-mon[27999]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:18.117 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:17 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1970062956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:18.117 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:17 smithi087 podman[66140]: 2024-04-22 13:29:17.699483766 +0000 UTC m=+0.017901025 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:18.117 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:17 smithi087 podman[66140]: 2024-04-22 13:29:17.829323022 +0000 UTC m=+0.147740271 container create 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T13:29:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:17 smithi120 ceph-mon[31101]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:17 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1970062956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:18.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:17 smithi132 ceph-mon[31531]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:18.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:17 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1970062956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 podman[66140]: 2024-04-22 13:29:18.249074205 +0000 UTC m=+0.567491456 container init 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 podman[66140]: 2024-04-22 13:29:18.25159888 +0000 UTC m=+0.570016131 container start 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 ceph-mds[66159]: -- 172.21.15.87:0/1677250090 <== mon.0 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffd872fa40 con 0x55ffd8702800 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d[66155]: starting mds.d at 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 bash[66140]: 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c 2024-04-22T13:29:18.737 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:29:18 smithi087 systemd[1]: Started Ceph mds.d for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:19.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:19.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 -- 172.21.15.87:0/3029376423 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0101f90 msgr2=0x7fd6f01023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:19.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3029376423 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0101f90 0x7fd6f01023b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009900 tx=0x7fd6e002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 -- 172.21.15.87:0/3029376423 shutdown_connections 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3029376423 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0103b40 0x7fd6f010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3029376423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 0x7fd6f0103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3029376423 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0101f90 0x7fd6f01023b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 -- 172.21.15.87:0/3029376423 >> 172.21.15.87:0/3029376423 conn(0x7fd6f00fd500 msgr2=0x7fd6f00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 -- 172.21.15.87:0/3029376423 shutdown_connections 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.474+0000 7fd6f6288640 1 -- 172.21.15.87:0/3029376423 wait complete. 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 Processor -- start 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 -- start start 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0101f90 0x7fd6f0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:19.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 0x7fd6f0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f010dad0 con 0x7fd6f0103b40 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f010d950 con 0x7fd6f0103180 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f6288640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f010dc50 con 0x7fd6f0101f90 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f4825640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f4825640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49906/0 (socket says 172.21.15.87:49906) 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6f4825640 1 -- 172.21.15.87:0/3376870268 learned_addr learned my addr 172.21.15.87:0/3376870268 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6ef7fe640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 0x7fd6f0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.475+0000 7fd6effff640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0101f90 0x7fd6f0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 -- 172.21.15.87:0/3376870268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0101f90 msgr2=0x7fd6f0198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0101f90 0x7fd6f0198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 -- 172.21.15.87:0/3376870268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 msgr2=0x7fd6f0198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:19.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 0x7fd6f0198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 -- 172.21.15.87:0/3376870268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f019fd50 con 0x7fd6f0103b40 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f4825640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800eaa0 tx=0x7fd6e800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8004d20 con 0x7fd6f0103b40 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e8005020 con 0x7fd6f0103b40 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f019ffe0 con 0x7fd6f0103b40 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80116b0 con 0x7fd6f0103b40 2024-04-22T13:29:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.476+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f01a0500 con 0x7fd6f0103b40 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.477+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f01092b0 con 0x7fd6f0103b40 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.478+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd6e80051d0 con 0x7fd6f0103b40 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.479+0000 7fd6ed7fa640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 0x7fd6d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.479+0000 7fd6effff640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 0x7fd6d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.479+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6e80d00c0 con 0x7fd6f0103b40 2024-04-22T13:29:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.479+0000 7fd6effff640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 0x7fd6d80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005ab0 tx=0x7fd6e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.484+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6e8016030 con 0x7fd6f0103b40 2024-04-22T13:29:19.492 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.492 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.492 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: Deploying daemon mds.e on smithi120 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.87:6836/1472471745,v1:172.21.15.87:6837/1472471745] up:boot 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: fsmap 4 up:standby 2024-04-22T13:29:19.493 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:19 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: Deploying daemon mds.e on smithi120 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.87:6836/1472471745,v1:172.21.15.87:6837/1472471745] up:boot 2024-04-22T13:29:19.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: fsmap 4 up:standby 2024-04-22T13:29:19.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:19 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:29:19.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.700+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6f002d5c0 con 0x7fd6f0103b40 2024-04-22T13:29:19.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.700+0000 7fd6ed7fa640 1 -- 172.21.15.87:0/3376870268 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fd6e809cf40 con 0x7fd6f0103b40 2024-04-22T13:29:19.696 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:19.697 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T13:29:19.697 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 5 2024-04-22T13:29:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.702+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 msgr2=0x7fd6d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.702+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 0x7fd6d80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005ab0 tx=0x7fd6e0005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.702+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 msgr2=0x7fd6f019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.702+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800eaa0 tx=0x7fd6e800eea0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 shutdown_connections 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6d80a3bb0 0x7fd6d80a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6f0103b40 0x7fd6f019f6e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f0103180 0x7fd6f0198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 --2- 172.21.15.87:0/3376870268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd6f0101f90 0x7fd6f0198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 >> 172.21.15.87:0/3376870268 conn(0x7fd6f00fd500 msgr2=0x7fd6f0106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 shutdown_connections 2024-04-22T13:29:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:19.703+0000 7fd6f6288640 1 -- 172.21.15.87:0/3376870268 wait complete. 2024-04-22T13:29:19.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:19.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:19.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: Deploying daemon mds.e on smithi120 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.87:6836/1472471745,v1:172.21.15.87:6837/1472471745] up:boot 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: fsmap 4 up:standby 2024-04-22T13:29:19.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:19 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:29:20.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:20 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3376870268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:20.670 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:20 smithi120 systemd[1]: Starting Ceph mds.e for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:20.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:20 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3376870268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:20.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:20 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3376870268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:21.038 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:20 smithi120 podman[60804]: 2024-04-22 13:29:20.686160045 +0000 UTC m=+0.018105849 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:21.038 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:20 smithi120 podman[60804]: 2024-04-22 13:29:20.798317994 +0000 UTC m=+0.130263786 container create 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T13:29:21.038 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 podman[60804]: 2024-04-22 13:29:21.043077553 +0000 UTC m=+0.375023347 container init 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T13:29:21.420 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 podman[60804]: 2024-04-22 13:29:21.04569117 +0000 UTC m=+0.377636970 container start 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T13:29:21.420 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mds[60823]: -- 172.21.15.120:0/2600963857 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55daa0a27a40 con 0x55daa09fac00 2024-04-22T13:29:21.420 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: starting mds.e at 2024-04-22T13:29:21.420 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 bash[60804]: 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd 2024-04-22T13:29:21.420 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:29:21 smithi120 systemd[1]: Started Ceph mds.e for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:21.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:21 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:21 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:21.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 -- 172.21.15.87:0/2437037230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 msgr2=0x7f3eb810a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:21.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 --2- 172.21.15.87:0/2437037230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb810a3d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009ca0 tx=0x7f3eb402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 -- 172.21.15.87:0/2437037230 shutdown_connections 2024-04-22T13:29:21.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 --2- 172.21.15.87:0/2437037230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb810a3d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 --2- 172.21.15.87:0/2437037230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 0x7f3eb8103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 --2- 172.21.15.87:0/2437037230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 -- 172.21.15.87:0/2437037230 >> 172.21.15.87:0/2437037230 conn(0x7f3eb80fd500 msgr2=0x7f3eb80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 -- 172.21.15.87:0/2437037230 shutdown_connections 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.822+0000 7f3ebf825640 1 -- 172.21.15.87:0/2437037230 wait complete. 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 Processor -- start 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 -- start start 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 0x7f3eb8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810da90 con 0x7f3eb8103b40 2024-04-22T13:29:21.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810d910 con 0x7f3eb8103180 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebf825640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810dc10 con 0x7f3eb8101f90 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebddc2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebddc2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34546/0 (socket says 172.21.15.87:34546) 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebddc2640 1 -- 172.21.15.87:0/3797469613 learned_addr learned my addr 172.21.15.87:0/3797469613 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.823+0000 7f3ebcdc0640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 0x7f3eb8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebd5c1640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 -- 172.21.15.87:0/3797469613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 msgr2=0x7f3eb8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb8075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 -- 172.21.15.87:0/3797469613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 msgr2=0x7f3eb8074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 0x7f3eb8074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 -- 172.21.15.87:0/3797469613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb8075110 con 0x7f3eb8103b40 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebd5c1640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebddc2640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3eb402c700 tx=0x7f3eb40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:21.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4041020 con 0x7f3eb8103b40 2024-04-22T13:29:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eb4034d60 con 0x7f3eb8103b40 2024-04-22T13:29:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb81a21d0 con 0x7f3eb8103b40 2024-04-22T13:29:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4039040 con 0x7f3eb8103b40 2024-04-22T13:29:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.824+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81a2770 con 0x7f3eb8103b40 2024-04-22T13:29:21.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.825+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb8109330 con 0x7f3eb8103b40 2024-04-22T13:29:21.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.826+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3eb4044020 con 0x7f3eb8103b40 2024-04-22T13:29:21.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.827+0000 7f3eae7fc640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 0x7f3ea40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:21.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.827+0000 7f3ebd5c1640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 0x7f3ea40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:21.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.827+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3eb40f1e40 con 0x7f3eb8103b40 2024-04-22T13:29:21.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.827+0000 7f3ebd5c1640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 0x7f3ea40a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0005d00 tx=0x7f3eb0005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:21.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:21.832+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3eb404b020 con 0x7f3eb8103b40 2024-04-22T13:29:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:21.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:21.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:21.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:21 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:22.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.049+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb8062b50 con 0x7f3eb8103b40 2024-04-22T13:29:22.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.049+0000 7f3eae7fc640 1 -- 172.21.15.87:0/3797469613 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f3eb40becc0 con 0x7f3eb8103b40 2024-04-22T13:29:22.045 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:22.046 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T13:29:22.046 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 6 2024-04-22T13:29:22.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 msgr2=0x7f3ea40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:22.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 0x7f3ea40a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0005d00 tx=0x7f3eb0005ef0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 msgr2=0x7f3eb8074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3eb402c700 tx=0x7f3eb40093f0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 shutdown_connections 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3ea40a3c30 0x7f3ea40a60f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3eb8103b40 0x7f3eb8074880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3eb8103180 0x7f3eb8074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 --2- 172.21.15.87:0/3797469613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3eb8101f90 0x7f3eb8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 >> 172.21.15.87:0/3797469613 conn(0x7f3eb80fd500 msgr2=0x7f3eb8106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 shutdown_connections 2024-04-22T13:29:22.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:22.051+0000 7f3ebf825640 1 -- 172.21.15.87:0/3797469613 wait complete. 2024-04-22T13:29:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:22 smithi087 ceph-mon[27999]: Deploying daemon mds.f on smithi132 2024-04-22T13:29:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:22 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6834/1829745193,v1:172.21.15.120:6835/1829745193] up:boot 2024-04-22T13:29:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:22 smithi087 ceph-mon[27999]: fsmap 5 up:standby 2024-04-22T13:29:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:22 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:29:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3797469613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:22.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:22 smithi132 ceph-mon[31531]: Deploying daemon mds.f on smithi132 2024-04-22T13:29:22.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:22 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6834/1829745193,v1:172.21.15.120:6835/1829745193] up:boot 2024-04-22T13:29:22.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:22 smithi132 ceph-mon[31531]: fsmap 5 up:standby 2024-04-22T13:29:22.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:22 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:29:22.880 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3797469613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:22 smithi120 ceph-mon[31101]: Deploying daemon mds.f on smithi132 2024-04-22T13:29:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:22 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6834/1829745193,v1:172.21.15.120:6835/1829745193] up:boot 2024-04-22T13:29:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:22 smithi120 ceph-mon[31101]: fsmap 5 up:standby 2024-04-22T13:29:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:22 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:29:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3797469613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:23.463 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 systemd[1]: Starting Ceph mds.f for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:23.729 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:23 smithi132 ceph-mon[31531]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:23.730 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 podman[61929]: 2024-04-22 13:29:23.501726032 +0000 UTC m=+0.018582427 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:23.730 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 podman[61929]: 2024-04-22 13:29:23.621294045 +0000 UTC m=+0.138150431 container create bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T13:29:23.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:23 smithi087 ceph-mon[27999]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:23.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:23 smithi120 ceph-mon[31101]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:24.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 -- 172.21.15.87:0/2876793835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8102a20 msgr2=0x7f56d8104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 --2- 172.21.15.87:0/2876793835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8102a20 0x7f56d8104e10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009900 tx=0x7f56c002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 -- 172.21.15.87:0/2876793835 shutdown_connections 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 --2- 172.21.15.87:0/2876793835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 0x7f56d810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 --2- 172.21.15.87:0/2876793835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8105350 0x7f56d8107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 --2- 172.21.15.87:0/2876793835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8102a20 0x7f56d8104e10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.169+0000 7f56dd175640 1 -- 172.21.15.87:0/2876793835 >> 172.21.15.87:0/2876793835 conn(0x7f56d80fc410 msgr2=0x7f56d80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.170+0000 7f56dd175640 1 -- 172.21.15.87:0/2876793835 shutdown_connections 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.170+0000 7f56dd175640 1 -- 172.21.15.87:0/2876793835 wait complete. 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.170+0000 7f56dd175640 1 Processor -- start 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.170+0000 7f56dd175640 1 -- start start 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 0x7f56d8198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 0x7f56d819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810dae0 con 0x7f56d8105350 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810d960 con 0x7f56d8102a20 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f56d810dc60 con 0x7f56d8107c80 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34564/0 (socket says 172.21.15.87:34564) 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 -- 172.21.15.87:0/2303497513 learned_addr learned my addr 172.21.15.87:0/2303497513 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d759e640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 0x7f56d819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:24.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d6d9d640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 0x7f56d8198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 -- 172.21.15.87:0/2303497513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 msgr2=0x7f56d819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 0x7f56d819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 -- 172.21.15.87:0/2303497513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 msgr2=0x7f56d8198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 0x7f56d8198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 -- 172.21.15.87:0/2303497513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d819fda0 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d6d9d640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 0x7f56d8198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56d659c640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56c800d770 tx=0x7f56c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8017070 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d81a0090 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c8009c30 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c800f620 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.171+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d81a05e0 con 0x7f56d8105350 2024-04-22T13:29:24.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.173+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56d80fdf00 con 0x7f56d8105350 2024-04-22T13:29:24.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.174+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f56c801a070 con 0x7f56d8105350 2024-04-22T13:29:24.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.174+0000 7f56cffff640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 0x7f56c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:24.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.174+0000 7f56d6d9d640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 0x7f56c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:24.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.175+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f56c80cfac0 con 0x7f56d8105350 2024-04-22T13:29:24.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.175+0000 7f56d6d9d640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 0x7f56c40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006010 tx=0x7f56c002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:24.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.179+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56c8009de0 con 0x7f56d8105350 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 podman[61929]: 2024-04-22 13:29:23.848976236 +0000 UTC m=+0.365832617 container init bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 podman[61929]: 2024-04-22 13:29:23.851732477 +0000 UTC m=+0.368588870 container start bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 ceph-mds[61948]: -- 172.21.15.132:0/1384529651 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c46feda40 con 0x562c46fc1000 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: starting mds.f at 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 bash[61929]: bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 2024-04-22T13:29:24.213 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:29:23 smithi132 systemd[1]: Started Ceph mds.f for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:24.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.395+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56d802d810 con 0x7f56d8105350 2024-04-22T13:29:24.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.395+0000 7f56cffff640 1 -- 172.21.15.87:0/2303497513 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f56c809c940 con 0x7f56d8105350 2024-04-22T13:29:24.392 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:24.392 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T13:29:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 6 2024-04-22T13:29:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 msgr2=0x7f56c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:24.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 0x7f56c40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f56c0006010 tx=0x7f56c002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 msgr2=0x7f56d81986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56c800d770 tx=0x7f56c800dc30 comp rx=0 tx=0).stop 2024-04-22T13:29:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 shutdown_connections 2024-04-22T13:29:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.397+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56c40a3bb0 0x7f56c40a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56d8107c80 0x7f56d819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56d8105350 0x7f56d81986c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 --2- 172.21.15.87:0/2303497513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d8102a20 0x7f56d8198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 >> 172.21.15.87:0/2303497513 conn(0x7f56d80fc410 msgr2=0x7f56d81034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 shutdown_connections 2024-04-22T13:29:24.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:24.398+0000 7f56dd175640 1 -- 172.21.15.87:0/2303497513 wait complete. 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: Deploying daemon mds.c on smithi132 2024-04-22T13:29:25.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2303497513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6834/1119314155,v1:172.21.15.132:6835/1119314155] up:boot 2024-04-22T13:29:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: fsmap 6 up:standby 2024-04-22T13:29:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:29:25.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:25 smithi132 ceph-mon[31531]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: Deploying daemon mds.c on smithi132 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2303497513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6834/1119314155,v1:172.21.15.132:6835/1119314155] up:boot 2024-04-22T13:29:25.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: fsmap 6 up:standby 2024-04-22T13:29:25.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:29:25.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:25 smithi120 ceph-mon[31101]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: Deploying daemon mds.c on smithi132 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2303497513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6834/1119314155,v1:172.21.15.132:6835/1119314155] up:boot 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: fsmap 6 up:standby 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:29:25.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:24 smithi087 ceph-mon[27999]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:26.213 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 systemd[1]: Starting Ceph mds.c for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:26.406 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- 172.21.15.87:0/3145720402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0102a00 msgr2=0x7f9af0104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 --2- 172.21.15.87:0/3145720402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0102a00 0x7f9af0104df0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009ca0 tx=0x7f9ae002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- 172.21.15.87:0/3145720402 shutdown_connections 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 --2- 172.21.15.87:0/3145720402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 --2- 172.21.15.87:0/3145720402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0105330 0x7f9af0107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 --2- 172.21.15.87:0/3145720402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0102a00 0x7f9af0104df0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- 172.21.15.87:0/3145720402 >> 172.21.15.87:0/3145720402 conn(0x7f9af00fc410 msgr2=0x7f9af00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:26.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- 172.21.15.87:0/3145720402 shutdown_connections 2024-04-22T13:29:26.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- 172.21.15.87:0/3145720402 wait complete. 2024-04-22T13:29:26.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 Processor -- start 2024-04-22T13:29:26.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.521+0000 7f9af5900640 1 -- start start 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0102a00 0x7f9af0102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af0100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9af010dad0 con 0x7f9af0105330 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9af010d950 con 0x7f9af0102a00 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9af5900640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9af010dc50 con 0x7f9af0107c60 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aee7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34580/0 (socket says 172.21.15.87:34580) 2024-04-22T13:29:26.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aee7fc640 1 -- 172.21.15.87:0/1546966909 learned_addr learned my addr 172.21.15.87:0/1546966909 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af0100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aeeffd640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0102a00 0x7f9af0102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 -- 172.21.15.87:0/1546966909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0102a00 msgr2=0x7f9af0102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0102a00 0x7f9af0102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 -- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 msgr2=0x7f9af01009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 -- 172.21.15.87:0/1546966909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af0101780 con 0x7f9af0107c60 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aee7fc640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.522+0000 7f9aef7fe640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af0100ef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800e970 tx=0x7f9ae800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.523+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800ccc0 con 0x7f9af0107c60 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.523+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ae8005020 con 0x7f9af0107c60 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.523+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af01a22c0 con 0x7f9af0107c60 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.523+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8004a00 con 0x7f9af0107c60 2024-04-22T13:29:26.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.523+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af01a2810 con 0x7f9af0107c60 2024-04-22T13:29:26.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.524+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af00fdf00 con 0x7f9af0107c60 2024-04-22T13:29:26.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.525+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ae8004ba0 con 0x7f9af0107c60 2024-04-22T13:29:26.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.525+0000 7f9af48fe640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 0x7f9ad80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:26.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.525+0000 7f9aeeffd640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 0x7f9ad80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:26.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.525+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9ae80d0070 con 0x7f9af0107c60 2024-04-22T13:29:26.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.526+0000 7f9aeeffd640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 0x7f9ad80a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005e30 tx=0x7f9ae0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:26.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.529+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ae809ced0 con 0x7f9af0107c60 2024-04-22T13:29:26.610 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 podman[62238]: 2024-04-22 13:29:26.244945786 +0000 UTC m=+0.017874630 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:26.610 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 podman[62238]: 2024-04-22 13:29:26.352273671 +0000 UTC m=+0.125202541 container create 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:26.610 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 podman[62238]: 2024-04-22 13:29:26.588946995 +0000 UTC m=+0.361875823 container init 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T13:29:26.610 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 podman[62238]: 2024-04-22 13:29:26.591731033 +0000 UTC m=+0.364659872 container start 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T13:29:26.611 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 ceph-mds[62257]: -- 172.21.15.132:0/3682052701 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56244a839a40 con 0x56244a80d000 2024-04-22T13:29:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.750+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9af002d5c0 con 0x7f9af0107c60 2024-04-22T13:29:26.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.751+0000 7f9af48fe640 1 -- 172.21.15.87:0/1546966909 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f9ae809c7d0 con 0x7f9af0107c60 2024-04-22T13:29:26.747 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:26.747 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T13:29:26.747 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 8 2024-04-22T13:29:26.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 msgr2=0x7f9ad80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:26.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 0x7f9ad80a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005e30 tx=0x7f9ae0000f40 comp rx=0 tx=0).stop 2024-04-22T13:29:26.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 msgr2=0x7f9af0100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:26.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af0100ef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800e970 tx=0x7f9ae800ee30 comp rx=0 tx=0).stop 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 shutdown_connections 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9ad80a3bb0 0x7f9ad80a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9af0107c60 0x7f9af0100ef0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9af0105330 0x7f9af01009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 --2- 172.21.15.87:0/1546966909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9af0102a00 0x7f9af0102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 >> 172.21.15.87:0/1546966909 conn(0x7f9af00fc410 msgr2=0x7f9af01034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 shutdown_connections 2024-04-22T13:29:26.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:26.753+0000 7f9af5900640 1 -- 172.21.15.87:0/1546966909 wait complete. 2024-04-22T13:29:26.963 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: starting mds.c at 2024-04-22T13:29:26.963 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 bash[62238]: 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f 2024-04-22T13:29:26.963 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:29:26 smithi132 systemd[1]: Started Ceph mds.c for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:boot 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: fsmap 7 up:standby 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1546966909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:27.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:27.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:27.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:27 smithi120 ceph-mon[31101]: Deploying daemon mds.h on smithi120 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:boot 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: fsmap 7 up:standby 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1546966909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:27.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:27.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:27 smithi132 ceph-mon[31531]: Deploying daemon mds.h on smithi120 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:boot 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: fsmap 7 up:standby 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1546966909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:27.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:27 smithi087 ceph-mon[27999]: Deploying daemon mds.h on smithi120 2024-04-22T13:29:28.761 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 -- 172.21.15.87:0/573224579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea04101f20 msgr2=0x7fea04102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 --2- 172.21.15.87:0/573224579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea04101f20 0x7fea04102360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009900 tx=0x7fe9f402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 -- 172.21.15.87:0/573224579 shutdown_connections 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 --2- 172.21.15.87:0/573224579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04103ab0 0x7fea0410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 --2- 172.21.15.87:0/573224579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea041030f0 0x7fea04103570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 --2- 172.21.15.87:0/573224579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea04101f20 0x7fea04102360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 -- 172.21.15.87:0/573224579 >> 172.21.15.87:0/573224579 conn(0x7fea040fd500 msgr2=0x7fea040ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 -- 172.21.15.87:0/573224579 shutdown_connections 2024-04-22T13:29:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 -- 172.21.15.87:0/573224579 wait complete. 2024-04-22T13:29:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.878+0000 7fea09e81640 1 Processor -- start 2024-04-22T13:29:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 -- start start 2024-04-22T13:29:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 0x7fea041981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 0x7fea0419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410da80 con 0x7fea041030f0 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410d900 con 0x7fea04101f20 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea09e81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410dc00 con 0x7fea04103ab0 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34584/0 (socket says 172.21.15.87:34584) 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 -- 172.21.15.87:0/1236260415 learned_addr learned my addr 172.21.15.87:0/1236260415 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea037fe640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 0x7fea041981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea03fff640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 0x7fea0419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 -- 172.21.15.87:0/1236260415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 msgr2=0x7fea0419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 0x7fea0419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 -- 172.21.15.87:0/1236260415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 msgr2=0x7fea041981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 0x7fea041981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea02ffd640 1 -- 172.21.15.87:0/1236260415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0419fde0 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea03fff640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 0x7fea0419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.879+0000 7fea037fe640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 0x7fea041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea02ffd640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00d950 tx=0x7fe9ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec017030 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9ec004020 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea041a00d0 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0047a0 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.880+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea041a0670 con 0x7fea041030f0 2024-04-22T13:29:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.881+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea04109420 con 0x7fea041030f0 2024-04-22T13:29:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.882+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe9ec009a30 con 0x7fea041030f0 2024-04-22T13:29:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.883+0000 7fea00ff9640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 0x7fe9f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.883+0000 7fea037fe640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 0x7fe9f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.883+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe9ec0cf9a0 con 0x7fea041030f0 2024-04-22T13:29:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.883+0000 7fea037fe640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 0x7fe9f00a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005ab0 tx=0x7fe9f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:28.888+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9ec09c820 con 0x7fea041030f0 2024-04-22T13:29:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.101+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea0402d5c0 con 0x7fea041030f0 2024-04-22T13:29:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.102+0000 7fea00ff9640 1 -- 172.21.15.87:0/1236260415 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7fe9ec09c120 con 0x7fea041030f0 2024-04-22T13:29:29.098 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:29.098 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T13:29:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 8 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 msgr2=0x7fe9f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 0x7fe9f00a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005ab0 tx=0x7fe9f4005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 msgr2=0x7fea04198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00d950 tx=0x7fe9ec00de10 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 shutdown_connections 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe9f00a3c30 0x7fe9f00a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea04103ab0 0x7fea0419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea041030f0 0x7fea04198700 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 --2- 172.21.15.87:0/1236260415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fea04101f20 0x7fea041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 >> 172.21.15.87:0/1236260415 conn(0x7fea040fd500 msgr2=0x7fea041065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 shutdown_connections 2024-04-22T13:29:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:29.104+0000 7fea09e81640 1 -- 172.21.15.87:0/1236260415 wait complete. 2024-04-22T13:29:29.170 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:28 smithi120 systemd[1]: Starting Ceph mds.h for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:29.454 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 podman[61113]: 2024-04-22 13:29:29.091345422 +0000 UTC m=+0.017930509 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:29.454 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 podman[61113]: 2024-04-22 13:29:29.205302345 +0000 UTC m=+0.131887405 container create 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 podman[61113]: 2024-04-22 13:29:29.459137787 +0000 UTC m=+0.385722845 container init 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 podman[61113]: 2024-04-22 13:29:29.462049724 +0000 UTC m=+0.388634792 container start 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mds[61132]: -- 172.21.15.120:0/487418927 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555982501a40 con 0x5559824d4c00 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: starting mds.h at 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 bash[61113]: 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 2024-04-22T13:29:29.801 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:29:29 smithi120 systemd[1]: Started Ceph mds.h for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1236260415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:29 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1236260415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:30.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:29 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:30.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1236260415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:30.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:29 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:31.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:31.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:30 smithi087 ceph-mon[27999]: Deploying daemon mds.i on smithi132 2024-04-22T13:29:31.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:30 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6836/3391616456,v1:172.21.15.120:6837/3391616456] up:boot 2024-04-22T13:29:31.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:30 smithi087 ceph-mon[27999]: fsmap 8 up:standby 2024-04-22T13:29:31.138 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:30 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:29:31.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:30 smithi120 ceph-mon[31101]: Deploying daemon mds.i on smithi132 2024-04-22T13:29:31.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:30 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6836/3391616456,v1:172.21.15.120:6837/3391616456] up:boot 2024-04-22T13:29:31.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:30 smithi120 ceph-mon[31101]: fsmap 8 up:standby 2024-04-22T13:29:31.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:30 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:29:31.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:30 smithi132 ceph-mon[31531]: Deploying daemon mds.i on smithi132 2024-04-22T13:29:31.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:30 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6836/3391616456,v1:172.21.15.120:6837/3391616456] up:boot 2024-04-22T13:29:31.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:30 smithi132 ceph-mon[31531]: fsmap 8 up:standby 2024-04-22T13:29:31.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:30 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:29:31.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 -- 172.21.15.87:0/2388236682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1009f0 msgr2=0x7eff7c10a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:31.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 --2- 172.21.15.87:0/2388236682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1009f0 0x7eff7c10a5a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff74009900 tx=0x7eff7402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 -- 172.21.15.87:0/2388236682 shutdown_connections 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 --2- 172.21.15.87:0/2388236682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1009f0 0x7eff7c10a5a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 --2- 172.21.15.87:0/2388236682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 --2- 172.21.15.87:0/2388236682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1035c0 0x7eff7c1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.226+0000 7eff83474640 1 -- 172.21.15.87:0/2388236682 >> 172.21.15.87:0/2388236682 conn(0x7eff7c0fbb90 msgr2=0x7eff7c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- 172.21.15.87:0/2388236682 shutdown_connections 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- 172.21.15.87:0/2388236682 wait complete. 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 Processor -- start 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- start start 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:31.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1009f0 0x7eff7c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c10db00 con 0x7eff7c1035c0 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c10d980 con 0x7eff7c1009f0 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff83474640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c10dc80 con 0x7eff7c100030 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff81a11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.227+0000 7eff81a11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35088/0 (socket says 172.21.15.87:35088) 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 -- 172.21.15.87:0/673861366 learned_addr learned my addr 172.21.15.87:0/673861366 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81210640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 -- 172.21.15.87:0/673861366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 msgr2=0x7eff7c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 -- 172.21.15.87:0/673861366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1009f0 msgr2=0x7eff7c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1009f0 0x7eff7c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 -- 172.21.15.87:0/673861366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c19fcd0 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81210640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff81a11640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7eff74009230 tx=0x7eff740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74043020 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff74004aa0 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c19ff60 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74003c70 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.228+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c1a04d0 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.229+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7c109480 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.230+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7eff74046020 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.231+0000 7eff727fc640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 0x7eff680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.231+0000 7eff81210640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 0x7eff680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.231+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7eff740f26c0 con 0x7eff7c1035c0 2024-04-22T13:29:31.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.231+0000 7eff81210640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 0x7eff680a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff7c06a360 tx=0x7eff7800a800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:31.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.235+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff7404d020 con 0x7eff7c1035c0 2024-04-22T13:29:31.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.450+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff7c02d5c0 con 0x7eff7c1035c0 2024-04-22T13:29:31.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.451+0000 7eff727fc640 1 -- 172.21.15.87:0/673861366 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7eff740bf540 con 0x7eff7c1035c0 2024-04-22T13:29:31.447 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:31.447 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T13:29:31.447 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 9 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 msgr2=0x7eff680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 0x7eff680a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff7c06a360 tx=0x7eff7800a800 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 msgr2=0x7eff7c19f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7eff74009230 tx=0x7eff740042f0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 shutdown_connections 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7eff680a3c30 0x7eff680a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c1035c0 0x7eff7c19f660 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1009f0 0x7eff7c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 --2- 172.21.15.87:0/673861366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c100030 0x7eff7c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:31.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 >> 172.21.15.87:0/673861366 conn(0x7eff7c0fbb90 msgr2=0x7eff7c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:31.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 shutdown_connections 2024-04-22T13:29:31.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:31.453+0000 7eff83474640 1 -- 172.21.15.87:0/673861366 wait complete. 2024-04-22T13:29:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:31 smithi132 ceph-mon[31531]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:31 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/673861366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:32.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:31 smithi120 ceph-mon[31101]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:32.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:31 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/673861366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:32.226 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 podman[62543]: 2024-04-22 13:29:32.22832967 +0000 UTC m=+0.364078240 container init 927e4bf6acebd101b217c4ac4440655d5a0b3dcc1f9cc138719a2608b67248b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T13:29:32.226 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 podman[62543]: 2024-04-22 13:29:32.230906101 +0000 UTC m=+0.366654673 container start 927e4bf6acebd101b217c4ac4440655d5a0b3dcc1f9cc138719a2608b67248b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:29:32.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:31 smithi087 ceph-mon[27999]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:32.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:31 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/673861366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:32.713 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 ceph-mds[62562]: -- 172.21.15.132:0/3654754348 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d63ebdfa40 con 0x55d63ebb2800 2024-04-22T13:29:32.713 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: starting mds.i at 2024-04-22T13:29:32.713 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 bash[62543]: 927e4bf6acebd101b217c4ac4440655d5a0b3dcc1f9cc138719a2608b67248b4 2024-04-22T13:29:32.713 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:29:32 smithi132 systemd[1]: Started Ceph mds.i for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:33.461 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:33.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 -- 172.21.15.87:0/3586849449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103180 msgr2=0x7fc834103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:33.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 --2- 172.21.15.87:0/3586849449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103180 0x7fc834103600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc824009900 tx=0x7fc82402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:33.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 -- 172.21.15.87:0/3586849449 shutdown_connections 2024-04-22T13:29:33.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 --2- 172.21.15.87:0/3586849449 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103b40 0x7fc83410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 --2- 172.21.15.87:0/3586849449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103180 0x7fc834103600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 --2- 172.21.15.87:0/3586849449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 0x7fc8341023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 -- 172.21.15.87:0/3586849449 >> 172.21.15.87:0/3586849449 conn(0x7fc8340fd500 msgr2=0x7fc8340ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 -- 172.21.15.87:0/3586849449 shutdown_connections 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.564+0000 7fc83c30e640 1 -- 172.21.15.87:0/3586849449 wait complete. 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 Processor -- start 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- start start 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 0x7fc83419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103180 0x7fc83419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:33.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc83410db10 con 0x7fc834103b40 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc83410d990 con 0x7fc834101f90 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc83410dc90 con 0x7fc834103180 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35110/0 (socket says 172.21.15.87:35110) 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 -- 172.21.15.87:0/4221056917 learned_addr learned my addr 172.21.15.87:0/4221056917 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a0aa640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 0x7fc83419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc8398a9640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103180 0x7fc83419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 -- 172.21.15.87:0/4221056917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103180 msgr2=0x7fc83419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103180 0x7fc83419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 -- 172.21.15.87:0/4221056917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 msgr2=0x7fc83419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 0x7fc83419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 -- 172.21.15.87:0/4221056917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8341a2000 con 0x7fc834103b40 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83a8ab640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b970 tx=0x7fc82c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:33.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c019070 con 0x7fc834103b40 2024-04-22T13:29:33.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc82c014440 con 0x7fc834103b40 2024-04-22T13:29:33.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8341a22f0 con 0x7fc834103b40 2024-04-22T13:29:33.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0075d0 con 0x7fc834103b40 2024-04-22T13:29:33.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.565+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8341a2870 con 0x7fc834103b40 2024-04-22T13:29:33.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.566+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8341a2b30 con 0x7fc834103b40 2024-04-22T13:29:33.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.568+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc82c01c070 con 0x7fc834103b40 2024-04-22T13:29:33.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.568+0000 7fc82b7fe640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 0x7fc8200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:33.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.568+0000 7fc83a0aa640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 0x7fc8200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:33.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.568+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc82c0cfad0 con 0x7fc834103b40 2024-04-22T13:29:33.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.568+0000 7fc83a0aa640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 0x7fc8200a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc830005fd0 tx=0x7fc830005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:33.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.572+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc82c09c950 con 0x7fc834103b40 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: Deploying daemon mds.g on smithi087 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:boot 2024-04-22T13:29:33.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: fsmap 9 up:standby 2024-04-22T13:29:33.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:33 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: Deploying daemon mds.g on smithi087 2024-04-22T13:29:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:33.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:boot 2024-04-22T13:29:33.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: fsmap 9 up:standby 2024-04-22T13:29:33.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:33 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: Deploying daemon mds.g on smithi087 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:boot 2024-04-22T13:29:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: fsmap 9 up:standby 2024-04-22T13:29:33.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:33 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:29:33.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.798+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8341a2d00 con 0x7fc834103b40 2024-04-22T13:29:33.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.799+0000 7fc82b7fe640 1 -- 172.21.15.87:0/4221056917 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7fc82c09c250 con 0x7fc834103b40 2024-04-22T13:29:33.796 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:33.796 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T13:29:33.796 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 10 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 msgr2=0x7fc8200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 0x7fc8200a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc830005fd0 tx=0x7fc830005850 comp rx=0 tx=0).stop 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 msgr2=0x7fc8341a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b970 tx=0x7fc82c00be30 comp rx=0 tx=0).stop 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 shutdown_connections 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8200a3bb0 0x7fc8200a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc834103b40 0x7fc8341a1990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc834103180 0x7fc83419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 --2- 172.21.15.87:0/4221056917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc834101f90 0x7fc83419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:33.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 >> 172.21.15.87:0/4221056917 conn(0x7fc8340fd500 msgr2=0x7fc834074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:33.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 shutdown_connections 2024-04-22T13:29:33.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:33.801+0000 7fc83c30e640 1 -- 172.21.15.87:0/4221056917 wait complete. 2024-04-22T13:29:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:34 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4221056917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:34.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:34 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4221056917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:34 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4221056917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:34.737 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:34 smithi087 systemd[1]: Starting Ceph mds.g for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:35.035 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:34 smithi087 podman[66736]: 2024-04-22 13:29:34.691386305 +0000 UTC m=+0.017676899 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:35.035 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:34 smithi087 podman[66736]: 2024-04-22 13:29:34.918093213 +0000 UTC m=+0.244383832 container create 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 podman[66736]: 2024-04-22 13:29:35.171025806 +0000 UTC m=+0.497316366 container init 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 podman[66736]: 2024-04-22 13:29:35.173621492 +0000 UTC m=+0.499912060 container start 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mds[66755]: -- 172.21.15.87:0/1424646840 <== mon.0 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b134777a40 con 0x55b13474b000 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g[66751]: starting mds.g at 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 bash[66736]: 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 2024-04-22T13:29:35.411 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:29:35 smithi087 systemd[1]: Started Ceph mds.g for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:35 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:35 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:35.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:35 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:35.810 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:35.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.920+0000 7fd20c46a640 1 -- 172.21.15.87:0/2297317013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204100e70 msgr2=0x7fd204101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:35.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.920+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2297317013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204100e70 0x7fd204101290 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd200009900 tx=0x7fd20002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:35.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 -- 172.21.15.87:0/2297317013 shutdown_connections 2024-04-22T13:29:35.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2297317013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204102a10 0x7fd204109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:35.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2297317013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 0x7fd2041024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:35.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2297317013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204100e70 0x7fd204101290 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:35.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 -- 172.21.15.87:0/2297317013 >> 172.21.15.87:0/2297317013 conn(0x7fd2040fc470 msgr2=0x7fd2040fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:35.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 -- 172.21.15.87:0/2297317013 shutdown_connections 2024-04-22T13:29:35.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.921+0000 7fd20c46a640 1 -- 172.21.15.87:0/2297317013 wait complete. 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 Processor -- start 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 -- start start 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204100e70 0x7fd2041970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 0x7fd2041975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd20410ca00 con 0x7fd204102a10 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd20410c880 con 0x7fd204100e70 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20c46a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd20410cb80 con 0x7fd204102050 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35138/0 (socket says 172.21.15.87:35138) 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 -- 172.21.15.87:0/2184517271 learned_addr learned my addr 172.21.15.87:0/2184517271 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20a206640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204100e70 0x7fd2041970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:35.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd209a05640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 0x7fd2041975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 -- 172.21.15.87:0/2184517271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 msgr2=0x7fd2041975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 0x7fd2041975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 -- 172.21.15.87:0/2184517271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204100e70 msgr2=0x7fd2041970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204100e70 0x7fd2041970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.922+0000 7fd20aa07640 1 -- 172.21.15.87:0/2184517271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20419ecc0 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd20aa07640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00eaa0 tx=0x7fd1fc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc00cdb0 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20419efb0 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1fc005020 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0116d0 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.923+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20419f470 con 0x7fd204102a10 2024-04-22T13:29:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.924+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd204108210 con 0x7fd204102a10 2024-04-22T13:29:35.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.925+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd1fc0054a0 con 0x7fd204102a10 2024-04-22T13:29:35.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.925+0000 7fd1fb7fe640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 0x7fd1f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:35.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.925+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd1fc0d0030 con 0x7fd204102a10 2024-04-22T13:29:35.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.926+0000 7fd20a206640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 0x7fd1f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:35.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.926+0000 7fd20a206640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 0x7fd1f00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd200005ab0 tx=0x7fd200005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:35.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:35.930+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd1fc016030 con 0x7fd204102a10 2024-04-22T13:29:36.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.150+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd20402d5c0 con 0x7fd204102a10 2024-04-22T13:29:36.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.151+0000 7fd1fb7fe640 1 -- 172.21.15.87:0/2184517271 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7fd1fc09c860 con 0x7fd204102a10 2024-04-22T13:29:36.148 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:36.148 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T13:29:36.149 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 11 2024-04-22T13:29:36.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 msgr2=0x7fd1f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:36.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 0x7fd1f00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd200005ab0 tx=0x7fd200005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:36.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 msgr2=0x7fd20419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:36.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00eaa0 tx=0x7fd1fc00ef60 comp rx=0 tx=0).stop 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 shutdown_connections 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1f00a3bb0 0x7fd1f00a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd204102a10 0x7fd20419e650 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd204102050 0x7fd2041975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 --2- 172.21.15.87:0/2184517271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd204100e70 0x7fd2041970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.153+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 >> 172.21.15.87:0/2184517271 conn(0x7fd2040fc470 msgr2=0x7fd2041053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.154+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 shutdown_connections 2024-04-22T13:29:36.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:36.154+0000 7fd20c46a640 1 -- 172.21.15.87:0/2184517271 wait complete. 2024-04-22T13:29:36.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:36 smithi120 ceph-mon[31101]: Deploying daemon mds.a on smithi087 2024-04-22T13:29:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:36 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.87:6838/2616185101,v1:172.21.15.87:6839/2616185101] up:boot 2024-04-22T13:29:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:36 smithi120 ceph-mon[31101]: fsmap 10 up:standby 2024-04-22T13:29:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:29:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:36 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2184517271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:36 smithi087 ceph-mon[27999]: Deploying daemon mds.a on smithi087 2024-04-22T13:29:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:36 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.87:6838/2616185101,v1:172.21.15.87:6839/2616185101] up:boot 2024-04-22T13:29:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:36 smithi087 ceph-mon[27999]: fsmap 10 up:standby 2024-04-22T13:29:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:29:36.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:36 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2184517271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:36 smithi132 ceph-mon[31531]: Deploying daemon mds.a on smithi087 2024-04-22T13:29:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:36 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.87:6838/2616185101,v1:172.21.15.87:6839/2616185101] up:boot 2024-04-22T13:29:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:36 smithi132 ceph-mon[31531]: fsmap 10 up:standby 2024-04-22T13:29:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:29:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:36 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2184517271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:37 smithi120 ceph-mon[31101]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:37.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:37 smithi132 ceph-mon[31531]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:37.723 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:37 smithi087 ceph-mon[27999]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:37.724 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:37 smithi087 systemd[1]: Starting Ceph mds.a for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:37.986 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:37 smithi087 podman[67103]: 2024-04-22 13:29:37.6314725 +0000 UTC m=+0.018339584 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:37.987 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:37 smithi087 podman[67103]: 2024-04-22 13:29:37.756490599 +0000 UTC m=+0.143357724 container create 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T13:29:38.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:38.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 -- 172.21.15.87:0/82521952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890100e70 msgr2=0x7fc890101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 --2- 172.21.15.87:0/82521952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890100e70 0x7fc890101290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc88c009900 tx=0x7fc88c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 -- 172.21.15.87:0/82521952 shutdown_connections 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 --2- 172.21.15.87:0/82521952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc890102980 0x7fc8901092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 --2- 172.21.15.87:0/82521952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890101fc0 0x7fc890102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 --2- 172.21.15.87:0/82521952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890100e70 0x7fc890101290 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 -- 172.21.15.87:0/82521952 >> 172.21.15.87:0/82521952 conn(0x7fc8900fc3e0 msgr2=0x7fc8900fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 -- 172.21.15.87:0/82521952 shutdown_connections 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.274+0000 7fc8972a8640 1 -- 172.21.15.87:0/82521952 wait complete. 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 Processor -- start 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 -- start start 2024-04-22T13:29:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc890100e70 0x7fc890198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890102980 0x7fc89019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc89010dad0 con 0x7fc890101fc0 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc89010d950 con 0x7fc890102980 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc8972a8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc89010dc50 con 0x7fc890100e70 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35182/0 (socket says 172.21.15.87:35182) 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 -- 172.21.15.87:0/3378141773 learned_addr learned my addr 172.21.15.87:0/3378141773 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc895845640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890102980 0x7fc89019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 -- 172.21.15.87:0/3378141773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc890100e70 msgr2=0x7fc890198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:38.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc890100e70 0x7fc890198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 -- 172.21.15.87:0/3378141773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890102980 msgr2=0x7fc89019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890102980 0x7fc89019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.275+0000 7fc894843640 1 -- 172.21.15.87:0/3378141773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89019fd50 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc894843640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc88000e9c0 tx=0x7fc88000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88000cd80 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8800035f0 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89019ffe0 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880010690 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.276+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8901a0500 con 0x7fc890101fc0 2024-04-22T13:29:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.277+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc890108250 con 0x7fc890101fc0 2024-04-22T13:29:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.278+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc8800037a0 con 0x7fc890101fc0 2024-04-22T13:29:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.278+0000 7fc8867fc640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 0x7fc87c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.279+0000 7fc895044640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 0x7fc87c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.279+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc8800d0420 con 0x7fc890101fc0 2024-04-22T13:29:38.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.279+0000 7fc895044640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 0x7fc87c0a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc88c036040 tx=0x7fc88c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:38.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.282+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc88009d2a0 con 0x7fc890101fc0 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 podman[67103]: 2024-04-22 13:29:38.02578747 +0000 UTC m=+0.412654536 container init 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 podman[67103]: 2024-04-22 13:29:38.028339855 +0000 UTC m=+0.415206924 container start 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5) 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 ceph-mds[67122]: -- 172.21.15.87:0/2399522459 <== mon.0 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556dd88fa40 con 0x5556dd862c00 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a[67118]: starting mds.a at 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 bash[67103]: 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 2024-04-22T13:29:38.487 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:29:38 smithi087 systemd[1]: Started Ceph mds.a for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:38.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.500+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc89002d5c0 con 0x7fc890101fc0 2024-04-22T13:29:38.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.500+0000 7fc8867fc640 1 -- 172.21.15.87:0/3378141773 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fc88009cba0 con 0x7fc890101fc0 2024-04-22T13:29:38.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:38.496 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T13:29:38.497 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 12 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.502+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 msgr2=0x7fc87c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.502+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 0x7fc87c0a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc88c036040 tx=0x7fc88c000bc0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.502+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 msgr2=0x7fc890198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.502+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc88000e9c0 tx=0x7fc88000ee80 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.502+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 shutdown_connections 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc87c0a3bb0 0x7fc87c0a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc890102980 0x7fc89019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc890101fc0 0x7fc890198670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 --2- 172.21.15.87:0/3378141773 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc890100e70 0x7fc890198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:38.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 >> 172.21.15.87:0/3378141773 conn(0x7fc8900fc3e0 msgr2=0x7fc890105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:38.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 shutdown_connections 2024-04-22T13:29:38.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:38.503+0000 7fc8972a8640 1 -- 172.21.15.87:0/3378141773 wait complete. 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: Deploying daemon mds.b on smithi120 2024-04-22T13:29:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.87:6840/982852061,v1:172.21.15.87:6841/982852061] up:boot 2024-04-22T13:29:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: fsmap 11 up:standby 2024-04-22T13:29:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:29:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3378141773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:39.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:39 smithi120 ceph-mon[31101]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: Deploying daemon mds.b on smithi120 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.87:6840/982852061,v1:172.21.15.87:6841/982852061] up:boot 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: fsmap 11 up:standby 2024-04-22T13:29:39.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:29:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3378141773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:39.464 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:39 smithi132 ceph-mon[31531]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: Deploying daemon mds.b on smithi120 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.87:6840/982852061,v1:172.21.15.87:6841/982852061] up:boot 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: fsmap 11 up:standby 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3378141773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:39.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:39 smithi087 ceph-mon[27999]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:40.511 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:40.605 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 systemd[1]: Starting Ceph mds.b for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- 172.21.15.87:0/2304988703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714101810 msgr2=0x7fa714101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- 172.21.15.87:0/2304988703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714101810 0x7fa714101c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa70400aa10 tx=0x7fa70402cf70 comp rx=0 tx=0).stop 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- 172.21.15.87:0/2304988703 shutdown_connections 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- 172.21.15.87:0/2304988703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 0x7fa714108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- 172.21.15.87:0/2304988703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714101810 0x7fa714101c90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- 172.21.15.87:0/2304988703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714100620 0x7fa714100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- 172.21.15.87:0/2304988703 >> 172.21.15.87:0/2304988703 conn(0x7fa7140fbb90 msgr2=0x7fa7140fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- 172.21.15.87:0/2304988703 shutdown_connections 2024-04-22T13:29:40.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- 172.21.15.87:0/2304988703 wait complete. 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 Processor -- start 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- start start 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 0x7fa714198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 0x7fa71419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa71410b920 con 0x7fa714100620 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa71410b7a0 con 0x7fa714101810 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.626+0000 7fa71c021640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa71410baa0 con 0x7fa7141021d0 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39982/0 (socket says 172.21.15.87:39982) 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa7195bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 0x7fa714198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 -- 172.21.15.87:0/2737114390 learned_addr learned my addr 172.21.15.87:0/2737114390 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa71a5be640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 0x7fa71419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:40.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 -- 172.21.15.87:0/2737114390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 msgr2=0x7fa71419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 0x7fa71419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 -- 172.21.15.87:0/2737114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 msgr2=0x7fa714198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 0x7fa714198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 -- 172.21.15.87:0/2737114390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71419fdf0 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa7195bc640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 0x7fa714198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa719dbd640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa71000e9c0 tx=0x7fa71000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71000cd80 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7100035f0 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7141a00e0 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710010690 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.627+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7141a0680 con 0x7fa714100620 2024-04-22T13:29:40.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.628+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7141a0940 con 0x7fa714100620 2024-04-22T13:29:40.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.629+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa7100037a0 con 0x7fa714100620 2024-04-22T13:29:40.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.630+0000 7fa70affd640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 0x7fa7000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:40.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.630+0000 7fa7195bc640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 0x7fa7000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:40.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.630+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa7100d0370 con 0x7fa714100620 2024-04-22T13:29:40.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.630+0000 7fa7195bc640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 0x7fa7000a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa704004c20 tx=0x7fa704005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:40.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.635+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa71009d1f0 con 0x7fa714100620 2024-04-22T13:29:40.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.850+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7141a0b10 con 0x7fa714100620 2024-04-22T13:29:40.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.851+0000 7fa70affd640 1 -- 172.21.15.87:0/2737114390 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fa71009caf0 con 0x7fa714100620 2024-04-22T13:29:40.847 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:40.847 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T13:29:40.847 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 12 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 msgr2=0x7fa7000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 0x7fa7000a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa704004c20 tx=0x7fa704005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 msgr2=0x7fa7141981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa71000e9c0 tx=0x7fa71000ee80 comp rx=0 tx=0).stop 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 shutdown_connections 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa7000a3bb0 0x7fa7000a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7141021d0 0x7fa71419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa714101810 0x7fa714198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 --2- 172.21.15.87:0/2737114390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa714100620 0x7fa7141981d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:40.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 >> 172.21.15.87:0/2737114390 conn(0x7fa7140fbb90 msgr2=0x7fa714106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:40.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 shutdown_connections 2024-04-22T13:29:40.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:40.853+0000 7fa71c021640 1 -- 172.21.15.87:0/2737114390 wait complete. 2024-04-22T13:29:40.869 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 podman[61423]: 2024-04-22 13:29:40.513213687 +0000 UTC m=+0.018182741 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:29:40.869 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 podman[61423]: 2024-04-22 13:29:40.621199509 +0000 UTC m=+0.126168544 container create 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:29:40.870 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 podman[61423]: 2024-04-22 13:29:40.849420163 +0000 UTC m=+0.354389202 container init 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T13:29:40.870 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 podman[61423]: 2024-04-22 13:29:40.852048269 +0000 UTC m=+0.357017306 container start 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T13:29:41.169 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 ceph-mds[61442]: -- 172.21.15.120:0/2786199080 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55655b5a5a40 con 0x55655b578c00 2024-04-22T13:29:41.170 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: starting mds.b at 2024-04-22T13:29:41.170 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 bash[61423]: 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d 2024-04-22T13:29:41.170 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:40 smithi120 systemd[1]: Started Ceph mds.b for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2737114390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.108 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:41 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2737114390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.120 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:41 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:42.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2737114390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.126 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:42.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:41 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:42.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 -- 172.21.15.87:0/1105204243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40100e70 msgr2=0x7f9d40101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/1105204243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40100e70 0x7f9d40101290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d30009900 tx=0x7f9d3002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 -- 172.21.15.87:0/1105204243 shutdown_connections 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/1105204243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40102980 0x7f9d401092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/1105204243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40101fc0 0x7f9d40102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/1105204243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40100e70 0x7f9d40101290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 -- 172.21.15.87:0/1105204243 >> 172.21.15.87:0/1105204243 conn(0x7f9d400fc3e0 msgr2=0x7f9d400fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 -- 172.21.15.87:0/1105204243 shutdown_connections 2024-04-22T13:29:42.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 -- 172.21.15.87:0/1105204243 wait complete. 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.975+0000 7f9d45ceb640 1 Processor -- start 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 -- start start 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40100e70 0x7f9d40069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40102980 0x7f9d4019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010c9f0 con 0x7f9d40101fc0 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010c870 con 0x7f9d40102980 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d45ceb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010cb70 con 0x7f9d40100e70 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40004/0 (socket says 172.21.15.87:40004) 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 -- 172.21.15.87:0/2937172387 learned_addr learned my addr 172.21.15.87:0/2937172387 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3f7fe640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40100e70 0x7f9d40069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3ffff640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40102980 0x7f9d4019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:42.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 -- 172.21.15.87:0/2937172387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40100e70 msgr2=0x7f9d40069cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40100e70 0x7f9d40069cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 -- 172.21.15.87:0/2937172387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40102980 msgr2=0x7f9d4019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40102980 0x7f9d4019f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 -- 172.21.15.87:0/2937172387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4019fdb0 con 0x7f9d40101fc0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.976+0000 7f9d3effd640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400d950 tx=0x7f9d3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.977+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34017020 con 0x7f9d40101fc0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.977+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d401a00a0 con 0x7f9d40101fc0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.977+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d34009c30 con 0x7f9d40101fc0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.977+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400f6b0 con 0x7f9d40101fc0 2024-04-22T13:29:42.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.977+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d401a0640 con 0x7f9d40101fc0 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.978+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d40108200 con 0x7f9d40101fc0 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.979+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9d3401a020 con 0x7f9d40101fc0 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.979+0000 7f9d3cff9640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 0x7f9d280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.980+0000 7f9d3f7fe640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 0x7f9d280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.980+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9d340cfb40 con 0x7f9d40101fc0 2024-04-22T13:29:42.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.980+0000 7f9d3f7fe640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 0x7f9d280a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d30000c00 tx=0x7f9d3002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:42.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:42.984+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d34009de0 con 0x7f9d40101fc0 2024-04-22T13:29:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:42 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:boot 2024-04-22T13:29:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:42 smithi120 ceph-mon[31101]: fsmap 12 up:standby 2024-04-22T13:29:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:29:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:42 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.204+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d4002d7c0 con 0x7f9d40101fc0 2024-04-22T13:29:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.205+0000 7f9d3cff9640 1 -- 172.21.15.87:0/2937172387 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f9d3409c9c0 con 0x7f9d40101fc0 2024-04-22T13:29:43.200 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:43.201 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24407,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-22T13:29:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 13 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 msgr2=0x7f9d280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 0x7f9d280a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d30000c00 tx=0x7f9d3002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 msgr2=0x7f9d4019ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400d950 tx=0x7f9d3400de10 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 shutdown_connections 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d280a3bb0 0x7f9d280a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d40102980 0x7f9d4019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d40101fc0 0x7f9d4019ce80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 --2- 172.21.15.87:0/2937172387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d40100e70 0x7f9d40069cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 >> 172.21.15.87:0/2937172387 conn(0x7f9d400fc3e0 msgr2=0x7f9d401053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 shutdown_connections 2024-04-22T13:29:43.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.207+0000 7f9d45ceb640 1 -- 172.21.15.87:0/2937172387 wait complete. 2024-04-22T13:29:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:42 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:boot 2024-04-22T13:29:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:42 smithi132 ceph-mon[31531]: fsmap 12 up:standby 2024-04-22T13:29:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:29:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:42 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.213 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:42 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:boot 2024-04-22T13:29:43.213 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:42 smithi087 ceph-mon[27999]: fsmap 12 up:standby 2024-04-22T13:29:43.213 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:29:43.213 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:42 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- 172.21.15.87:0/2079211928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa698101fe0 msgr2=0x7fa698102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2079211928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa698101fe0 0x7fa698102460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa694009900 tx=0x7fa69402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- 172.21.15.87:0/2079211928 shutdown_connections 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2079211928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6981029a0 0x7fa6981092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2079211928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa698101fe0 0x7fa698102460 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2079211928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa698100e90 0x7fa6981012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- 172.21.15.87:0/2079211928 >> 172.21.15.87:0/2079211928 conn(0x7fa6980fc420 msgr2=0x7fa6980fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- 172.21.15.87:0/2079211928 shutdown_connections 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- 172.21.15.87:0/2079211928 wait complete. 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 Processor -- start 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- start start 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa698100e90 0x7fa6981981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa698101fe0 0x7fa698198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810c9d0 con 0x7fa6981029a0 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.322+0000 7fa6a0226640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810c850 con 0x7fa698101fe0 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa6a0226640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810cb50 con 0x7fa698100e90 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40016/0 (socket says 172.21.15.87:40016) 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 -- 172.21.15.87:0/2699355988 learned_addr learned my addr 172.21.15.87:0/2699355988 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69dfc2640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa698100e90 0x7fa6981981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 -- 172.21.15.87:0/2699355988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa698100e90 msgr2=0x7fa6981981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa698100e90 0x7fa6981981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 -- 172.21.15.87:0/2699355988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa698101fe0 msgr2=0x7fa698198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa698101fe0 0x7fa698198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 -- 172.21.15.87:0/2699355988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69819fe10 con 0x7fa6981029a0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa69e7c3640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa69000dac0 tx=0x7fa69000df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690017070 con 0x7fa6981029a0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6981a0100 con 0x7fa6981029a0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa69000be30 con 0x7fa6981029a0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690010460 con 0x7fa6981029a0 2024-04-22T13:29:43.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.323+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6981a06a0 con 0x7fa6981029a0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.325+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6981a08a0 con 0x7fa6981029a0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.325+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa690003f90 con 0x7fa6981029a0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.326+0000 7fa68effd640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 0x7fa6840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.326+0000 7fa69dfc2640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 0x7fa6840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.326+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6900d01b0 con 0x7fa6981029a0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.326+0000 7fa69dfc2640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 0x7fa6840a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6880058c0 tx=0x7fa68800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.330+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa690020020 con 0x7fa6981029a0 2024-04-22T13:29:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.537+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6981a0fb0 con 0x7fa6981029a0 2024-04-22T13:29:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.538+0000 7fa68effd640 1 -- 172.21.15.87:0/2699355988 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18475 (secure 0 0 0) 0x7fa69009d030 con 0x7fa6981029a0 2024-04-22T13:29:43.535 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:43.535 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":81,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:28:53.811375+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:29:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 msgr2=0x7fa6840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 0x7fa6840a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6880058c0 tx=0x7fa68800a2d0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 msgr2=0x7fa69819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa69000dac0 tx=0x7fa69000df80 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 shutdown_connections 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6840a3bb0 0x7fa6840a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6981029a0 0x7fa69819f7a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa698101fe0 0x7fa698198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 --2- 172.21.15.87:0/2699355988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa698100e90 0x7fa6981981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 >> 172.21.15.87:0/2699355988 conn(0x7fa6980fc420 msgr2=0x7fa6980fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 shutdown_connections 2024-04-22T13:29:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.540+0000 7fa6a0226640 1 -- 172.21.15.87:0/2699355988 wait complete. 2024-04-22T13:29:43.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:29:43.549 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:29:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 -- 172.21.15.87:0/3379610262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 msgr2=0x7f43b0104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 --2- 172.21.15.87:0/3379610262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0104e00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009900 tx=0x7f43a002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 -- 172.21.15.87:0/3379610262 shutdown_connections 2024-04-22T13:29:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 --2- 172.21.15.87:0/3379610262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0107c70 0x7f43b010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 --2- 172.21.15.87:0/3379610262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0105340 0x7f43b0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 --2- 172.21.15.87:0/3379610262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0104e00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 -- 172.21.15.87:0/3379610262 >> 172.21.15.87:0/3379610262 conn(0x7f43b00fc400 msgr2=0x7f43b00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 -- 172.21.15.87:0/3379610262 shutdown_connections 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 -- 172.21.15.87:0/3379610262 wait complete. 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.660+0000 7f43b529a640 1 Processor -- start 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 -- start start 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0105340 0x7f43b01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0107c70 0x7f43b019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f43b010dac0 con 0x7f43b0102a10 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43b010d940 con 0x7f43b0107c70 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43b529a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f43b010dc40 con 0x7f43b0105340 2024-04-22T13:29:43.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40034/0 (socket says 172.21.15.87:40034) 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 -- 172.21.15.87:0/3533329358 learned_addr learned my addr 172.21.15.87:0/3533329358 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43af7fe640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0107c70 0x7f43b019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43a7fff640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0105340 0x7f43b01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 -- 172.21.15.87:0/3533329358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0105340 msgr2=0x7f43b01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0105340 0x7f43b01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 -- 172.21.15.87:0/3533329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0107c70 msgr2=0x7f43b019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0107c70 0x7f43b019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 -- 172.21.15.87:0/3533329358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b019fe90 con 0x7f43b0102a10 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.661+0000 7f43aeffd640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43a0006010 tx=0x7f43a0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.662+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0043020 con 0x7f43b0102a10 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.662+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b01a0120 con 0x7f43b0102a10 2024-04-22T13:29:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.662+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a0004440 con 0x7f43b0102a10 2024-04-22T13:29:43.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.662+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0003bf0 con 0x7f43b0102a10 2024-04-22T13:29:43.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.662+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b01a06c0 con 0x7f43b0102a10 2024-04-22T13:29:43.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.663+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b00fdef0 con 0x7f43b0102a10 2024-04-22T13:29:43.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.664+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f43a0046020 con 0x7f43b0102a10 2024-04-22T13:29:43.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.665+0000 7f43acff9640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 0x7f439c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.665+0000 7f43a7fff640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 0x7f439c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.665+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f43a00f2810 con 0x7f43b0102a10 2024-04-22T13:29:43.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.665+0000 7f43a7fff640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 0x7f439c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43b0199780 tx=0x7f4398005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.669+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43a004d020 con 0x7f43b0102a10 2024-04-22T13:29:43.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:43.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.855+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f43b0062b50 con 0x7f43b0102a10 2024-04-22T13:29:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.855+0000 7f43acff9640 1 -- 172.21.15.87:0/3533329358 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f43a00bf690 con 0x7f43b0102a10 2024-04-22T13:29:43.851 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:29:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 msgr2=0x7f439c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 0x7f439c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43b0199780 tx=0x7f4398005f90 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 msgr2=0x7f43b0198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f43a0006010 tx=0x7f43a0004040 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 shutdown_connections 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f439c0a3c30 0x7f439c0a60f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.857+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43b0107c70 0x7f43b019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.858+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43b0105340 0x7f43b01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.858+0000 7f43b529a640 1 --2- 172.21.15.87:0/3533329358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43b0102a10 0x7f43b0198270 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.858+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 >> 172.21.15.87:0/3533329358 conn(0x7f43b00fc400 msgr2=0x7f43b01034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.858+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 shutdown_connections 2024-04-22T13:29:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.858+0000 7f43b529a640 1 -- 172.21.15.87:0/3533329358 wait complete. 2024-04-22T13:29:43.865 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T13:29:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 -- 172.21.15.87:0/3391153484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 msgr2=0x7f31741012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 --2- 172.21.15.87:0/3391153484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f31741012b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3170009900 tx=0x7f317002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 -- 172.21.15.87:0/3391153484 shutdown_connections 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 --2- 172.21.15.87:0/3391153484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 0x7f31741092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 --2- 172.21.15.87:0/3391153484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 0x7f3174102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 --2- 172.21.15.87:0/3391153484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f31741012b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 -- 172.21.15.87:0/3391153484 >> 172.21.15.87:0/3391153484 conn(0x7f31740fc400 msgr2=0x7f31740fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 -- 172.21.15.87:0/3391153484 shutdown_connections 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.969+0000 7f317be2e640 1 -- 172.21.15.87:0/3391153484 wait complete. 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 Processor -- start 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 -- start start 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 0x7f31741987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 0x7f317419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f317410c9e0 con 0x7f3174100e90 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f317410c860 con 0x7f3174101fe0 2024-04-22T13:29:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317be2e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f317410cb60 con 0x7f31741029a0 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40046/0 (socket says 172.21.15.87:40046) 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 -- 172.21.15.87:0/1339629281 learned_addr learned my addr 172.21.15.87:0/1339629281 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f317a3cb640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 0x7f317419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f31793c9640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 0x7f31741987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 -- 172.21.15.87:0/1339629281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 msgr2=0x7f317419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 0x7f317419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 -- 172.21.15.87:0/1339629281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 msgr2=0x7f31741987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 0x7f31741987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 -- 172.21.15.87:0/1339629281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317419fe30 con 0x7f3174100e90 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.970+0000 7f3179bca640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3170006010 tx=0x7f3170005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.971+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170043020 con 0x7f3174100e90 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.971+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31700043d0 con 0x7f3174100e90 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.971+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31741a00c0 con 0x7f3174100e90 2024-04-22T13:29:43.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.971+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170003b80 con 0x7f3174100e90 2024-04-22T13:29:43.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.971+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31741a0610 con 0x7f3174100e90 2024-04-22T13:29:43.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.972+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31741a0810 con 0x7f3174100e90 2024-04-22T13:29:43.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.973+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3170046020 con 0x7f3174100e90 2024-04-22T13:29:43.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.973+0000 7f316affd640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 0x7f31600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:43.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.973+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f31700f2130 con 0x7f3174100e90 2024-04-22T13:29:43.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.973+0000 7f31793c9640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 0x7f31600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:43.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.974+0000 7f31793c9640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 0x7f31600a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3174199720 tx=0x7f3164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:43.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:43.978+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f317004d020 con 0x7f3174100e90 2024-04-22T13:29:44.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2937172387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2699355988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:43 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.186+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f31741a09e0 con 0x7f3174100e90 2024-04-22T13:29:44.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.187+0000 7f316affd640 1 -- 172.21.15.87:0/1339629281 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f31700befb0 con 0x7f3174100e90 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout:[ 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout: "poolnum": 1, 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout: "poolname": ".mgr" 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:29:44.183 INFO:teuthology.orchestra.run.smithi087.stdout:] 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 msgr2=0x7f31600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 0x7f31600a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3174199720 tx=0x7f3164005f90 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 msgr2=0x7f3174198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3170006010 tx=0x7f3170005e40 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 shutdown_connections 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f31600a3bb0 0x7f31600a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741029a0 0x7f317419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3174101fe0 0x7f31741987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 --2- 172.21.15.87:0/1339629281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3174100e90 0x7f3174198260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 >> 172.21.15.87:0/1339629281 conn(0x7f31740fc400 msgr2=0x7f31740fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:44.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 shutdown_connections 2024-04-22T13:29:44.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.189+0000 7f317be2e640 1 -- 172.21.15.87:0/1339629281 wait complete. 2024-04-22T13:29:44.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2937172387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:44.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:44.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2699355988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:43 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.200 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T13:29:44.200 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2937172387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2699355988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:43 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:29:44.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 -- 172.21.15.87:0/2010296839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 msgr2=0x7fdf7c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 --2- 172.21.15.87:0/2010296839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c104e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009900 tx=0x7fdf7002bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 -- 172.21.15.87:0/2010296839 shutdown_connections 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 --2- 172.21.15.87:0/2010296839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c107c80 0x7fdf7c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 --2- 172.21.15.87:0/2010296839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c105350 0x7fdf7c107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 --2- 172.21.15.87:0/2010296839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c104e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 -- 172.21.15.87:0/2010296839 >> 172.21.15.87:0/2010296839 conn(0x7fdf7c0fc410 msgr2=0x7fdf7c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 -- 172.21.15.87:0/2010296839 shutdown_connections 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.316+0000 7fdf831ef640 1 -- 172.21.15.87:0/2010296839 wait complete. 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 Processor -- start 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 -- start start 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c105350 0x7fdf7c1009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c107c80 0x7fdf7c100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:44.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c10dae0 con 0x7fdf7c102a20 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c10d960 con 0x7fdf7c107c80 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf831ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c10dc60 con 0x7fdf7c105350 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40054/0 (socket says 172.21.15.87:40054) 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 -- 172.21.15.87:0/491862220 learned_addr learned my addr 172.21.15.87:0/491862220 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf7bfff640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c105350 0x7fdf7c1009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf8178c640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c107c80 0x7fdf7c100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 -- 172.21.15.87:0/491862220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c105350 msgr2=0x7fdf7c1009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c105350 0x7fdf7c1009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 -- 172.21.15.87:0/491862220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c107c80 msgr2=0x7fdf7c100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c107c80 0x7fdf7c100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.317+0000 7fdf80f8b640 1 -- 172.21.15.87:0/491862220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c1017d0 con 0x7fdf7c102a20 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf80f8b640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006010 tx=0x7fdf70004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:44.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70043020 con 0x7fdf7c102a20 2024-04-22T13:29:44.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf70004440 con 0x7fdf7c102a20 2024-04-22T13:29:44.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70003bf0 con 0x7fdf7c102a20 2024-04-22T13:29:44.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c1a2320 con 0x7fdf7c102a20 2024-04-22T13:29:44.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.318+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c1a2890 con 0x7fdf7c102a20 2024-04-22T13:29:44.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.319+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c0fdf00 con 0x7fdf7c102a20 2024-04-22T13:29:44.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.320+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf70046020 con 0x7fdf7c102a20 2024-04-22T13:29:44.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.320+0000 7fdf79ffb640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 0x7fdf6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:44.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.321+0000 7fdf7bfff640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 0x7fdf6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:44.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.321+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdf700f21c0 con 0x7fdf7c102a20 2024-04-22T13:29:44.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.321+0000 7fdf7bfff640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 0x7fdf6c0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c1020b0 tx=0x7fdf68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:44.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.325+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf7004d020 con 0x7fdf7c102a20 2024-04-22T13:29:44.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:44.507+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fdf7c02d5c0 con 0x7fdf7c102a20 2024-04-22T13:29:45.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:44 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3533329358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:45.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:44 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1339629281' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T13:29:45.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:44 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T13:29:45.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:44 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3533329358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:45.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:44 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1339629281' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T13:29:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:44 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T13:29:45.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:44 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3533329358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:29:45.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:44 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1339629281' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T13:29:45.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:44 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T13:29:45.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.306+0000 7fdf79ffb640 1 -- 172.21.15.87:0/491862220 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fdf700bf040 con 0x7fdf7c102a20 2024-04-22T13:29:45.303 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cephfs_metadata' created 2024-04-22T13:29:45.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 msgr2=0x7fdf6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 0x7fdf6c0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c1020b0 tx=0x7fdf68005f90 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 msgr2=0x7fdf7c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006010 tx=0x7fdf70004040 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 shutdown_connections 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdf6c0a3c30 0x7fdf6c0a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf7c107c80 0x7fdf7c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf7c105350 0x7fdf7c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 --2- 172.21.15.87:0/491862220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf7c102a20 0x7fdf7c102340 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.310+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 >> 172.21.15.87:0/491862220 conn(0x7fdf7c0fc410 msgr2=0x7fdf7c103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.312+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 shutdown_connections 2024-04-22T13:29:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.312+0000 7fdf831ef640 1 -- 172.21.15.87:0/491862220 wait complete. 2024-04-22T13:29:45.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 -- 172.21.15.87:0/4050493315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101b10 msgr2=0x7f078c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 --2- 172.21.15.87:0/4050493315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101b10 0x7f078c1093b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0784009ca0 tx=0x7f078402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 -- 172.21.15.87:0/4050493315 shutdown_connections 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 --2- 172.21.15.87:0/4050493315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101b10 0x7f078c1093b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 --2- 172.21.15.87:0/4050493315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101150 0x7f078c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 --2- 172.21.15.87:0/4050493315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 0x7f078c102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 -- 172.21.15.87:0/4050493315 >> 172.21.15.87:0/4050493315 conn(0x7f078c0fbb80 msgr2=0x7f078c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:45.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 -- 172.21.15.87:0/4050493315 shutdown_connections 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.439+0000 7f079223e640 1 -- 172.21.15.87:0/4050493315 wait complete. 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 Processor -- start 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 -- start start 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 0x7f078c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 0x7f078c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f078c10c9a0 con 0x7f078c101150 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f078c10c820 con 0x7f078c101b10 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f079223e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f078c10cb20 con 0x7f078c102420 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f078b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f078b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40078/0 (socket says 172.21.15.87:40078) 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f078b7fe640 1 -- 172.21.15.87:0/3130903126 learned_addr learned my addr 172.21.15.87:0/3130903126 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.440+0000 7f078bfff640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 0x7f078c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:45.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078affd640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 0x7f078c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 -- 172.21.15.87:0/3130903126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 msgr2=0x7f078c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 0x7f078c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 -- 172.21.15.87:0/3130903126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 msgr2=0x7f078c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 0x7f078c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 -- 172.21.15.87:0/3130903126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c19fd60 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078affd640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 0x7f078c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f078b7fe640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f077c00c970 tx=0x7f077c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c007c10 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c19fff0 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f077c007db0 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c00f7d0 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.441+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c1a0500 con 0x7f078c101150 2024-04-22T13:29:45.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.442+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c1a07c0 con 0x7f078c101150 2024-04-22T13:29:45.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.443+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f077c00fa50 con 0x7f078c101150 2024-04-22T13:29:45.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.444+0000 7f0788ff9640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 0x7f07740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:45.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.444+0000 7f078affd640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 0x7f07740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:45.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.444+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f077c0d0640 con 0x7f078c101150 2024-04-22T13:29:45.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.444+0000 7f078affd640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 0x7f07740a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f078c199650 tx=0x7f0780005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:45.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.449+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f077c09d470 con 0x7f078c101150 2024-04-22T13:29:45.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:45.634+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f078c1a0990 con 0x7f078c101150 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T13:29:46.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/491862220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:29:46.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:29:46.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:29:46.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T13:29:46.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.309+0000 7f0788ff9640 1 -- 172.21.15.87:0/3130903126 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f077c09cd70 con 0x7f078c101150 2024-04-22T13:29:46.305 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cephfs_data' created 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 msgr2=0x7f07740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 0x7f07740a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f078c199650 tx=0x7f0780005d30 comp rx=0 tx=0).stop 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 msgr2=0x7f078c198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f077c00c970 tx=0x7f077c00ce30 comp rx=0 tx=0).stop 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 shutdown_connections 2024-04-22T13:29:46.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f07740a3bb0 0x7f07740a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078c102420 0x7f078c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f078c101b10 0x7f078c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.312+0000 7f079223e640 1 --2- 172.21.15.87:0/3130903126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f078c101150 0x7f078c198140 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.313+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 >> 172.21.15.87:0/3130903126 conn(0x7f078c0fbb80 msgr2=0x7f078c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:46.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.313+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 shutdown_connections 2024-04-22T13:29:46.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.313+0000 7f079223e640 1 -- 172.21.15.87:0/3130903126 wait complete. 2024-04-22T13:29:46.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- 172.21.15.87:0/2751377890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60100e60 msgr2=0x7f1a60101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/2751377890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60100e60 0x7f1a60101280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009900 tx=0x7f1a5402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- 172.21.15.87:0/2751377890 shutdown_connections 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/2751377890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60102a00 0x7f1a60109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/2751377890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 0x7f1a601024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/2751377890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60100e60 0x7f1a60101280 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- 172.21.15.87:0/2751377890 >> 172.21.15.87:0/2751377890 conn(0x7f1a600fc420 msgr2=0x7f1a600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:46.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- 172.21.15.87:0/2751377890 shutdown_connections 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- 172.21.15.87:0/2751377890 wait complete. 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 Processor -- start 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- start start 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 0x7f1a601981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 0x7f1a60198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010db30 con 0x7f1a60102a00 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010d9b0 con 0x7f1a60102040 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.439+0000 7f1a66fd5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6010dcb0 con 0x7f1a60100e60 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40102/0 (socket says 172.21.15.87:40102) 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 -- 172.21.15.87:0/3871596369 learned_addr learned my addr 172.21.15.87:0/3871596369 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:46.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a5ffff640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 0x7f1a60198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a64d71640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 0x7f1a601981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 -- 172.21.15.87:0/3871596369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 msgr2=0x7f1a601981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 0x7f1a601981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 -- 172.21.15.87:0/3871596369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 msgr2=0x7f1a60198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 0x7f1a60198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 -- 172.21.15.87:0/3871596369 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6019fe00 con 0x7f1a60102a00 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a64d71640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 0x7f1a601981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a65572640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800cad0 tx=0x7f1a5800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5800ad60 con 0x7f1a60102a00 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a58005020 con 0x7f1a60102a00 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5800f6b0 con 0x7f1a60102a00 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a601a00f0 con 0x7f1a60102a00 2024-04-22T13:29:46.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.440+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a601a0640 con 0x7f1a60102a00 2024-04-22T13:29:46.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.441+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a601082a0 con 0x7f1a60102a00 2024-04-22T13:29:46.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.442+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1a580051d0 con 0x7f1a60102a00 2024-04-22T13:29:46.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.443+0000 7f1a5dffb640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 0x7f1a500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:46.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.443+0000 7f1a64d71640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 0x7f1a500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:46.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.443+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f1a580d0290 con 0x7f1a60102a00 2024-04-22T13:29:46.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.443+0000 7f1a64d71640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 0x7f1a500a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a54006010 tx=0x7f1a540095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:46.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.448+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a58014030 con 0x7f1a60102a00 2024-04-22T13:29:46.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:46.664+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1a6002d5c0 con 0x7f1a60102a00 2024-04-22T13:29:47.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.323+0000 7f1a5dffb640 1 -- 172.21.15.87:0/3871596369 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f1a5809ce20 con 0x7f1a60102a00 2024-04-22T13:29:47.319 INFO:teuthology.orchestra.run.smithi087.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T13:29:47.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 msgr2=0x7f1a500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 0x7f1a500a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a54006010 tx=0x7f1a540095f0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 msgr2=0x7f1a6019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800cad0 tx=0x7f1a5800ced0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 shutdown_connections 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a500a3bb0 0x7f1a500a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a60102a00 0x7f1a6019f790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a60102040 0x7f1a60198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 --2- 172.21.15.87:0/3871596369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a60100e60 0x7f1a601981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.327+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 >> 172.21.15.87:0/3871596369 conn(0x7f1a600fc420 msgr2=0x7f1a60105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.328+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 shutdown_connections 2024-04-22T13:29:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.328+0000 7f1a66fd5640 1 -- 172.21.15.87:0/3871596369 wait complete. 2024-04-22T13:29:47.344 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 -- 172.21.15.87:0/3013442568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd20100630 msgr2=0x7efd20100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 --2- 172.21.15.87:0/3013442568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd20100630 0x7efd20100a50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009900 tx=0x7efd1c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 -- 172.21.15.87:0/3013442568 shutdown_connections 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 --2- 172.21.15.87:0/3013442568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd201021e0 0x7efd20108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 --2- 172.21.15.87:0/3013442568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 --2- 172.21.15.87:0/3013442568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd20100630 0x7efd20100a50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 -- 172.21.15.87:0/3013442568 >> 172.21.15.87:0/3013442568 conn(0x7efd200fbba0 msgr2=0x7efd200fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 -- 172.21.15.87:0/3013442568 shutdown_connections 2024-04-22T13:29:47.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.448+0000 7efd276c0640 1 -- 172.21.15.87:0/3013442568 wait complete. 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 Processor -- start 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 -- start start 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd20100630 0x7efd201981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010b8f0 con 0x7efd201021e0 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010b770 con 0x7efd20100630 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010ba70 con 0x7efd20101820 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40120/0 (socket says 172.21.15.87:40120) 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 -- 172.21.15.87:0/2144601040 learned_addr learned my addr 172.21.15.87:0/2144601040 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd2545c640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd20100630 0x7efd201981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd24c5b640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 -- 172.21.15.87:0/2144601040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 msgr2=0x7efd20198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 -- 172.21.15.87:0/2144601040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd20100630 msgr2=0x7efd201981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd20100630 0x7efd201981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 -- 172.21.15.87:0/2144601040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2019fe00 con 0x7efd201021e0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd24c5b640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd25c5d640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efd1800dd70 tx=0x7efd1800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18004d20 con 0x7efd201021e0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd1800be40 con 0x7efd201021e0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd201a00f0 con 0x7efd201021e0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.449+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18005610 con 0x7efd201021e0 2024-04-22T13:29:47.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.450+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd201a0640 con 0x7efd201021e0 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.451+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd201079d0 con 0x7efd201021e0 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.451+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efd180057b0 con 0x7efd201021e0 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.452+0000 7efd167fc640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 0x7efd0c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.452+0000 7efd2545c640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 0x7efd0c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.452+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7efd180d0960 con 0x7efd201021e0 2024-04-22T13:29:47.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.452+0000 7efd2545c640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 0x7efd0c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd1c005ab0 tx=0x7efd1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:47.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.457+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd1809d490 con 0x7efd201021e0 2024-04-22T13:29:47.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.636+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd20062b50 con 0x7efd201021e0 2024-04-22T13:29:47.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.637+0000 7efd167fc640 1 -- 172.21.15.87:0/2144601040 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22251 (secure 0 0 0) 0x7efd1809cd90 con 0x7efd201021e0 2024-04-22T13:29:47.634 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:47.634 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":84,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:29:46.666176+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 msgr2=0x7efd0c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 0x7efd0c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd1c005ab0 tx=0x7efd1c005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 msgr2=0x7efd2019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efd1800dd70 tx=0x7efd1800b730 comp rx=0 tx=0).stop 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 shutdown_connections 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7efd0c0a3c30 0x7efd0c0a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd201021e0 0x7efd2019f790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20101820 0x7efd20198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 --2- 172.21.15.87:0/2144601040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd20100630 0x7efd201981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 >> 172.21.15.87:0/2144601040 conn(0x7efd200fbba0 msgr2=0x7efd20104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:47.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 shutdown_connections 2024-04-22T13:29:47.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.640+0000 7efd276c0640 1 -- 172.21.15.87:0/2144601040 wait complete. 2024-04-22T13:29:47.645 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:47 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T13:29:47.645 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:47 smithi087 ceph-mon[27999]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T13:29:47.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:47 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T13:29:47.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:47 smithi087 ceph-mon[27999]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T13:29:47.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:47 smithi087 ceph-mon[27999]: pgmap v288: 97 pgs: 17 creating+peering, 4 active+clean, 76 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:47.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:29:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:47 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T13:29:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:47 smithi120 ceph-mon[31101]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T13:29:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:47 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T13:29:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:47 smithi120 ceph-mon[31101]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T13:29:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:47 smithi120 ceph-mon[31101]: pgmap v288: 97 pgs: 17 creating+peering, 4 active+clean, 76 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:47.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:47 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3130903126' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T13:29:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:47 smithi132 ceph-mon[31531]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T13:29:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:47 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T13:29:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:47 smithi132 ceph-mon[31531]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T13:29:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:47 smithi132 ceph-mon[31531]: pgmap v288: 97 pgs: 17 creating+peering, 4 active+clean, 76 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- 172.21.15.87:0/2555308099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 msgr2=0x7f981c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- 172.21.15.87:0/2555308099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1012b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f980c009900 tx=0x7f980c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- 172.21.15.87:0/2555308099 shutdown_connections 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- 172.21.15.87:0/2555308099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c1029a0 0x7f981c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- 172.21.15.87:0/2555308099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c101fe0 0x7f981c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- 172.21.15.87:0/2555308099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1012b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- 172.21.15.87:0/2555308099 >> 172.21.15.87:0/2555308099 conn(0x7f981c0fc400 msgr2=0x7f981c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- 172.21.15.87:0/2555308099 shutdown_connections 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- 172.21.15.87:0/2555308099 wait complete. 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 Processor -- start 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- start start 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 0x7f981c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c1029a0 0x7f981c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f981c10ca80 con 0x7f981c100e90 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f981c10c900 con 0x7f981c1029a0 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.749+0000 7f98242e4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f981c10cc00 con 0x7f981c101fe0 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40136/0 (socket says 172.21.15.87:40136) 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 -- 172.21.15.87:0/1973195877 learned_addr learned my addr 172.21.15.87:0/1973195877 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:47.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f982187f640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 0x7f981c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822881640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c1029a0 0x7f981c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 -- 172.21.15.87:0/1973195877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 msgr2=0x7f981c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 0x7f981c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 -- 172.21.15.87:0/1973195877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c1029a0 msgr2=0x7f981c19f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c1029a0 0x7f981c19f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 -- 172.21.15.87:0/1973195877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981c19fef0 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f982187f640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 0x7f981c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f9822080640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f980c02c840 tx=0x7f980c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c043020 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f980c004a80 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981c1a0180 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c003c40 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.750+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f981c1a0750 con 0x7f981c100e90 2024-04-22T13:29:47.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.751+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f981c1a0950 con 0x7f981c100e90 2024-04-22T13:29:47.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.753+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f980c046020 con 0x7f981c100e90 2024-04-22T13:29:47.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.753+0000 7f98137fe640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 0x7f98080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:47.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.753+0000 7f982187f640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 0x7f98080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:47.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.753+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f980c0f2e60 con 0x7f981c100e90 2024-04-22T13:29:47.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.754+0000 7f982187f640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 0x7f98080a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f981c1997e0 tx=0x7f9818008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:47.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.757+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f980c04d020 con 0x7f981c100e90 2024-04-22T13:29:47.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.938+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f981c1a0bc0 con 0x7f981c100e90 2024-04-22T13:29:47.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.939+0000 7f98137fe640 1 -- 172.21.15.87:0/1973195877 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22251 (secure 0 0 0) 0x7f980c03e450 con 0x7f981c100e90 2024-04-22T13:29:47.935 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:47.935 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":84,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:29:46.666176+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 msgr2=0x7f98080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 0x7f98080a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f981c1997e0 tx=0x7f9818008040 comp rx=0 tx=0).stop 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 msgr2=0x7f981c1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f980c02c840 tx=0x7f980c004270 comp rx=0 tx=0).stop 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 shutdown_connections 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f98080a3bb0 0x7f98080a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f981c1029a0 0x7f981c19f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f981c101fe0 0x7f981c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 --2- 172.21.15.87:0/1973195877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f981c100e90 0x7f981c1982d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:47.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 >> 172.21.15.87:0/1973195877 conn(0x7f981c0fc400 msgr2=0x7f981c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:47.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 shutdown_connections 2024-04-22T13:29:47.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:47.941+0000 7f98242e4640 1 -- 172.21.15.87:0/1973195877 wait complete. 2024-04-22T13:29:47.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T13:29:48.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 -- 172.21.15.87:0/2300459099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103300 msgr2=0x7fd680103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 --2- 172.21.15.87:0/2300459099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103300 0x7fd680103780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd674009930 tx=0x7fd67402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 -- 172.21.15.87:0/2300459099 shutdown_connections 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 --2- 172.21.15.87:0/2300459099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103cc0 0x7fd68010a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 --2- 172.21.15.87:0/2300459099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103300 0x7fd680103780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 --2- 172.21.15.87:0/2300459099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 -- 172.21.15.87:0/2300459099 >> 172.21.15.87:0/2300459099 conn(0x7fd6800fbb80 msgr2=0x7fd6800fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 -- 172.21.15.87:0/2300459099 shutdown_connections 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.053+0000 7fd685bca640 1 -- 172.21.15.87:0/2300459099 wait complete. 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 Processor -- start 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 -- start start 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 0x7fd680198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd68010db00 con 0x7fd680103cc0 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd68010d980 con 0x7fd680103300 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd685bca640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd68010dc80 con 0x7fd680100290 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd67ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd67ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40152/0 (socket says 172.21.15.87:40152) 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.054+0000 7fd67ffff640 1 -- 172.21.15.87:0/2933974582 learned_addr learned my addr 172.21.15.87:0/2933974582 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 -- 172.21.15.87:0/2933974582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 msgr2=0x7fd6801981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67f7fe640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67effd640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 0x7fd680198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 -- 172.21.15.87:0/2933974582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 msgr2=0x7fd680198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 0x7fd680198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 -- 172.21.15.87:0/2933974582 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68019fe00 con 0x7fd680103cc0 2024-04-22T13:29:48.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67f7fe640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67effd640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 0x7fd680198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67ffff640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c970 tx=0x7fd67800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678007bb0 con 0x7fd680103cc0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6801a00f0 con 0x7fd680103cc0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd678007d50 con 0x7fd680103cc0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678004e60 con 0x7fd680103cc0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.055+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801a05f0 con 0x7fd680103cc0 2024-04-22T13:29:48.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.056+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd680109550 con 0x7fd680103cc0 2024-04-22T13:29:48.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.057+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd678005000 con 0x7fd680103cc0 2024-04-22T13:29:48.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.058+0000 7fd67cff9640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 0x7fd6680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.058+0000 7fd67f7fe640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 0x7fd6680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.058+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fd6780d03c0 con 0x7fd680103cc0 2024-04-22T13:29:48.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.058+0000 7fd67f7fe640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 0x7fd6680a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd68006a290 tx=0x7fd670008370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:48.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.062+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd678014030 con 0x7fd680103cc0 2024-04-22T13:29:48.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.278+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd68002d5c0 con 0x7fd680103cc0 2024-04-22T13:29:48.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.332+0000 7fd67cff9640 1 -- 172.21.15.87:0/2933974582 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fd67809cef0 con 0x7fd680103cc0 2024-04-22T13:29:48.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 msgr2=0x7fd6680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 0x7fd6680a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd68006a290 tx=0x7fd670008370 comp rx=0 tx=0).stop 2024-04-22T13:29:48.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 msgr2=0x7fd68019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c970 tx=0x7fd67800ce30 comp rx=0 tx=0).stop 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 shutdown_connections 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6680a3bb0 0x7fd6680a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd680103cc0 0x7fd68019f790 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd680103300 0x7fd680198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 --2- 172.21.15.87:0/2933974582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd680100290 0x7fd6801981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 >> 172.21.15.87:0/2933974582 conn(0x7fd6800fbb80 msgr2=0x7fd680106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 shutdown_connections 2024-04-22T13:29:48.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.334+0000 7fd685bca640 1 -- 172.21.15.87:0/2933974582 wait complete. 2024-04-22T13:29:48.343 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T13:29:48.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.447+0000 7f6b84975640 1 -- 172.21.15.87:0/1900790208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 msgr2=0x7f6b801092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.447+0000 7f6b84975640 1 --2- 172.21.15.87:0/1900790208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b801092c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009ca0 tx=0x7f6b7802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- 172.21.15.87:0/1900790208 shutdown_connections 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- 172.21.15.87:0/1900790208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b801092c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- 172.21.15.87:0/1900790208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 0x7f6b80102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- 172.21.15.87:0/1900790208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 0x7f6b801012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- 172.21.15.87:0/1900790208 >> 172.21.15.87:0/1900790208 conn(0x7f6b800fc400 msgr2=0x7f6b800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- 172.21.15.87:0/1900790208 shutdown_connections 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- 172.21.15.87:0/1900790208 wait complete. 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 Processor -- start 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- start start 2024-04-22T13:29:48.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 0x7f6b8010b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 0x7f6b80109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010db70 con 0x7f6b801029a0 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010d9f0 con 0x7f6b80101fe0 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.448+0000 7f6b84975640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010dcf0 con 0x7f6b80100e90 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40166/0 (socket says 172.21.15.87:40166) 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 -- 172.21.15.87:0/4262649829 learned_addr learned my addr 172.21.15.87:0/4262649829 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7dd9b640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 0x7f6b80109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7e59c640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 0x7f6b8010b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 -- 172.21.15.87:0/4262649829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 msgr2=0x7f6b8010b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 0x7f6b8010b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 -- 172.21.15.87:0/4262649829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 msgr2=0x7f6b80109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:48.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 0x7f6b80109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:48.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 -- 172.21.15.87:0/4262649829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8010ac40 con 0x7f6b801029a0 2024-04-22T13:29:48.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b7ed9d640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b7802c700 tx=0x7f6b78034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78039040 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b801a22d0 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b78044070 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780042a0 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.449+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b801a2820 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.450+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b801081a0 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.451+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b78003970 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.452+0000 7f6b6f7fe640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 0x7f6b680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.452+0000 7f6b7e59c640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 0x7f6b680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.452+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6b780f26e0 con 0x7f6b801029a0 2024-04-22T13:29:48.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.452+0000 7f6b7e59c640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 0x7f6b680a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009890 tx=0x7f6b74008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:48.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.456+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b780bf210 con 0x7f6b801029a0 2024-04-22T13:29:48.614 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2144601040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1973195877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T13:29:48.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:48 smithi132 ceph-mon[31531]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T13:29:48.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:48.672+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f6b80062b50 con 0x7f6b801029a0 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2144601040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1973195877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T13:29:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:48 smithi120 ceph-mon[31101]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T13:29:48.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3871596369' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2144601040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1973195877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T13:29:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:48 smithi087 ceph-mon[27999]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T13:29:49.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.305+0000 7f6b6f7fe640 1 -- 172.21.15.87:0/4262649829 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f6b780beb10 con 0x7f6b801029a0 2024-04-22T13:29:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 msgr2=0x7f6b680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 0x7f6b680a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009890 tx=0x7f6b74008040 comp rx=0 tx=0).stop 2024-04-22T13:29:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 msgr2=0x7f6b8010a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b7802c700 tx=0x7f6b78034760 comp rx=0 tx=0).stop 2024-04-22T13:29:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 shutdown_connections 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6b680a3bb0 0x7f6b680a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b801029a0 0x7f6b8010a3b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b80101fe0 0x7f6b80109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 --2- 172.21.15.87:0/4262649829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b80100e90 0x7f6b8010b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 >> 172.21.15.87:0/4262649829 conn(0x7f6b800fc400 msgr2=0x7f6b80105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.307+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 shutdown_connections 2024-04-22T13:29:49.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.308+0000 7f6b84975640 1 -- 172.21.15.87:0/4262649829 wait complete. 2024-04-22T13:29:49.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 -- 172.21.15.87:0/246881472 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 msgr2=0x7fdce810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 --2- 172.21.15.87:0/246881472 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce810a3e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009ca0 tx=0x7fdce002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 -- 172.21.15.87:0/246881472 shutdown_connections 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 --2- 172.21.15.87:0/246881472 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce810a3e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 --2- 172.21.15.87:0/246881472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 0x7fdce8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 --2- 172.21.15.87:0/246881472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 -- 172.21.15.87:0/246881472 >> 172.21.15.87:0/246881472 conn(0x7fdce80fd530 msgr2=0x7fdce80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 -- 172.21.15.87:0/246881472 shutdown_connections 2024-04-22T13:29:49.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.429+0000 7fdceea2e640 1 -- 172.21.15.87:0/246881472 wait complete. 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 Processor -- start 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 -- start start 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 0x7fdce8198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdce810dae0 con 0x7fdce8103b50 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdce810d960 con 0x7fdce8101fa0 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdceea2e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdce810dc60 con 0x7fdce8103190 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdcecfcb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdcecfcb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40170/0 (socket says 172.21.15.87:40170) 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.430+0000 7fdcecfcb640 1 -- 172.21.15.87:0/3736955135 learned_addr learned my addr 172.21.15.87:0/3736955135 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce77fe640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 0x7fdce8198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce7fff640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:49.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 -- 172.21.15.87:0/3736955135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 msgr2=0x7fdce8198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 0x7fdce8198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 -- 172.21.15.87:0/3736955135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 msgr2=0x7fdce8198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce8198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 -- 172.21.15.87:0/3736955135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce819fd20 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce7fff640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce8198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdcecfcb640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdce002c700 tx=0x7fdce00093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce0041020 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdce0034d60 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce819ffb0 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce0039040 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.431+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce81a0410 con 0x7fdce8103b50 2024-04-22T13:29:49.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.432+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce8109260 con 0x7fdce8103b50 2024-04-22T13:29:49.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.433+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdce0044020 con 0x7fdce8103b50 2024-04-22T13:29:49.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.434+0000 7fdce57fa640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 0x7fdcd00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:49.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.434+0000 7fdce7fff640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 0x7fdcd00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:49.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.434+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fdce00f2210 con 0x7fdce8103b50 2024-04-22T13:29:49.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.434+0000 7fdce7fff640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 0x7fdcd00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdce8102ff0 tx=0x7fdcd8005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:49.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.438+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdce004b020 con 0x7fdce8103b50 2024-04-22T13:29:49.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:49.656+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fdce802d5c0 con 0x7fdce8103b50 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:active 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: pgmap v291: 97 pgs: 23 creating+peering, 18 active+clean, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T13:29:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:49 smithi120 ceph-mon[31101]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:active 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: pgmap v291: 97 pgs: 23 creating+peering, 18 active+clean, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T13:29:49.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:49 smithi132 ceph-mon[31531]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T13:29:49.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2933974582' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] up:active 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: pgmap v291: 97 pgs: 23 creating+peering, 18 active+clean, 56 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4262649829' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T13:29:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:49 smithi087 ceph-mon[27999]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T13:29:50.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.312+0000 7fdce57fa640 1 -- 172.21.15.87:0/3736955135 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7fdce00ba030 con 0x7fdce8103b50 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 msgr2=0x7fdcd00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 0x7fdcd00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdce8102ff0 tx=0x7fdcd8005db0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 msgr2=0x7fdce819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdce002c700 tx=0x7fdce00093f0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 shutdown_connections 2024-04-22T13:29:50.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdcd00a3bb0 0x7fdcd00a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8103b50 0x7fdce819f6b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8103190 0x7fdce8198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 --2- 172.21.15.87:0/3736955135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101fa0 0x7fdce8198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 >> 172.21.15.87:0/3736955135 conn(0x7fdce80fd530 msgr2=0x7fdce8106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 shutdown_connections 2024-04-22T13:29:50.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.314+0000 7fdceea2e640 1 -- 172.21.15.87:0/3736955135 wait complete. 2024-04-22T13:29:50.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- 172.21.15.87:0/3299281168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 msgr2=0x7fb23410c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 --2- 172.21.15.87:0/3299281168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb23410c2e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009ca0 tx=0x7fb22c02c180 comp rx=0 tx=0).stop 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- 172.21.15.87:0/3299281168 shutdown_connections 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 --2- 172.21.15.87:0/3299281168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb23410c2e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 --2- 172.21.15.87:0/3299281168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234074930 0x7fb234074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 --2- 172.21.15.87:0/3299281168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234075f10 0x7fb234074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- 172.21.15.87:0/3299281168 >> 172.21.15.87:0/3299281168 conn(0x7fb2340fd530 msgr2=0x7fb2340ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- 172.21.15.87:0/3299281168 shutdown_connections 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- 172.21.15.87:0/3299281168 wait complete. 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 Processor -- start 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.438+0000 7fb23c670640 1 -- start start 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 0x7fb23419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234075f10 0x7fb23419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb23410fcf0 con 0x7fb234109e50 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb23410fb70 con 0x7fb234075f10 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb23410fe70 con 0x7fb234074930 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33912/0 (socket says 172.21.15.87:33912) 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 -- 172.21.15.87:0/3633176375 learned_addr learned my addr 172.21.15.87:0/3633176375 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb239c0b640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234075f10 0x7fb23419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23a40c640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 0x7fb23419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 -- 172.21.15.87:0/3633176375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 msgr2=0x7fb23419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 0x7fb23419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 -- 172.21.15.87:0/3633176375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234075f10 msgr2=0x7fb23419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234075f10 0x7fb23419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 -- 172.21.15.87:0/3633176375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2341a1f70 con 0x7fb234109e50 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23a40c640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 0x7fb23419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23ac0d640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb22c005e30 tx=0x7fb22c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c039040 con 0x7fb234109e50 2024-04-22T13:29:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2341a2200 con 0x7fb234109e50 2024-04-22T13:29:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb22c044070 con 0x7fb234109e50 2024-04-22T13:29:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0328a0 con 0x7fb234109e50 2024-04-22T13:29:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.439+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2341a26f0 con 0x7fb234109e50 2024-04-22T13:29:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.440+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb234108c10 con 0x7fb234109e50 2024-04-22T13:29:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.441+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb22c032b00 con 0x7fb234109e50 2024-04-22T13:29:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.442+0000 7fb22b7fe640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 0x7fb2200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.442+0000 7fb23a40c640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 0x7fb2200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.442+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb22c0f3300 con 0x7fb234109e50 2024-04-22T13:29:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.443+0000 7fb23a40c640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 0x7fb2200a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb2240058c0 tx=0x7fb224005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:50.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.446+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb22c0bfe30 con 0x7fb234109e50 2024-04-22T13:29:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:50.660+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb23402d5c0 con 0x7fb234109e50 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:active 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: Cluster is now healthy 2024-04-22T13:29:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T13:29:50.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T13:29:50.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:50 smithi120 ceph-mon[31101]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:active 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: Cluster is now healthy 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T13:29:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:50 smithi132 ceph-mon[31531]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T13:29:50.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3736955135' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] up:active 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: Cluster is now healthy 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T13:29:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:50 smithi087 ceph-mon[27999]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T13:29:51.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.318+0000 7fb22b7fe640 1 -- 172.21.15.87:0/3633176375 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7fb22c0bf730 con 0x7fb234109e50 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 msgr2=0x7fb2200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 0x7fb2200a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb2240058c0 tx=0x7fb224005ec0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 msgr2=0x7fb2341a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb22c005e30 tx=0x7fb22c003c20 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 shutdown_connections 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb2200a3bb0 0x7fb2200a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb234109e50 0x7fb2341a1900 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb234075f10 0x7fb23419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 --2- 172.21.15.87:0/3633176375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234074930 0x7fb23419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 >> 172.21.15.87:0/3633176375 conn(0x7fb2340fd530 msgr2=0x7fb234105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 shutdown_connections 2024-04-22T13:29:51.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.320+0000 7fb23c670640 1 -- 172.21.15.87:0/3633176375 wait complete. 2024-04-22T13:29:51.328 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T13:29:51.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T13:29:51.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- 172.21.15.87:0/2993608914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c101810 msgr2=0x7f498c101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 --2- 172.21.15.87:0/2993608914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c101810 0x7f498c101c90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f497402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- 172.21.15.87:0/2993608914 shutdown_connections 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 --2- 172.21.15.87:0/2993608914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 0x7f498c108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 --2- 172.21.15.87:0/2993608914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c101810 0x7f498c101c90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 --2- 172.21.15.87:0/2993608914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c100620 0x7f498c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- 172.21.15.87:0/2993608914 >> 172.21.15.87:0/2993608914 conn(0x7f498c0fbb90 msgr2=0x7f498c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- 172.21.15.87:0/2993608914 shutdown_connections 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- 172.21.15.87:0/2993608914 wait complete. 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 Processor -- start 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.435+0000 7f49919b2640 1 -- start start 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c101810 0x7f498c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 0x7f498c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f498c10c9e0 con 0x7f498c100620 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f498c10c860 con 0x7f498c101810 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f498c10cb60 con 0x7f498c1021d0 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33928/0 (socket says 172.21.15.87:33928) 2024-04-22T13:29:51.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 -- 172.21.15.87:0/1550649734 learned_addr learned my addr 172.21.15.87:0/1550649734 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498a7fc640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c101810 0x7f498c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498b7fe640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 0x7f498c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 -- 172.21.15.87:0/1550649734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 msgr2=0x7f498c19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 0x7f498c19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 -- 172.21.15.87:0/1550649734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c101810 msgr2=0x7f498c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c101810 0x7f498c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 -- 172.21.15.87:0/1550649734 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c19fe10 con 0x7f498c100620 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f498affd640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f498400e9c0 tx=0x7f498400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498400cd80 con 0x7f498c100620 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49840035f0 con 0x7f498c100620 2024-04-22T13:29:51.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498c1a0100 con 0x7f498c100620 2024-04-22T13:29:51.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984010690 con 0x7f498c100620 2024-04-22T13:29:51.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.436+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498c1a0620 con 0x7f498c100620 2024-04-22T13:29:51.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.437+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f498c1a08e0 con 0x7f498c100620 2024-04-22T13:29:51.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.438+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f49840037a0 con 0x7f498c100620 2024-04-22T13:29:51.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.439+0000 7f49909b0640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 0x7f49780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.439+0000 7f498a7fc640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 0x7f49780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.439+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f49840d06c0 con 0x7f498c100620 2024-04-22T13:29:51.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.440+0000 7f498a7fc640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 0x7f49780a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4974009230 tx=0x7f4974005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:51.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.444+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f498409d1f0 con 0x7f498c100620 2024-04-22T13:29:51.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.645+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f498c109b30 con 0x7f49780a3bb0 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: pgmap v292: 97 pgs: 23 creating+peering, 53 active+clean, 21 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 564 B/s wr, 5 op/s 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:active 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T13:29:51.647 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:29:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.665+0000 7f49909b0640 1 -- 172.21.15.87:0/1550649734 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f498c109b30 con 0x7f49780a3bb0 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.667+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 msgr2=0x7f49780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.667+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 0x7f49780a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4974009230 tx=0x7f4974005fd0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.667+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 msgr2=0x7f498c198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.667+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f498400e9c0 tx=0x7f498400ee80 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 shutdown_connections 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f49780a3bb0 0x7f49780a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f498c1021d0 0x7f498c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f498c101810 0x7f498c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 --2- 172.21.15.87:0/1550649734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f498c100620 0x7f498c198240 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 >> 172.21.15.87:0/1550649734 conn(0x7f498c0fbb90 msgr2=0x7f498c109420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:51.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 shutdown_connections 2024-04-22T13:29:51.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.668+0000 7f49919b2640 1 -- 172.21.15.87:0/1550649734 wait complete. 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: pgmap v292: 97 pgs: 23 creating+peering, 53 active+clean, 21 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 564 B/s wr, 5 op/s 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:active 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T13:29:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:29:51.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: pgmap v292: 97 pgs: 23 creating+peering, 53 active+clean, 21 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 564 B/s wr, 5 op/s 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3633176375' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] up:active 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T13:29:51.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:51 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:29:51.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3825397605 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac103b40 msgr2=0x7ff0ac10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3825397605 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac103b40 0x7ff0ac10a3d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009ca0 tx=0x7ff0a402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3825397605 shutdown_connections 2024-04-22T13:29:51.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3825397605 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac103b40 0x7ff0ac10a3d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3825397605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 0x7ff0ac103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3825397605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac101f90 0x7ff0ac1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3825397605 >> 172.21.15.87:0/3825397605 conn(0x7ff0ac0fd500 msgr2=0x7ff0ac0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3825397605 shutdown_connections 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.782+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3825397605 wait complete. 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 Processor -- start 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 -- start start 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 0x7ff0ac198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac103b40 0x7ff0ac19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac10dad0 con 0x7ff0ac101f90 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac10d950 con 0x7ff0ac103b40 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0b25c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac10dc50 con 0x7ff0ac103180 2024-04-22T13:29:51.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33954/0 (socket says 172.21.15.87:33954) 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0abfff640 1 -- 172.21.15.87:0/3828683292 learned_addr learned my addr 172.21.15.87:0/3828683292 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.783+0000 7ff0ab7fe640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 0x7ff0ac198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0b0b64640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac103b40 0x7ff0ac19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 -- 172.21.15.87:0/3828683292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 msgr2=0x7ff0ac198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 0x7ff0ac198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 -- 172.21.15.87:0/3828683292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac103b40 msgr2=0x7ff0ac19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac103b40 0x7ff0ac19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 -- 172.21.15.87:0/3828683292 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac19fd70 con 0x7ff0ac101f90 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0abfff640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e9c0 tx=0x7ff09c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00cd80 con 0x7ff0ac101f90 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff09c0035f0 con 0x7ff0ac101f90 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac1a0060 con 0x7ff0ac101f90 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c010690 con 0x7ff0ac101f90 2024-04-22T13:29:51.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.784+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac1a05b0 con 0x7ff0ac101f90 2024-04-22T13:29:51.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.785+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ac109380 con 0x7ff0ac101f90 2024-04-22T13:29:51.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.786+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff09c0037a0 con 0x7ff0ac101f90 2024-04-22T13:29:51.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.787+0000 7ff0a97fa640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 0x7ff0940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:51.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.787+0000 7ff0ab7fe640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 0x7ff0940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:51.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.787+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff09c0d06c0 con 0x7ff0ac101f90 2024-04-22T13:29:51.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.787+0000 7ff0ab7fe640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 0x7ff0940a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac199660 tx=0x7ff0a000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:51.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.791+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff09c09d1f0 con 0x7ff0ac101f90 2024-04-22T13:29:51.986 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:29:51.650+0000 7fa3975bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:29:51.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:29:51.650+0000 7fa3975bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:29:51.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:29:51.650+0000 7fa3975bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:29:51.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:29:51.650+0000 7fa3975bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:29:51.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:29:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:29:51.650+0000 7fa3975bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:29:51.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:51.994+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff0ac062390 con 0x7ff0940a3bb0 2024-04-22T13:29:52.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.018+0000 7ff0a97fa640 1 -- 172.21.15.87:0/3828683292 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff0ac062390 con 0x7ff0940a3bb0 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 msgr2=0x7ff0940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 0x7ff0940a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac199660 tx=0x7ff0a000a2d0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 msgr2=0x7ff0ac198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e9c0 tx=0x7ff09c00ee80 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 shutdown_connections 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff0940a3bb0 0x7ff0940a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0ac103b40 0x7ff0ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0ac103180 0x7ff0ac198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 --2- 172.21.15.87:0/3828683292 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0ac101f90 0x7ff0ac198150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 >> 172.21.15.87:0/3828683292 conn(0x7ff0ac0fd500 msgr2=0x7ff0ac106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 shutdown_connections 2024-04-22T13:29:52.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.020+0000 7ff0b25c7640 1 -- 172.21.15.87:0/3828683292 wait complete. 2024-04-22T13:29:52.027 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.133+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1073313681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8102980 msgr2=0x7f0ca81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.133+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1073313681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8102980 0x7f0ca81092a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009ca0 tx=0x7f0ca002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1073313681 shutdown_connections 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1073313681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8102980 0x7f0ca81092a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1073313681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8101fc0 0x7f0ca8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1073313681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8100e70 0x7f0ca8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1073313681 >> 172.21.15.87:0/1073313681 conn(0x7f0ca80fc3e0 msgr2=0x7f0ca80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1073313681 shutdown_connections 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1073313681 wait complete. 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 Processor -- start 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- start start 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8100e70 0x7f0ca8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8102980 0x7f0ca819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca810c9a0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca810c820 con 0x7f0ca8100e70 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cb00d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca810cb20 con 0x7f0ca8102980 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cad66d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cad66d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33966/0 (socket says 172.21.15.87:33966) 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.134+0000 7f0cad66d640 1 -- 172.21.15.87:0/1668771264 learned_addr learned my addr 172.21.15.87:0/1668771264 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cade6e640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8100e70 0x7f0ca8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cae66f640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8102980 0x7f0ca819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 -- 172.21.15.87:0/1668771264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8102980 msgr2=0x7f0ca819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8102980 0x7f0ca819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 -- 172.21.15.87:0/1668771264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8100e70 msgr2=0x7f0ca8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8100e70 0x7f0ca8198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 -- 172.21.15.87:0/1668771264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca819fd90 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cad66d640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9c0 tx=0x7f0ca400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400cd80 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca40035f0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca81a0080 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4010690 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.135+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca81a05d0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.136+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca81a0890 con 0x7f0ca8101fc0 2024-04-22T13:29:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.137+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0ca40037a0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.138+0000 7f0c9effd640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 0x7f0c940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.138+0000 7f0cade6e640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 0x7f0c940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.138+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0ca40d06c0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.138+0000 7f0cade6e640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 0x7f0c940a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c980058c0 tx=0x7f0c9800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.142+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ca409d1f0 con 0x7f0ca8101fc0 2024-04-22T13:29:52.169 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: 2024-04-22T13:29:51.815+0000 7f31cb107700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T13:29:52.170 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: 2024-04-22T13:29:51.815+0000 7f31cb107700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T13:29:52.170 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:29:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: 2024-04-22T13:29:51.815+0000 7f31cb107700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T13:29:52.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.348+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0ca80fe5b0 con 0x7f0c940a3bb0 2024-04-22T13:29:52.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.354+0000 7f0c9effd640 1 -- 172.21.15.87:0/1668771264 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0ca80fe5b0 con 0x7f0c940a3bb0 2024-04-22T13:29:52.349 INFO:teuthology.orchestra.run.smithi087.stdout:/volumes/qa/sv_0/ad720548-d50e-4a93-8d97-659706cbb04f 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 msgr2=0x7f0c940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 0x7f0c940a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0c980058c0 tx=0x7f0c9800a2d0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 msgr2=0x7f0ca81986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400e9c0 tx=0x7f0ca400ee80 comp rx=0 tx=0).stop 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 shutdown_connections 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0c940a3bb0 0x7f0c940a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ca8102980 0x7f0ca819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ca8101fc0 0x7f0ca81986b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 --2- 172.21.15.87:0/1668771264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ca8100e70 0x7f0ca8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 >> 172.21.15.87:0/1668771264 conn(0x7f0ca80fc3e0 msgr2=0x7f0ca80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 shutdown_connections 2024-04-22T13:29:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.356+0000 7f0cb00d2640 1 -- 172.21.15.87:0/1668771264 wait complete. 2024-04-22T13:29:52.363 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- 172.21.15.87:0/3971473725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd81029a0 msgr2=0x7fedd81092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 --2- 172.21.15.87:0/3971473725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd81029a0 0x7fedd81092c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009ca0 tx=0x7fedd002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- 172.21.15.87:0/3971473725 shutdown_connections 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 --2- 172.21.15.87:0/3971473725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd81029a0 0x7fedd81092c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 --2- 172.21.15.87:0/3971473725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101fe0 0x7fedd8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 --2- 172.21.15.87:0/3971473725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8100e90 0x7fedd81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- 172.21.15.87:0/3971473725 >> 172.21.15.87:0/3971473725 conn(0x7fedd80fc400 msgr2=0x7fedd80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- 172.21.15.87:0/3971473725 shutdown_connections 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- 172.21.15.87:0/3971473725 wait complete. 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 Processor -- start 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.471+0000 7fede072b640 1 -- start start 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8101fe0 0x7fedd819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd81029a0 0x7fedd819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fedd810ca40 con 0x7fedd8100e90 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedd810c8c0 con 0x7fedd81029a0 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fedd810cbc0 con 0x7fedd8101fe0 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33990/0 (socket says 172.21.15.87:33990) 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 -- 172.21.15.87:0/1026601635 learned_addr learned my addr 172.21.15.87:0/1026601635 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7feddecc8640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd81029a0 0x7fedd819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedddcc6640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8101fe0 0x7fedd819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 -- 172.21.15.87:0/1026601635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8101fe0 msgr2=0x7fedd819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8101fe0 0x7fedd819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 -- 172.21.15.87:0/1026601635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd81029a0 msgr2=0x7fedd819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd81029a0 0x7fedd819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 -- 172.21.15.87:0/1026601635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd819fd80 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedde4c7640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fedd400b4a0 tx=0x7fedd400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4014070 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd4010ac0 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd81a0070 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4017c30 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.472+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81a0570 con 0x7fedd8100e90 2024-04-22T13:29:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.473+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd81081a0 con 0x7fedd8100e90 2024-04-22T13:29:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.474+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fedd4017070 con 0x7fedd8100e90 2024-04-22T13:29:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.475+0000 7fedcf7fe640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 0x7fedc40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.475+0000 7fedddcc6640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 0x7fedc40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.475+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fedd40cf650 con 0x7fedd8100e90 2024-04-22T13:29:52.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.475+0000 7fedddcc6640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 0x7fedc40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fedc80058c0 tx=0x7fedc8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.479+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedd409c180 con 0x7fedd8100e90 2024-04-22T13:29:52.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:52 smithi120 ceph-mon[31101]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:52.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.684+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fedd8062390 con 0x7fedc40a3bb0 2024-04-22T13:29:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.713+0000 7fedcf7fe640 1 -- 172.21.15.87:0/1026601635 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fedd8062390 con 0x7fedc40a3bb0 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 msgr2=0x7fedc40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 0x7fedc40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fedc80058c0 tx=0x7fedc8005850 comp rx=0 tx=0).stop 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 msgr2=0x7fedd806b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fedd400b4a0 tx=0x7fedd400b960 comp rx=0 tx=0).stop 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 shutdown_connections 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fedc40a3bb0 0x7fedc40a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd81029a0 0x7fedd819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedd8101fe0 0x7fedd819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 --2- 172.21.15.87:0/1026601635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd8100e90 0x7fedd806b160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 >> 172.21.15.87:0/1026601635 conn(0x7fedd80fc400 msgr2=0x7fedd8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 shutdown_connections 2024-04-22T13:29:52.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.715+0000 7fede072b640 1 -- 172.21.15.87:0/1026601635 wait complete. 2024-04-22T13:29:52.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:52 smithi132 ceph-mon[31531]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:52.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:52 smithi087 ceph-mon[27999]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:52.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T13:29:52.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- 172.21.15.87:0/4165943711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 msgr2=0x7f2da810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/4165943711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da810a0d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009ca0 tx=0x7f2da002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- 172.21.15.87:0/4165943711 shutdown_connections 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/4165943711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da810a0d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/4165943711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 0x7f2da8107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/4165943711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 0x7f2da8104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- 172.21.15.87:0/4165943711 >> 172.21.15.87:0/4165943711 conn(0x7f2da80fc430 msgr2=0x7f2da80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- 172.21.15.87:0/4165943711 shutdown_connections 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- 172.21.15.87:0/4165943711 wait complete. 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 Processor -- start 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.829+0000 7f2dae3ef640 1 -- start start 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 0x7f2da8198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 0x7f2da8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810dae0 con 0x7f2da8107c80 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810d960 con 0x7f2da8105350 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810dc60 con 0x7f2da8102a20 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34000/0 (socket says 172.21.15.87:34000) 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 -- 172.21.15.87:0/1567043346 learned_addr learned my addr 172.21.15.87:0/1567043346 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2da77fe640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 0x7f2da8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2da7fff640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 0x7f2da8198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 -- 172.21.15.87:0/1567043346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 msgr2=0x7f2da8198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 0x7f2da8198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 -- 172.21.15.87:0/1567043346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 msgr2=0x7f2da8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 0x7f2da8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 -- 172.21.15.87:0/1567043346 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da819fd70 con 0x7f2da8107c80 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dac98c640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2da002c700 tx=0x7f2da0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0039040 con 0x7f2da8107c80 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2da0044070 con 0x7f2da8107c80 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da81a0000 con 0x7f2da8107c80 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da00042a0 con 0x7f2da8107c80 2024-04-22T13:29:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.830+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da81a04f0 con 0x7f2da8107c80 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.831+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da80fdf20 con 0x7f2da8107c80 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.832+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2da0003970 con 0x7f2da8107c80 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.833+0000 7f2da57fa640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 0x7f2d900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.833+0000 7f2da7fff640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 0x7f2d900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.833+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2da00f2690 con 0x7f2da8107c80 2024-04-22T13:29:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.833+0000 7f2da7fff640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 0x7f2d900a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d98005fd0 tx=0x7f2d98005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:52.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:52.837+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2da00bf1c0 con 0x7f2da8107c80 2024-04-22T13:29:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.042+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f2da8062420 con 0x7f2d900a3bb0 2024-04-22T13:29:53.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.047+0000 7f2da57fa640 1 -- 172.21.15.87:0/1567043346 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2da8062420 con 0x7f2d900a3bb0 2024-04-22T13:29:53.042 INFO:teuthology.orchestra.run.smithi087.stdout:/volumes/qa/sv_1/8f72c7cd-96ba-4153-be89-fe989bd2e42e 2024-04-22T13:29:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 msgr2=0x7f2d900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 0x7f2d900a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d98005fd0 tx=0x7f2d98005850 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 msgr2=0x7f2da819f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2da002c700 tx=0x7f2da0034760 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 shutdown_connections 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2d900a3bb0 0x7f2d900a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da8107c80 0x7f2da819f700 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2da8105350 0x7f2da8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 --2- 172.21.15.87:0/1567043346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da8102a20 0x7f2da8198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 >> 172.21.15.87:0/1567043346 conn(0x7f2da80fc430 msgr2=0x7f2da8103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 shutdown_connections 2024-04-22T13:29:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.049+0000 7f2dae3ef640 1 -- 172.21.15.87:0/1567043346 wait complete. 2024-04-22T13:29:53.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:29:53.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 -- 172.21.15.87:0/3717941629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101ad0 msgr2=0x7f6940109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 --2- 172.21.15.87:0/3717941629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101ad0 0x7f6940109370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6938009cd0 tx=0x7f693802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 -- 172.21.15.87:0/3717941629 shutdown_connections 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 --2- 172.21.15.87:0/3717941629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101ad0 0x7f6940109370 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 --2- 172.21.15.87:0/3717941629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6940101110 0x7f6940101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 --2- 172.21.15.87:0/3717941629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102470 0x7f6940102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 -- 172.21.15.87:0/3717941629 >> 172.21.15.87:0/3717941629 conn(0x7f69400fbb80 msgr2=0x7f69400fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 -- 172.21.15.87:0/3717941629 shutdown_connections 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.158+0000 7f6948310640 1 -- 172.21.15.87:0/3717941629 wait complete. 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 Processor -- start 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- start start 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101ad0 0x7f694019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6940102470 0x7f69401a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f694010c9e0 con 0x7f6940101110 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f694010c860 con 0x7f6940101ad0 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f694010cb60 con 0x7f6940102470 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34012/0 (socket says 172.21.15.87:34012) 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 -- 172.21.15.87:0/892511023 learned_addr learned my addr 172.21.15.87:0/892511023 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69458ab640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101ad0 0x7f694019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 -- 172.21.15.87:0/892511023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6940102470 msgr2=0x7f69401a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6940102470 0x7f69401a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 -- 172.21.15.87:0/892511023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101ad0 msgr2=0x7f694019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101ad0 0x7f694019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 -- 172.21.15.87:0/892511023 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69401a1fe0 con 0x7f6940101110 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69460ac640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f693c00d770 tx=0x7f693c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c017070 con 0x7f6940101110 2024-04-22T13:29:53.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c009c30 con 0x7f6940101110 2024-04-22T13:29:53.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c003cc0 con 0x7f6940101110 2024-04-22T13:29:53.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69401a22d0 con 0x7f6940101110 2024-04-22T13:29:53.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.159+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69401a2820 con 0x7f6940101110 2024-04-22T13:29:53.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.161+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69401a2a80 con 0x7f6940101110 2024-04-22T13:29:53.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.162+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f693c003e60 con 0x7f6940101110 2024-04-22T13:29:53.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.162+0000 7f69377fe640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 0x7f692c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.162+0000 7f69458ab640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 0x7f692c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.162+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f693c0d0130 con 0x7f6940101110 2024-04-22T13:29:53.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.163+0000 7f69458ab640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 0x7f692c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f694019b8d0 tx=0x7f6930005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.166+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f693c009de0 con 0x7f6940101110 2024-04-22T13:29:53.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.365+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f693c098bd0 con 0x7f6940101110 2024-04-22T13:29:53.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.385+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69401a2c50 con 0x7f6940101110 2024-04-22T13:29:53.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.385+0000 7f69377fe640 1 -- 172.21.15.87:0/892511023 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7f693c09cc60 con 0x7f6940101110 2024-04-22T13:29:53.381 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:53.381 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:29:53.382 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:29:53.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 msgr2=0x7f692c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 0x7f692c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f694019b8d0 tx=0x7f6930005c60 comp rx=0 tx=0).stop 2024-04-22T13:29:53.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 msgr2=0x7f694019a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f693c00d770 tx=0x7f693c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 shutdown_connections 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f692c0a3bb0 0x7f692c0a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6940102470 0x7f69401a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.387+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101ad0 0x7f694019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.388+0000 7f6948310640 1 --2- 172.21.15.87:0/892511023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6940101110 0x7f694019a430 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.388+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 >> 172.21.15.87:0/892511023 conn(0x7f69400fbb80 msgr2=0x7f69400fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.388+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 shutdown_connections 2024-04-22T13:29:53.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.388+0000 7f6948310640 1 -- 172.21.15.87:0/892511023 wait complete. 2024-04-22T13:29:53.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:29:53.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 -- 172.21.15.87:0/2739801704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 msgr2=0x7fcc181012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 --2- 172.21.15.87:0/2739801704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc181012a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009900 tx=0x7fcc1402bc30 comp rx=0 tx=0).stop 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 -- 172.21.15.87:0/2739801704 shutdown_connections 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 --2- 172.21.15.87:0/2739801704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc18102990 0x7fcc181092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 --2- 172.21.15.87:0/2739801704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18101fd0 0x7fcc18102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 --2- 172.21.15.87:0/2739801704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc181012a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 -- 172.21.15.87:0/2739801704 >> 172.21.15.87:0/2739801704 conn(0x7fcc180fc410 msgr2=0x7fcc180fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 -- 172.21.15.87:0/2739801704 shutdown_connections 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.497+0000 7fcc2019f640 1 -- 172.21.15.87:0/2739801704 wait complete. 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 Processor -- start 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 -- start start 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc18101fd0 0x7fcc1819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18102990 0x7fcc181a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1810c9b0 con 0x7fcc18100e80 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1810c830 con 0x7fcc18101fd0 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc2019f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1810cb30 con 0x7fcc18102990 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34034/0 (socket says 172.21.15.87:34034) 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 -- 172.21.15.87:0/3748148886 learned_addr learned my addr 172.21.15.87:0/3748148886 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1e73c640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18102990 0x7fcc181a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 -- 172.21.15.87:0/3748148886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18102990 msgr2=0x7fcc181a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18102990 0x7fcc181a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 -- 172.21.15.87:0/3748148886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc18101fd0 msgr2=0x7fcc1819a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc18101fd0 0x7fcc1819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.498+0000 7fcc1df3b640 1 -- 172.21.15.87:0/3748148886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc181a1f90 con 0x7fcc18100e80 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc1df3b640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036040 tx=0x7fcc140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14043020 con 0x7fcc18100e80 2024-04-22T13:29:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc140054c0 con 0x7fcc18100e80 2024-04-22T13:29:53.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc181a2220 con 0x7fcc18100e80 2024-04-22T13:29:53.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14003db0 con 0x7fcc18100e80 2024-04-22T13:29:53.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.499+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc181a2770 con 0x7fcc18100e80 2024-04-22T13:29:53.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.500+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc181a2970 con 0x7fcc18100e80 2024-04-22T13:29:53.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.501+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc14046020 con 0x7fcc18100e80 2024-04-22T13:29:53.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.501+0000 7fcc0effd640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 0x7fcc040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.502+0000 7fcc1d73a640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 0x7fcc040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.502+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcc140f2870 con 0x7fcc18100e80 2024-04-22T13:29:53.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.502+0000 7fcc1d73a640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 0x7fcc040a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc1819b880 tx=0x7fcc08005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.506+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc1404d020 con 0x7fcc18100e80 2024-04-22T13:29:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:53 smithi120 ceph-mon[31101]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:53 smithi120 ceph-mon[31101]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:53 smithi120 ceph-mon[31101]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:53 smithi120 ceph-mon[31101]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 19 op/s 2024-04-22T13:29:53.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.686+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc181a2c60 con 0x7fcc18100e80 2024-04-22T13:29:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.688+0000 7fcc0effd640 1 -- 172.21.15.87:0/3748148886 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22238 (secure 0 0 0) 0x7fcc140bf3b0 con 0x7fcc18100e80 2024-04-22T13:29:53.684 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:53.684 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":85,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:29:48.311419+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 msgr2=0x7fcc040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 0x7fcc040a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcc1819b880 tx=0x7fcc08005c90 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 msgr2=0x7fcc1819a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036040 tx=0x7fcc140042f0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 shutdown_connections 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcc040a3c40 0x7fcc040a6100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc18102990 0x7fcc181a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc18101fd0 0x7fcc1819a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 --2- 172.21.15.87:0/3748148886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc18100e80 0x7fcc1819a370 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 >> 172.21.15.87:0/3748148886 conn(0x7fcc180fc410 msgr2=0x7fcc180fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 shutdown_connections 2024-04-22T13:29:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.690+0000 7fcc2019f640 1 -- 172.21.15.87:0/3748148886 wait complete. 2024-04-22T13:29:53.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:53 smithi087 ceph-mon[27999]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:53 smithi087 ceph-mon[27999]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:53 smithi087 ceph-mon[27999]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:53 smithi087 ceph-mon[27999]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 19 op/s 2024-04-22T13:29:53.698 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T13:29:53.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:29:53.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:53 smithi132 ceph-mon[31531]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:53 smithi132 ceph-mon[31531]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:53 smithi132 ceph-mon[31531]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:53.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:53 smithi132 ceph-mon[31531]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 19 op/s 2024-04-22T13:29:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 -- 172.21.15.87:0/2064520341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103b40 msgr2=0x7f69cc10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 --2- 172.21.15.87:0/2064520341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103b40 0x7f69cc10a3d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009ca0 tx=0x7f69c402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 -- 172.21.15.87:0/2064520341 shutdown_connections 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 --2- 172.21.15.87:0/2064520341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103b40 0x7f69cc10a3d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 --2- 172.21.15.87:0/2064520341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc103180 0x7f69cc103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 --2- 172.21.15.87:0/2064520341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc101f90 0x7f69cc1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 -- 172.21.15.87:0/2064520341 >> 172.21.15.87:0/2064520341 conn(0x7f69cc0fd500 msgr2=0x7f69cc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 -- 172.21.15.87:0/2064520341 shutdown_connections 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 -- 172.21.15.87:0/2064520341 wait complete. 2024-04-22T13:29:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.801+0000 7f69d2243640 1 Processor -- start 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 -- start start 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc101f90 0x7f69cc1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc103b40 0x7f69cc19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc10db30 con 0x7f69cc103180 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc10d9b0 con 0x7f69cc101f90 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69d2243640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc10dcb0 con 0x7f69cc103b40 2024-04-22T13:29:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34052/0 (socket says 172.21.15.87:34052) 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 -- 172.21.15.87:0/2373882193 learned_addr learned my addr 172.21.15.87:0/2373882193 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69cb7fe640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc101f90 0x7f69cc1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69cbfff640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc103b40 0x7f69cc19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 -- 172.21.15.87:0/2373882193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc103b40 msgr2=0x7f69cc19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc103b40 0x7f69cc19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 -- 172.21.15.87:0/2373882193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc101f90 msgr2=0x7f69cc1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc101f90 0x7f69cc1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.803+0000 7f69caffd640 1 -- 172.21.15.87:0/2373882193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69cc19fdf0 con 0x7f69cc103180 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69caffd640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f69c000c890 tx=0x7f69c000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0004220 con 0x7f69cc103180 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69c00043c0 con 0x7f69cc103180 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69cc1a00e0 con 0x7f69cc103180 2024-04-22T13:29:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0004a00 con 0x7f69cc103180 2024-04-22T13:29:53.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.804+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69cc1a0630 con 0x7f69cc103180 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.805+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69cc1092b0 con 0x7f69cc103180 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.806+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f69c00079b0 con 0x7f69cc103180 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.806+0000 7f69c8ff9640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 0x7f69b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.806+0000 7f69cb7fe640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 0x7f69b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.806+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f69c00d0070 con 0x7f69cc103180 2024-04-22T13:29:53.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.807+0000 7f69cb7fe640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 0x7f69b40a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69cc102fe0 tx=0x7f69bc009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:53.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.811+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69c0014030 con 0x7f69cc103180 2024-04-22T13:29:53.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.988+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f69cc062390 con 0x7f69b40a3c40 2024-04-22T13:29:53.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.995+0000 7f69c8ff9640 1 -- 172.21.15.87:0/2373882193 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258921 (secure 0 0 0) 0x7f69cc062390 con 0x7f69b40a3c40 2024-04-22T13:29:53.994 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:53.996 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:29:53.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 msgr2=0x7f69b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 0x7f69b40a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69cc102fe0 tx=0x7f69bc009110 comp rx=0 tx=0).stop 2024-04-22T13:29:53.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 msgr2=0x7f69cc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:53.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f69c000c890 tx=0x7f69c000cc90 comp rx=0 tx=0).stop 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 shutdown_connections 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f69b40a3c40 0x7f69b40a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69cc103b40 0x7f69cc19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69cc103180 0x7f69cc198710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 --2- 172.21.15.87:0/2373882193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc101f90 0x7f69cc1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 >> 172.21.15.87:0/2373882193 conn(0x7f69cc0fd500 msgr2=0x7f69cc106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 shutdown_connections 2024-04-22T13:29:53.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:53.998+0000 7f69d2243640 1 -- 172.21.15.87:0/2373882193 wait complete. 2024-04-22T13:29:54.005 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-04-22T13:29:52.751279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595478,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":144,"num_read_kb":126,"num_write":297,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364368,"kb_used_data":11024,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577456,"statfs":{"total":1151940427776,"available":1151567314944,"internally_reserved":0,"allocated":11288576,"data_stored":7029710,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5110,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":116,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440675"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787913+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:29:48.787913+0000","last_peered":"2024-04-22T13:29:48.787913+0000","last_clean":"2024-04-22T13:29:48.787913+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:29:48.787913+0000","last_undegraded":"2024-04-22T13:29:48.787913+0000","last_fullsized":"2024-04-22T13:29:48.787913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346771+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:29:47.346771+0000","last_peered":"2024-04-22T13:29:47.346771+0000","last_clean":"2024-04-22T13:29:47.346771+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:29:47.346771+0000","last_undegraded":"2024-04-22T13:29:47.346771+0000","last_fullsized":"2024-04-22T13:29:47.346771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:00:59.407004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346874+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:29:47.346874+0000","last_peered":"2024-04-22T13:29:47.346874+0000","last_clean":"2024-04-22T13:29:47.346874+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:29:47.346874+0000","last_undegraded":"2024-04-22T13:29:47.346874+0000","last_fullsized":"2024-04-22T13:29:47.346874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:19:08.112691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012273+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:29:49.012273+0000","last_peered":"2024-04-22T13:29:49.012273+0000","last_clean":"2024-04-22T13:29:49.012273+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:29:49.012273+0000","last_undegraded":"2024-04-22T13:29:49.012273+0000","last_fullsized":"2024-04-22T13:29:49.012273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339713+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:29:48.339713+0000","last_peered":"2024-04-22T13:29:48.339713+0000","last_clean":"2024-04-22T13:29:48.339713+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:29:48.339713+0000","last_undegraded":"2024-04-22T13:29:48.339713+0000","last_fullsized":"2024-04-22T13:29:48.339713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012806+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:29:49.012806+0000","last_peered":"2024-04-22T13:29:49.012806+0000","last_clean":"2024-04-22T13:29:49.012806+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:29:49.012806+0000","last_undegraded":"2024-04-22T13:29:49.012806+0000","last_fullsized":"2024-04-22T13:29:49.012806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012170+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:29:49.012170+0000","last_peered":"2024-04-22T13:29:49.012170+0000","last_clean":"2024-04-22T13:29:49.012170+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:29:49.012170+0000","last_undegraded":"2024-04-22T13:29:49.012170+0000","last_fullsized":"2024-04-22T13:29:49.012170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339040+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:29:48.339040+0000","last_peered":"2024-04-22T13:29:48.339040+0000","last_clean":"2024-04-22T13:29:48.339040+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:29:48.339040+0000","last_undegraded":"2024-04-22T13:29:48.339040+0000","last_fullsized":"2024-04-22T13:29:48.339040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316533+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:29:48.316533+0000","last_peered":"2024-04-22T13:29:48.316533+0000","last_clean":"2024-04-22T13:29:48.316533+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:29:48.316533+0000","last_undegraded":"2024-04-22T13:29:48.316533+0000","last_fullsized":"2024-04-22T13:29:48.316533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013450+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:29:49.013450+0000","last_peered":"2024-04-22T13:29:49.013450+0000","last_clean":"2024-04-22T13:29:49.013450+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:29:49.013450+0000","last_undegraded":"2024-04-22T13:29:49.013450+0000","last_fullsized":"2024-04-22T13:29:49.013450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012867+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:29:49.012867+0000","last_peered":"2024-04-22T13:29:49.012867+0000","last_clean":"2024-04-22T13:29:49.012867+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:29:49.012867+0000","last_undegraded":"2024-04-22T13:29:49.012867+0000","last_fullsized":"2024-04-22T13:29:49.012867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012106+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:29:49.012106+0000","last_peered":"2024-04-22T13:29:49.012106+0000","last_clean":"2024-04-22T13:29:49.012106+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:29:49.012106+0000","last_undegraded":"2024-04-22T13:29:49.012106+0000","last_fullsized":"2024-04-22T13:29:49.012106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317065+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:29:48.317065+0000","last_peered":"2024-04-22T13:29:48.317065+0000","last_clean":"2024-04-22T13:29:48.317065+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:29:48.317065+0000","last_undegraded":"2024-04-22T13:29:48.317065+0000","last_fullsized":"2024-04-22T13:29:48.317065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346623+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:29:47.346623+0000","last_peered":"2024-04-22T13:29:47.346623+0000","last_clean":"2024-04-22T13:29:47.346623+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:29:47.346623+0000","last_undegraded":"2024-04-22T13:29:47.346623+0000","last_fullsized":"2024-04-22T13:29:47.346623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:50.062670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346739+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:29:47.346739+0000","last_peered":"2024-04-22T13:29:47.346739+0000","last_clean":"2024-04-22T13:29:47.346739+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:29:47.346739+0000","last_undegraded":"2024-04-22T13:29:47.346739+0000","last_fullsized":"2024-04-22T13:29:47.346739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:41.241534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315938+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:29:48.315938+0000","last_peered":"2024-04-22T13:29:48.315938+0000","last_clean":"2024-04-22T13:29:48.315938+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:29:48.315938+0000","last_undegraded":"2024-04-22T13:29:48.315938+0000","last_fullsized":"2024-04-22T13:29:48.315938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787938+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:29:48.787938+0000","last_peered":"2024-04-22T13:29:48.787938+0000","last_clean":"2024-04-22T13:29:48.787938+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:29:48.787938+0000","last_undegraded":"2024-04-22T13:29:48.787938+0000","last_fullsized":"2024-04-22T13:29:48.787938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337704+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:29:48.337704+0000","last_peered":"2024-04-22T13:29:48.337704+0000","last_clean":"2024-04-22T13:29:48.337704+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:29:48.337704+0000","last_undegraded":"2024-04-22T13:29:48.337704+0000","last_fullsized":"2024-04-22T13:29:48.337704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316037+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:29:48.316037+0000","last_peered":"2024-04-22T13:29:48.316037+0000","last_clean":"2024-04-22T13:29:48.316037+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:29:48.316037+0000","last_undegraded":"2024-04-22T13:29:48.316037+0000","last_fullsized":"2024-04-22T13:29:48.316037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.329562+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:29:50.329562+0000","last_peered":"2024-04-22T13:29:50.329562+0000","last_clean":"2024-04-22T13:29:50.329562+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:29:50.329562+0000","last_undegraded":"2024-04-22T13:29:50.329562+0000","last_fullsized":"2024-04-22T13:29:50.329562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339356+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:29:48.339356+0000","last_peered":"2024-04-22T13:29:48.339356+0000","last_clean":"2024-04-22T13:29:48.339356+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:29:48.339356+0000","last_undegraded":"2024-04-22T13:29:48.339356+0000","last_fullsized":"2024-04-22T13:29:48.339356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339492+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:29:48.339492+0000","last_peered":"2024-04-22T13:29:48.339492+0000","last_clean":"2024-04-22T13:29:48.339492+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:29:48.339492+0000","last_undegraded":"2024-04-22T13:29:48.339492+0000","last_fullsized":"2024-04-22T13:29:48.339492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346602+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:29:47.346602+0000","last_peered":"2024-04-22T13:29:47.346602+0000","last_clean":"2024-04-22T13:29:47.346602+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:29:47.346602+0000","last_undegraded":"2024-04-22T13:29:47.346602+0000","last_fullsized":"2024-04-22T13:29:47.346602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:15:47.058791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339845+0000","last_change":"2024-04-22T13:29:46.335219+0000","last_active":"2024-04-22T13:29:48.339845+0000","last_peered":"2024-04-22T13:29:48.339845+0000","last_clean":"2024-04-22T13:29:48.339845+0000","last_became_active":"2024-04-22T13:29:46.334462+0000","last_became_peered":"2024-04-22T13:29:46.334462+0000","last_unstale":"2024-04-22T13:29:48.339845+0000","last_undegraded":"2024-04-22T13:29:48.339845+0000","last_fullsized":"2024-04-22T13:29:48.339845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:20:26.262802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316132+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:29:48.316132+0000","last_peered":"2024-04-22T13:29:48.316132+0000","last_clean":"2024-04-22T13:29:48.316132+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:29:48.316132+0000","last_undegraded":"2024-04-22T13:29:48.316132+0000","last_fullsized":"2024-04-22T13:29:48.316132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339695+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:29:48.339695+0000","last_peered":"2024-04-22T13:29:48.339695+0000","last_clean":"2024-04-22T13:29:48.339695+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:29:48.339695+0000","last_undegraded":"2024-04-22T13:29:48.339695+0000","last_fullsized":"2024-04-22T13:29:48.339695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012224+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:29:49.012224+0000","last_peered":"2024-04-22T13:29:49.012224+0000","last_clean":"2024-04-22T13:29:49.012224+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:29:49.012224+0000","last_undegraded":"2024-04-22T13:29:49.012224+0000","last_fullsized":"2024-04-22T13:29:49.012224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318726+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:29:49.318726+0000","last_peered":"2024-04-22T13:29:49.318726+0000","last_clean":"2024-04-22T13:29:49.318726+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:29:49.318726+0000","last_undegraded":"2024-04-22T13:29:49.318726+0000","last_fullsized":"2024-04-22T13:29:49.318726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339565+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:29:48.339565+0000","last_peered":"2024-04-22T13:29:48.339565+0000","last_clean":"2024-04-22T13:29:48.339565+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:29:48.339565+0000","last_undegraded":"2024-04-22T13:29:48.339565+0000","last_fullsized":"2024-04-22T13:29:48.339565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328080+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:29:50.328080+0000","last_peered":"2024-04-22T13:29:50.328080+0000","last_clean":"2024-04-22T13:29:50.328080+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:29:50.328080+0000","last_undegraded":"2024-04-22T13:29:50.328080+0000","last_fullsized":"2024-04-22T13:29:50.328080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012131+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:29:49.012131+0000","last_peered":"2024-04-22T13:29:49.012131+0000","last_clean":"2024-04-22T13:29:49.012131+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:29:49.012131+0000","last_undegraded":"2024-04-22T13:29:49.012131+0000","last_fullsized":"2024-04-22T13:29:49.012131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328010+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:29:50.328010+0000","last_peered":"2024-04-22T13:29:50.328010+0000","last_clean":"2024-04-22T13:29:50.328010+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:29:50.328010+0000","last_undegraded":"2024-04-22T13:29:50.328010+0000","last_fullsized":"2024-04-22T13:29:50.328010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339275+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:29:48.339275+0000","last_peered":"2024-04-22T13:29:48.339275+0000","last_clean":"2024-04-22T13:29:48.339275+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:29:48.339275+0000","last_undegraded":"2024-04-22T13:29:48.339275+0000","last_fullsized":"2024-04-22T13:29:48.339275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011733+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:29:49.011733+0000","last_peered":"2024-04-22T13:29:49.011733+0000","last_clean":"2024-04-22T13:29:49.011733+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:29:49.011733+0000","last_undegraded":"2024-04-22T13:29:49.011733+0000","last_fullsized":"2024-04-22T13:29:49.011733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337515+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:29:48.337515+0000","last_peered":"2024-04-22T13:29:48.337515+0000","last_clean":"2024-04-22T13:29:48.337515+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:29:48.337515+0000","last_undegraded":"2024-04-22T13:29:48.337515+0000","last_fullsized":"2024-04-22T13:29:48.337515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318916+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:29:49.318916+0000","last_peered":"2024-04-22T13:29:49.318916+0000","last_clean":"2024-04-22T13:29:49.318916+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:29:49.318916+0000","last_undegraded":"2024-04-22T13:29:49.318916+0000","last_fullsized":"2024-04-22T13:29:49.318916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316399+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:29:48.316399+0000","last_peered":"2024-04-22T13:29:48.316399+0000","last_clean":"2024-04-22T13:29:48.316399+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:29:48.316399+0000","last_undegraded":"2024-04-22T13:29:48.316399+0000","last_fullsized":"2024-04-22T13:29:48.316399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.317341+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:29:49.317341+0000","last_peered":"2024-04-22T13:29:49.317341+0000","last_clean":"2024-04-22T13:29:49.317341+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:29:49.317341+0000","last_undegraded":"2024-04-22T13:29:49.317341+0000","last_fullsized":"2024-04-22T13:29:49.317341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346829+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:29:47.346829+0000","last_peered":"2024-04-22T13:29:47.346829+0000","last_clean":"2024-04-22T13:29:47.346829+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:29:47.346829+0000","last_undegraded":"2024-04-22T13:29:47.346829+0000","last_fullsized":"2024-04-22T13:29:47.346829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:33:29.788429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012282+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:29:49.012282+0000","last_peered":"2024-04-22T13:29:49.012282+0000","last_clean":"2024-04-22T13:29:49.012282+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:29:49.012282+0000","last_undegraded":"2024-04-22T13:29:49.012282+0000","last_fullsized":"2024-04-22T13:29:49.012282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316220+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:29:48.316220+0000","last_peered":"2024-04-22T13:29:48.316220+0000","last_clean":"2024-04-22T13:29:48.316220+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:29:48.316220+0000","last_undegraded":"2024-04-22T13:29:48.316220+0000","last_fullsized":"2024-04-22T13:29:48.316220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316864+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:29:49.316864+0000","last_peered":"2024-04-22T13:29:49.316864+0000","last_clean":"2024-04-22T13:29:49.316864+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:29:49.316864+0000","last_undegraded":"2024-04-22T13:29:49.316864+0000","last_fullsized":"2024-04-22T13:29:49.316864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339712+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:29:48.339712+0000","last_peered":"2024-04-22T13:29:48.339712+0000","last_clean":"2024-04-22T13:29:48.339712+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:29:48.339712+0000","last_undegraded":"2024-04-22T13:29:48.339712+0000","last_fullsized":"2024-04-22T13:29:48.339712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.326977+0000","last_change":"2024-04-22T13:29:46.318247+0000","last_active":"2024-04-22T13:29:50.326977+0000","last_peered":"2024-04-22T13:29:50.326977+0000","last_clean":"2024-04-22T13:29:50.326977+0000","last_became_active":"2024-04-22T13:29:46.317889+0000","last_became_peered":"2024-04-22T13:29:46.317889+0000","last_unstale":"2024-04-22T13:29:50.326977+0000","last_undegraded":"2024-04-22T13:29:50.326977+0000","last_fullsized":"2024-04-22T13:29:50.326977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:13:00.018160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316972+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:29:48.316972+0000","last_peered":"2024-04-22T13:29:48.316972+0000","last_clean":"2024-04-22T13:29:48.316972+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:29:48.316972+0000","last_undegraded":"2024-04-22T13:29:48.316972+0000","last_fullsized":"2024-04-22T13:29:48.316972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318669+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:29:49.318669+0000","last_peered":"2024-04-22T13:29:49.318669+0000","last_clean":"2024-04-22T13:29:49.318669+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:29:49.318669+0000","last_undegraded":"2024-04-22T13:29:49.318669+0000","last_fullsized":"2024-04-22T13:29:49.318669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012654+0000","last_change":"2024-04-22T13:29:47.337180+0000","last_active":"2024-04-22T13:29:49.012654+0000","last_peered":"2024-04-22T13:29:49.012654+0000","last_clean":"2024-04-22T13:29:49.012654+0000","last_became_active":"2024-04-22T13:29:47.336818+0000","last_became_peered":"2024-04-22T13:29:47.336818+0000","last_unstale":"2024-04-22T13:29:49.012654+0000","last_undegraded":"2024-04-22T13:29:49.012654+0000","last_fullsized":"2024-04-22T13:29:49.012654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:55:33.179558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:51.330594+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:29:51.330594+0000","last_peered":"2024-04-22T13:29:51.330594+0000","last_clean":"2024-04-22T13:29:51.330594+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:29:51.330594+0000","last_undegraded":"2024-04-22T13:29:51.330594+0000","last_fullsized":"2024-04-22T13:29:51.330594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012040+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:29:49.012040+0000","last_peered":"2024-04-22T13:29:49.012040+0000","last_clean":"2024-04-22T13:29:49.012040+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:29:49.012040+0000","last_undegraded":"2024-04-22T13:29:49.012040+0000","last_fullsized":"2024-04-22T13:29:49.012040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'10","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.317238+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:29:49.317238+0000","last_peered":"2024-04-22T13:29:49.317238+0000","last_clean":"2024-04-22T13:29:49.317238+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:29:49.317238+0000","last_undegraded":"2024-04-22T13:29:49.317238+0000","last_fullsized":"2024-04-22T13:29:49.317238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011842+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:29:49.011842+0000","last_peered":"2024-04-22T13:29:49.011842+0000","last_clean":"2024-04-22T13:29:49.011842+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:29:49.011842+0000","last_undegraded":"2024-04-22T13:29:49.011842+0000","last_fullsized":"2024-04-22T13:29:49.011842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339764+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:29:48.339764+0000","last_peered":"2024-04-22T13:29:48.339764+0000","last_clean":"2024-04-22T13:29:48.339764+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:29:48.339764+0000","last_undegraded":"2024-04-22T13:29:48.339764+0000","last_fullsized":"2024-04-22T13:29:48.339764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787828+0000","last_change":"2024-04-22T13:29:47.344166+0000","last_active":"2024-04-22T13:29:48.787828+0000","last_peered":"2024-04-22T13:29:48.787828+0000","last_clean":"2024-04-22T13:29:48.787828+0000","last_became_active":"2024-04-22T13:29:47.343813+0000","last_became_peered":"2024-04-22T13:29:47.343813+0000","last_unstale":"2024-04-22T13:29:48.787828+0000","last_undegraded":"2024-04-22T13:29:48.787828+0000","last_fullsized":"2024-04-22T13:29:48.787828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:22:08.710050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337520+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:29:48.337520+0000","last_peered":"2024-04-22T13:29:48.337520+0000","last_clean":"2024-04-22T13:29:48.337520+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:29:48.337520+0000","last_undegraded":"2024-04-22T13:29:48.337520+0000","last_fullsized":"2024-04-22T13:29:48.337520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013397+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:29:49.013397+0000","last_peered":"2024-04-22T13:29:49.013397+0000","last_clean":"2024-04-22T13:29:49.013397+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:29:49.013397+0000","last_undegraded":"2024-04-22T13:29:49.013397+0000","last_fullsized":"2024-04-22T13:29:49.013397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012413+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:29:49.012413+0000","last_peered":"2024-04-22T13:29:49.012413+0000","last_clean":"2024-04-22T13:29:49.012413+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:29:49.012413+0000","last_undegraded":"2024-04-22T13:29:49.012413+0000","last_fullsized":"2024-04-22T13:29:49.012413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013129+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:29:49.013129+0000","last_peered":"2024-04-22T13:29:49.013129+0000","last_clean":"2024-04-22T13:29:49.013129+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:29:49.013129+0000","last_undegraded":"2024-04-22T13:29:49.013129+0000","last_fullsized":"2024-04-22T13:29:49.013129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012001+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:29:49.012001+0000","last_peered":"2024-04-22T13:29:49.012001+0000","last_clean":"2024-04-22T13:29:49.012001+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:29:49.012001+0000","last_undegraded":"2024-04-22T13:29:49.012001+0000","last_fullsized":"2024-04-22T13:29:49.012001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.330042+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:29:50.330042+0000","last_peered":"2024-04-22T13:29:50.330042+0000","last_clean":"2024-04-22T13:29:50.330042+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:29:50.330042+0000","last_undegraded":"2024-04-22T13:29:50.330042+0000","last_fullsized":"2024-04-22T13:29:50.330042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339478+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:29:48.339478+0000","last_peered":"2024-04-22T13:29:48.339478+0000","last_clean":"2024-04-22T13:29:48.339478+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:29:48.339478+0000","last_undegraded":"2024-04-22T13:29:48.339478+0000","last_fullsized":"2024-04-22T13:29:48.339478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:51.328665+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:29:51.328665+0000","last_peered":"2024-04-22T13:29:51.328665+0000","last_clean":"2024-04-22T13:29:51.328665+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:29:51.328665+0000","last_undegraded":"2024-04-22T13:29:51.328665+0000","last_fullsized":"2024-04-22T13:29:51.328665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011987+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:29:49.011987+0000","last_peered":"2024-04-22T13:29:49.011987+0000","last_clean":"2024-04-22T13:29:49.011987+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:29:49.011987+0000","last_undegraded":"2024-04-22T13:29:49.011987+0000","last_fullsized":"2024-04-22T13:29:49.011987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316826+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:29:48.316826+0000","last_peered":"2024-04-22T13:29:48.316826+0000","last_clean":"2024-04-22T13:29:48.316826+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:29:48.316826+0000","last_undegraded":"2024-04-22T13:29:48.316826+0000","last_fullsized":"2024-04-22T13:29:48.316826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.326352+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:29:50.326352+0000","last_peered":"2024-04-22T13:29:50.326352+0000","last_clean":"2024-04-22T13:29:50.326352+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:29:50.326352+0000","last_undegraded":"2024-04-22T13:29:50.326352+0000","last_fullsized":"2024-04-22T13:29:50.326352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316288+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:29:49.316288+0000","last_peered":"2024-04-22T13:29:49.316288+0000","last_clean":"2024-04-22T13:29:49.316288+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:29:49.316288+0000","last_undegraded":"2024-04-22T13:29:49.316288+0000","last_fullsized":"2024-04-22T13:29:49.316288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317167+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:29:48.317167+0000","last_peered":"2024-04-22T13:29:48.317167+0000","last_clean":"2024-04-22T13:29:48.317167+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:29:48.317167+0000","last_undegraded":"2024-04-22T13:29:48.317167+0000","last_fullsized":"2024-04-22T13:29:48.317167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328423+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:29:50.328423+0000","last_peered":"2024-04-22T13:29:50.328423+0000","last_clean":"2024-04-22T13:29:50.328423+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:29:50.328423+0000","last_undegraded":"2024-04-22T13:29:50.328423+0000","last_fullsized":"2024-04-22T13:29:50.328423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339589+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:29:48.339589+0000","last_peered":"2024-04-22T13:29:48.339589+0000","last_clean":"2024-04-22T13:29:48.339589+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:29:48.339589+0000","last_undegraded":"2024-04-22T13:29:48.339589+0000","last_fullsized":"2024-04-22T13:29:48.339589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.313997+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:29:49.313997+0000","last_peered":"2024-04-22T13:29:49.313997+0000","last_clean":"2024-04-22T13:29:49.313997+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:29:49.313997+0000","last_undegraded":"2024-04-22T13:29:49.313997+0000","last_fullsized":"2024-04-22T13:29:49.313997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346792+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:29:47.346792+0000","last_peered":"2024-04-22T13:29:47.346792+0000","last_clean":"2024-04-22T13:29:47.346792+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:29:47.346792+0000","last_undegraded":"2024-04-22T13:29:47.346792+0000","last_fullsized":"2024-04-22T13:29:47.346792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:31:33.682392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316852+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:29:49.316852+0000","last_peered":"2024-04-22T13:29:49.316852+0000","last_clean":"2024-04-22T13:29:49.316852+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:29:49.316852+0000","last_undegraded":"2024-04-22T13:29:49.316852+0000","last_fullsized":"2024-04-22T13:29:49.316852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339402+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:29:48.339402+0000","last_peered":"2024-04-22T13:29:48.339402+0000","last_clean":"2024-04-22T13:29:48.339402+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:29:48.339402+0000","last_undegraded":"2024-04-22T13:29:48.339402+0000","last_fullsized":"2024-04-22T13:29:48.339402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.319140+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:29:49.319140+0000","last_peered":"2024-04-22T13:29:49.319140+0000","last_clean":"2024-04-22T13:29:49.319140+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:29:49.319140+0000","last_undegraded":"2024-04-22T13:29:49.319140+0000","last_fullsized":"2024-04-22T13:29:49.319140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787619+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:29:48.787619+0000","last_peered":"2024-04-22T13:29:48.787619+0000","last_clean":"2024-04-22T13:29:48.787619+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:29:48.787619+0000","last_undegraded":"2024-04-22T13:29:48.787619+0000","last_fullsized":"2024-04-22T13:29:48.787619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315445+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:29:48.315445+0000","last_peered":"2024-04-22T13:29:48.315445+0000","last_clean":"2024-04-22T13:29:48.315445+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:29:48.315445+0000","last_undegraded":"2024-04-22T13:29:48.315445+0000","last_fullsized":"2024-04-22T13:29:48.315445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339347+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:29:48.339347+0000","last_peered":"2024-04-22T13:29:48.339347+0000","last_clean":"2024-04-22T13:29:48.339347+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:29:48.339347+0000","last_undegraded":"2024-04-22T13:29:48.339347+0000","last_fullsized":"2024-04-22T13:29:48.339347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316800+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:29:49.316800+0000","last_peered":"2024-04-22T13:29:49.316800+0000","last_clean":"2024-04-22T13:29:49.316800+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:29:49.316800+0000","last_undegraded":"2024-04-22T13:29:49.316800+0000","last_fullsized":"2024-04-22T13:29:49.316800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012610+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:29:49.012610+0000","last_peered":"2024-04-22T13:29:49.012610+0000","last_clean":"2024-04-22T13:29:49.012610+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:29:49.012610+0000","last_undegraded":"2024-04-22T13:29:49.012610+0000","last_fullsized":"2024-04-22T13:29:49.012610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.327942+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:29:50.327942+0000","last_peered":"2024-04-22T13:29:50.327942+0000","last_clean":"2024-04-22T13:29:50.327942+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:29:50.327942+0000","last_undegraded":"2024-04-22T13:29:50.327942+0000","last_fullsized":"2024-04-22T13:29:50.327942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012940+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:29:49.012940+0000","last_peered":"2024-04-22T13:29:49.012940+0000","last_clean":"2024-04-22T13:29:49.012940+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:29:49.012940+0000","last_undegraded":"2024-04-22T13:29:49.012940+0000","last_fullsized":"2024-04-22T13:29:49.012940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012097+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:29:49.012097+0000","last_peered":"2024-04-22T13:29:49.012097+0000","last_clean":"2024-04-22T13:29:49.012097+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:29:49.012097+0000","last_undegraded":"2024-04-22T13:29:49.012097+0000","last_fullsized":"2024-04-22T13:29:49.012097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012674+0000","last_change":"2024-04-22T13:29:47.344012+0000","last_active":"2024-04-22T13:29:49.012674+0000","last_peered":"2024-04-22T13:29:49.012674+0000","last_clean":"2024-04-22T13:29:49.012674+0000","last_became_active":"2024-04-22T13:29:47.343615+0000","last_became_peered":"2024-04-22T13:29:47.343615+0000","last_unstale":"2024-04-22T13:29:49.012674+0000","last_undegraded":"2024-04-22T13:29:49.012674+0000","last_fullsized":"2024-04-22T13:29:49.012674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:33:50.110940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317099+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:29:48.317099+0000","last_peered":"2024-04-22T13:29:48.317099+0000","last_clean":"2024-04-22T13:29:48.317099+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:29:48.317099+0000","last_undegraded":"2024-04-22T13:29:48.317099+0000","last_fullsized":"2024-04-22T13:29:48.317099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012934+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:29:49.012934+0000","last_peered":"2024-04-22T13:29:49.012934+0000","last_clean":"2024-04-22T13:29:49.012934+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:29:49.012934+0000","last_undegraded":"2024-04-22T13:29:49.012934+0000","last_fullsized":"2024-04-22T13:29:49.012934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787789+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:29:48.787789+0000","last_peered":"2024-04-22T13:29:48.787789+0000","last_clean":"2024-04-22T13:29:48.787789+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:29:48.787789+0000","last_undegraded":"2024-04-22T13:29:48.787789+0000","last_fullsized":"2024-04-22T13:29:48.787789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316058+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:29:48.316058+0000","last_peered":"2024-04-22T13:29:48.316058+0000","last_clean":"2024-04-22T13:29:48.316058+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:29:48.316058+0000","last_undegraded":"2024-04-22T13:29:48.316058+0000","last_fullsized":"2024-04-22T13:29:48.316058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315735+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:29:48.315735+0000","last_peered":"2024-04-22T13:29:48.315735+0000","last_clean":"2024-04-22T13:29:48.315735+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:29:48.315735+0000","last_undegraded":"2024-04-22T13:29:48.315735+0000","last_fullsized":"2024-04-22T13:29:48.315735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346945+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:29:47.346945+0000","last_peered":"2024-04-22T13:29:47.346945+0000","last_clean":"2024-04-22T13:29:47.346945+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:29:47.346945+0000","last_undegraded":"2024-04-22T13:29:47.346945+0000","last_fullsized":"2024-04-22T13:29:47.346945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:19.108638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317162+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:29:48.317162+0000","last_peered":"2024-04-22T13:29:48.317162+0000","last_clean":"2024-04-22T13:29:48.317162+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:29:48.317162+0000","last_undegraded":"2024-04-22T13:29:48.317162+0000","last_fullsized":"2024-04-22T13:29:48.317162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787867+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:29:48.787867+0000","last_peered":"2024-04-22T13:29:48.787867+0000","last_clean":"2024-04-22T13:29:48.787867+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:29:48.787867+0000","last_undegraded":"2024-04-22T13:29:48.787867+0000","last_fullsized":"2024-04-22T13:29:48.787867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316094+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:29:48.316094+0000","last_peered":"2024-04-22T13:29:48.316094+0000","last_clean":"2024-04-22T13:29:48.316094+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:29:48.316094+0000","last_undegraded":"2024-04-22T13:29:48.316094+0000","last_fullsized":"2024-04-22T13:29:48.316094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339510+0000","last_change":"2024-04-22T13:29:47.337619+0000","last_active":"2024-04-22T13:29:48.339510+0000","last_peered":"2024-04-22T13:29:48.339510+0000","last_clean":"2024-04-22T13:29:48.339510+0000","last_became_active":"2024-04-22T13:29:47.337247+0000","last_became_peered":"2024-04-22T13:29:47.337247+0000","last_unstale":"2024-04-22T13:29:48.339510+0000","last_undegraded":"2024-04-22T13:29:48.339510+0000","last_fullsized":"2024-04-22T13:29:48.339510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:26.843612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011843+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:29:49.011843+0000","last_peered":"2024-04-22T13:29:49.011843+0000","last_clean":"2024-04-22T13:29:49.011843+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:29:49.011843+0000","last_undegraded":"2024-04-22T13:29:49.011843+0000","last_fullsized":"2024-04-22T13:29:49.011843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339241+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:29:48.339241+0000","last_peered":"2024-04-22T13:29:48.339241+0000","last_clean":"2024-04-22T13:29:48.339241+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:29:48.339241+0000","last_undegraded":"2024-04-22T13:29:48.339241+0000","last_fullsized":"2024-04-22T13:29:48.339241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316092+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:29:48.316092+0000","last_peered":"2024-04-22T13:29:48.316092+0000","last_clean":"2024-04-22T13:29:48.316092+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:29:48.316092+0000","last_undegraded":"2024-04-22T13:29:48.316092+0000","last_fullsized":"2024-04-22T13:29:48.316092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317101+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:29:48.317101+0000","last_peered":"2024-04-22T13:29:48.317101+0000","last_clean":"2024-04-22T13:29:48.317101+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:29:48.317101+0000","last_undegraded":"2024-04-22T13:29:48.317101+0000","last_fullsized":"2024-04-22T13:29:48.317101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787832+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:29:48.787832+0000","last_peered":"2024-04-22T13:29:48.787832+0000","last_clean":"2024-04-22T13:29:48.787832+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:29:48.787832+0000","last_undegraded":"2024-04-22T13:29:48.787832+0000","last_fullsized":"2024-04-22T13:29:48.787832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281019,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5110,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":116,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":13494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1030681,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":905216,"data_stored":430041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:29:54.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- 172.21.15.87:0/3279686955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5780101b60 msgr2=0x7f5780101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- 172.21.15.87:0/3279686955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5780101b60 0x7f5780101fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5778009ca0 tx=0x7f577802c180 comp rx=0 tx=0).stop 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- 172.21.15.87:0/3279686955 shutdown_connections 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- 172.21.15.87:0/3279686955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5780101b60 0x7f5780101fe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- 172.21.15.87:0/3279686955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57801011c0 0x7f5780101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- 172.21.15.87:0/3279686955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5780100860 0x7f5780100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- 172.21.15.87:0/3279686955 >> 172.21.15.87:0/3279686955 conn(0x7f57800fc3e0 msgr2=0x7f57800fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:54.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- 172.21.15.87:0/3279686955 shutdown_connections 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- 172.21.15.87:0/3279686955 wait complete. 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 Processor -- start 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- start start 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5780100860 0x7f578019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5780101b60 0x7f57801a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f578010fd10 con 0x7f57801011c0 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f578010fb90 con 0x7f5780101b60 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f57860d9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f578010fe90 con 0x7f5780100860 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f5777fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f5777fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34066/0 (socket says 172.21.15.87:34066) 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.149+0000 7f5777fff640 1 -- 172.21.15.87:0/2134121238 learned_addr learned my addr 172.21.15.87:0/2134121238 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f577f7fe640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5780100860 0x7f578019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 -- 172.21.15.87:0/2134121238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5780100860 msgr2=0x7f578019a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5780100860 0x7f578019a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 -- 172.21.15.87:0/2134121238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5780101b60 msgr2=0x7f57801a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5780101b60 0x7f57801a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 -- 172.21.15.87:0/2134121238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57801a1fd0 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f5777fff640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f576800d770 tx=0x7f576800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768017070 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57680054d0 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768004c30 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57801a22c0 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.150+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57801a2840 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.151+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f578010b7a0 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.152+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5768004dd0 con 0x7f57801011c0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.153+0000 7f577d7fa640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 0x7f576c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.153+0000 7f577f7fe640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 0x7f576c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.153+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f57680d0390 con 0x7f57801011c0 2024-04-22T13:29:54.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.153+0000 7f577f7fe640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 0x7f576c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5770009890 tx=0x7f5770008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:54.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.156+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f576809cdc0 con 0x7f57801011c0 2024-04-22T13:29:54.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.335+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5780062390 con 0x7f576c0a3c40 2024-04-22T13:29:54.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.342+0000 7f577d7fa640 1 -- 172.21.15.87:0/2134121238 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258921 (secure 0 0 0) 0x7f5780062390 con 0x7f576c0a3c40 2024-04-22T13:29:54.341 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:54.343 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:29:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.345+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 msgr2=0x7f576c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.345+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 0x7f576c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5770009890 tx=0x7f5770008040 comp rx=0 tx=0).stop 2024-04-22T13:29:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.345+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 msgr2=0x7f578019a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:54.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.345+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f576800d770 tx=0x7f576800dc30 comp rx=0 tx=0).stop 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 shutdown_connections 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f576c0a3c40 0x7f576c0a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5780101b60 0x7f57801a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57801011c0 0x7f578019a8f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 --2- 172.21.15.87:0/2134121238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5780100860 0x7f578019a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 >> 172.21.15.87:0/2134121238 conn(0x7f57800fc3e0 msgr2=0x7f57800fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 shutdown_connections 2024-04-22T13:29:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:54.346+0000 7f57860d9640 1 -- 172.21.15.87:0/2134121238 wait complete. 2024-04-22T13:29:54.353 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-04-22T13:29:52.751279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595478,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":144,"num_read_kb":126,"num_write":297,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364368,"kb_used_data":11024,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577456,"statfs":{"total":1151940427776,"available":1151567314944,"internally_reserved":0,"allocated":11288576,"data_stored":7029710,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5110,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":116,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440675"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787913+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:29:48.787913+0000","last_peered":"2024-04-22T13:29:48.787913+0000","last_clean":"2024-04-22T13:29:48.787913+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:29:48.787913+0000","last_undegraded":"2024-04-22T13:29:48.787913+0000","last_fullsized":"2024-04-22T13:29:48.787913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346771+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:29:47.346771+0000","last_peered":"2024-04-22T13:29:47.346771+0000","last_clean":"2024-04-22T13:29:47.346771+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:29:47.346771+0000","last_undegraded":"2024-04-22T13:29:47.346771+0000","last_fullsized":"2024-04-22T13:29:47.346771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:00:59.407004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346874+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:29:47.346874+0000","last_peered":"2024-04-22T13:29:47.346874+0000","last_clean":"2024-04-22T13:29:47.346874+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:29:47.346874+0000","last_undegraded":"2024-04-22T13:29:47.346874+0000","last_fullsized":"2024-04-22T13:29:47.346874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:19:08.112691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012273+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:29:49.012273+0000","last_peered":"2024-04-22T13:29:49.012273+0000","last_clean":"2024-04-22T13:29:49.012273+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:29:49.012273+0000","last_undegraded":"2024-04-22T13:29:49.012273+0000","last_fullsized":"2024-04-22T13:29:49.012273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339713+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:29:48.339713+0000","last_peered":"2024-04-22T13:29:48.339713+0000","last_clean":"2024-04-22T13:29:48.339713+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:29:48.339713+0000","last_undegraded":"2024-04-22T13:29:48.339713+0000","last_fullsized":"2024-04-22T13:29:48.339713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012806+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:29:49.012806+0000","last_peered":"2024-04-22T13:29:49.012806+0000","last_clean":"2024-04-22T13:29:49.012806+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:29:49.012806+0000","last_undegraded":"2024-04-22T13:29:49.012806+0000","last_fullsized":"2024-04-22T13:29:49.012806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012170+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:29:49.012170+0000","last_peered":"2024-04-22T13:29:49.012170+0000","last_clean":"2024-04-22T13:29:49.012170+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:29:49.012170+0000","last_undegraded":"2024-04-22T13:29:49.012170+0000","last_fullsized":"2024-04-22T13:29:49.012170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339040+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:29:48.339040+0000","last_peered":"2024-04-22T13:29:48.339040+0000","last_clean":"2024-04-22T13:29:48.339040+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:29:48.339040+0000","last_undegraded":"2024-04-22T13:29:48.339040+0000","last_fullsized":"2024-04-22T13:29:48.339040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316533+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:29:48.316533+0000","last_peered":"2024-04-22T13:29:48.316533+0000","last_clean":"2024-04-22T13:29:48.316533+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:29:48.316533+0000","last_undegraded":"2024-04-22T13:29:48.316533+0000","last_fullsized":"2024-04-22T13:29:48.316533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013450+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:29:49.013450+0000","last_peered":"2024-04-22T13:29:49.013450+0000","last_clean":"2024-04-22T13:29:49.013450+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:29:49.013450+0000","last_undegraded":"2024-04-22T13:29:49.013450+0000","last_fullsized":"2024-04-22T13:29:49.013450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012867+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:29:49.012867+0000","last_peered":"2024-04-22T13:29:49.012867+0000","last_clean":"2024-04-22T13:29:49.012867+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:29:49.012867+0000","last_undegraded":"2024-04-22T13:29:49.012867+0000","last_fullsized":"2024-04-22T13:29:49.012867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012106+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:29:49.012106+0000","last_peered":"2024-04-22T13:29:49.012106+0000","last_clean":"2024-04-22T13:29:49.012106+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:29:49.012106+0000","last_undegraded":"2024-04-22T13:29:49.012106+0000","last_fullsized":"2024-04-22T13:29:49.012106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317065+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:29:48.317065+0000","last_peered":"2024-04-22T13:29:48.317065+0000","last_clean":"2024-04-22T13:29:48.317065+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:29:48.317065+0000","last_undegraded":"2024-04-22T13:29:48.317065+0000","last_fullsized":"2024-04-22T13:29:48.317065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346623+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:29:47.346623+0000","last_peered":"2024-04-22T13:29:47.346623+0000","last_clean":"2024-04-22T13:29:47.346623+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:29:47.346623+0000","last_undegraded":"2024-04-22T13:29:47.346623+0000","last_fullsized":"2024-04-22T13:29:47.346623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:50.062670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346739+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:29:47.346739+0000","last_peered":"2024-04-22T13:29:47.346739+0000","last_clean":"2024-04-22T13:29:47.346739+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:29:47.346739+0000","last_undegraded":"2024-04-22T13:29:47.346739+0000","last_fullsized":"2024-04-22T13:29:47.346739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:41.241534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315938+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:29:48.315938+0000","last_peered":"2024-04-22T13:29:48.315938+0000","last_clean":"2024-04-22T13:29:48.315938+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:29:48.315938+0000","last_undegraded":"2024-04-22T13:29:48.315938+0000","last_fullsized":"2024-04-22T13:29:48.315938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787938+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:29:48.787938+0000","last_peered":"2024-04-22T13:29:48.787938+0000","last_clean":"2024-04-22T13:29:48.787938+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:29:48.787938+0000","last_undegraded":"2024-04-22T13:29:48.787938+0000","last_fullsized":"2024-04-22T13:29:48.787938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337704+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:29:48.337704+0000","last_peered":"2024-04-22T13:29:48.337704+0000","last_clean":"2024-04-22T13:29:48.337704+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:29:48.337704+0000","last_undegraded":"2024-04-22T13:29:48.337704+0000","last_fullsized":"2024-04-22T13:29:48.337704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316037+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:29:48.316037+0000","last_peered":"2024-04-22T13:29:48.316037+0000","last_clean":"2024-04-22T13:29:48.316037+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:29:48.316037+0000","last_undegraded":"2024-04-22T13:29:48.316037+0000","last_fullsized":"2024-04-22T13:29:48.316037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.329562+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:29:50.329562+0000","last_peered":"2024-04-22T13:29:50.329562+0000","last_clean":"2024-04-22T13:29:50.329562+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:29:50.329562+0000","last_undegraded":"2024-04-22T13:29:50.329562+0000","last_fullsized":"2024-04-22T13:29:50.329562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339356+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:29:48.339356+0000","last_peered":"2024-04-22T13:29:48.339356+0000","last_clean":"2024-04-22T13:29:48.339356+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:29:48.339356+0000","last_undegraded":"2024-04-22T13:29:48.339356+0000","last_fullsized":"2024-04-22T13:29:48.339356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339492+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:29:48.339492+0000","last_peered":"2024-04-22T13:29:48.339492+0000","last_clean":"2024-04-22T13:29:48.339492+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:29:48.339492+0000","last_undegraded":"2024-04-22T13:29:48.339492+0000","last_fullsized":"2024-04-22T13:29:48.339492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346602+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:29:47.346602+0000","last_peered":"2024-04-22T13:29:47.346602+0000","last_clean":"2024-04-22T13:29:47.346602+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:29:47.346602+0000","last_undegraded":"2024-04-22T13:29:47.346602+0000","last_fullsized":"2024-04-22T13:29:47.346602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:15:47.058791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339845+0000","last_change":"2024-04-22T13:29:46.335219+0000","last_active":"2024-04-22T13:29:48.339845+0000","last_peered":"2024-04-22T13:29:48.339845+0000","last_clean":"2024-04-22T13:29:48.339845+0000","last_became_active":"2024-04-22T13:29:46.334462+0000","last_became_peered":"2024-04-22T13:29:46.334462+0000","last_unstale":"2024-04-22T13:29:48.339845+0000","last_undegraded":"2024-04-22T13:29:48.339845+0000","last_fullsized":"2024-04-22T13:29:48.339845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:20:26.262802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316132+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:29:48.316132+0000","last_peered":"2024-04-22T13:29:48.316132+0000","last_clean":"2024-04-22T13:29:48.316132+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:29:48.316132+0000","last_undegraded":"2024-04-22T13:29:48.316132+0000","last_fullsized":"2024-04-22T13:29:48.316132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339695+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:29:48.339695+0000","last_peered":"2024-04-22T13:29:48.339695+0000","last_clean":"2024-04-22T13:29:48.339695+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:29:48.339695+0000","last_undegraded":"2024-04-22T13:29:48.339695+0000","last_fullsized":"2024-04-22T13:29:48.339695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012224+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:29:49.012224+0000","last_peered":"2024-04-22T13:29:49.012224+0000","last_clean":"2024-04-22T13:29:49.012224+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:29:49.012224+0000","last_undegraded":"2024-04-22T13:29:49.012224+0000","last_fullsized":"2024-04-22T13:29:49.012224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318726+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:29:49.318726+0000","last_peered":"2024-04-22T13:29:49.318726+0000","last_clean":"2024-04-22T13:29:49.318726+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:29:49.318726+0000","last_undegraded":"2024-04-22T13:29:49.318726+0000","last_fullsized":"2024-04-22T13:29:49.318726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339565+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:29:48.339565+0000","last_peered":"2024-04-22T13:29:48.339565+0000","last_clean":"2024-04-22T13:29:48.339565+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:29:48.339565+0000","last_undegraded":"2024-04-22T13:29:48.339565+0000","last_fullsized":"2024-04-22T13:29:48.339565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328080+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:29:50.328080+0000","last_peered":"2024-04-22T13:29:50.328080+0000","last_clean":"2024-04-22T13:29:50.328080+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:29:50.328080+0000","last_undegraded":"2024-04-22T13:29:50.328080+0000","last_fullsized":"2024-04-22T13:29:50.328080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012131+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:29:49.012131+0000","last_peered":"2024-04-22T13:29:49.012131+0000","last_clean":"2024-04-22T13:29:49.012131+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:29:49.012131+0000","last_undegraded":"2024-04-22T13:29:49.012131+0000","last_fullsized":"2024-04-22T13:29:49.012131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328010+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:29:50.328010+0000","last_peered":"2024-04-22T13:29:50.328010+0000","last_clean":"2024-04-22T13:29:50.328010+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:29:50.328010+0000","last_undegraded":"2024-04-22T13:29:50.328010+0000","last_fullsized":"2024-04-22T13:29:50.328010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339275+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:29:48.339275+0000","last_peered":"2024-04-22T13:29:48.339275+0000","last_clean":"2024-04-22T13:29:48.339275+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:29:48.339275+0000","last_undegraded":"2024-04-22T13:29:48.339275+0000","last_fullsized":"2024-04-22T13:29:48.339275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011733+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:29:49.011733+0000","last_peered":"2024-04-22T13:29:49.011733+0000","last_clean":"2024-04-22T13:29:49.011733+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:29:49.011733+0000","last_undegraded":"2024-04-22T13:29:49.011733+0000","last_fullsized":"2024-04-22T13:29:49.011733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337515+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:29:48.337515+0000","last_peered":"2024-04-22T13:29:48.337515+0000","last_clean":"2024-04-22T13:29:48.337515+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:29:48.337515+0000","last_undegraded":"2024-04-22T13:29:48.337515+0000","last_fullsized":"2024-04-22T13:29:48.337515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318916+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:29:49.318916+0000","last_peered":"2024-04-22T13:29:49.318916+0000","last_clean":"2024-04-22T13:29:49.318916+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:29:49.318916+0000","last_undegraded":"2024-04-22T13:29:49.318916+0000","last_fullsized":"2024-04-22T13:29:49.318916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316399+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:29:48.316399+0000","last_peered":"2024-04-22T13:29:48.316399+0000","last_clean":"2024-04-22T13:29:48.316399+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:29:48.316399+0000","last_undegraded":"2024-04-22T13:29:48.316399+0000","last_fullsized":"2024-04-22T13:29:48.316399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.317341+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:29:49.317341+0000","last_peered":"2024-04-22T13:29:49.317341+0000","last_clean":"2024-04-22T13:29:49.317341+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:29:49.317341+0000","last_undegraded":"2024-04-22T13:29:49.317341+0000","last_fullsized":"2024-04-22T13:29:49.317341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346829+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:29:47.346829+0000","last_peered":"2024-04-22T13:29:47.346829+0000","last_clean":"2024-04-22T13:29:47.346829+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:29:47.346829+0000","last_undegraded":"2024-04-22T13:29:47.346829+0000","last_fullsized":"2024-04-22T13:29:47.346829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:33:29.788429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012282+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:29:49.012282+0000","last_peered":"2024-04-22T13:29:49.012282+0000","last_clean":"2024-04-22T13:29:49.012282+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:29:49.012282+0000","last_undegraded":"2024-04-22T13:29:49.012282+0000","last_fullsized":"2024-04-22T13:29:49.012282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316220+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:29:48.316220+0000","last_peered":"2024-04-22T13:29:48.316220+0000","last_clean":"2024-04-22T13:29:48.316220+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:29:48.316220+0000","last_undegraded":"2024-04-22T13:29:48.316220+0000","last_fullsized":"2024-04-22T13:29:48.316220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316864+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:29:49.316864+0000","last_peered":"2024-04-22T13:29:49.316864+0000","last_clean":"2024-04-22T13:29:49.316864+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:29:49.316864+0000","last_undegraded":"2024-04-22T13:29:49.316864+0000","last_fullsized":"2024-04-22T13:29:49.316864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339712+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:29:48.339712+0000","last_peered":"2024-04-22T13:29:48.339712+0000","last_clean":"2024-04-22T13:29:48.339712+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:29:48.339712+0000","last_undegraded":"2024-04-22T13:29:48.339712+0000","last_fullsized":"2024-04-22T13:29:48.339712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.326977+0000","last_change":"2024-04-22T13:29:46.318247+0000","last_active":"2024-04-22T13:29:50.326977+0000","last_peered":"2024-04-22T13:29:50.326977+0000","last_clean":"2024-04-22T13:29:50.326977+0000","last_became_active":"2024-04-22T13:29:46.317889+0000","last_became_peered":"2024-04-22T13:29:46.317889+0000","last_unstale":"2024-04-22T13:29:50.326977+0000","last_undegraded":"2024-04-22T13:29:50.326977+0000","last_fullsized":"2024-04-22T13:29:50.326977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:13:00.018160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316972+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:29:48.316972+0000","last_peered":"2024-04-22T13:29:48.316972+0000","last_clean":"2024-04-22T13:29:48.316972+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:29:48.316972+0000","last_undegraded":"2024-04-22T13:29:48.316972+0000","last_fullsized":"2024-04-22T13:29:48.316972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.318669+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:29:49.318669+0000","last_peered":"2024-04-22T13:29:49.318669+0000","last_clean":"2024-04-22T13:29:49.318669+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:29:49.318669+0000","last_undegraded":"2024-04-22T13:29:49.318669+0000","last_fullsized":"2024-04-22T13:29:49.318669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012654+0000","last_change":"2024-04-22T13:29:47.337180+0000","last_active":"2024-04-22T13:29:49.012654+0000","last_peered":"2024-04-22T13:29:49.012654+0000","last_clean":"2024-04-22T13:29:49.012654+0000","last_became_active":"2024-04-22T13:29:47.336818+0000","last_became_peered":"2024-04-22T13:29:47.336818+0000","last_unstale":"2024-04-22T13:29:49.012654+0000","last_undegraded":"2024-04-22T13:29:49.012654+0000","last_fullsized":"2024-04-22T13:29:49.012654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:55:33.179558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:51.330594+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:29:51.330594+0000","last_peered":"2024-04-22T13:29:51.330594+0000","last_clean":"2024-04-22T13:29:51.330594+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:29:51.330594+0000","last_undegraded":"2024-04-22T13:29:51.330594+0000","last_fullsized":"2024-04-22T13:29:51.330594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012040+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:29:49.012040+0000","last_peered":"2024-04-22T13:29:49.012040+0000","last_clean":"2024-04-22T13:29:49.012040+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:29:49.012040+0000","last_undegraded":"2024-04-22T13:29:49.012040+0000","last_fullsized":"2024-04-22T13:29:49.012040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'10","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.317238+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:29:49.317238+0000","last_peered":"2024-04-22T13:29:49.317238+0000","last_clean":"2024-04-22T13:29:49.317238+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:29:49.317238+0000","last_undegraded":"2024-04-22T13:29:49.317238+0000","last_fullsized":"2024-04-22T13:29:49.317238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011842+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:29:49.011842+0000","last_peered":"2024-04-22T13:29:49.011842+0000","last_clean":"2024-04-22T13:29:49.011842+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:29:49.011842+0000","last_undegraded":"2024-04-22T13:29:49.011842+0000","last_fullsized":"2024-04-22T13:29:49.011842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339764+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:29:48.339764+0000","last_peered":"2024-04-22T13:29:48.339764+0000","last_clean":"2024-04-22T13:29:48.339764+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:29:48.339764+0000","last_undegraded":"2024-04-22T13:29:48.339764+0000","last_fullsized":"2024-04-22T13:29:48.339764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787828+0000","last_change":"2024-04-22T13:29:47.344166+0000","last_active":"2024-04-22T13:29:48.787828+0000","last_peered":"2024-04-22T13:29:48.787828+0000","last_clean":"2024-04-22T13:29:48.787828+0000","last_became_active":"2024-04-22T13:29:47.343813+0000","last_became_peered":"2024-04-22T13:29:47.343813+0000","last_unstale":"2024-04-22T13:29:48.787828+0000","last_undegraded":"2024-04-22T13:29:48.787828+0000","last_fullsized":"2024-04-22T13:29:48.787828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:22:08.710050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.337520+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:29:48.337520+0000","last_peered":"2024-04-22T13:29:48.337520+0000","last_clean":"2024-04-22T13:29:48.337520+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:29:48.337520+0000","last_undegraded":"2024-04-22T13:29:48.337520+0000","last_fullsized":"2024-04-22T13:29:48.337520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013397+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:29:49.013397+0000","last_peered":"2024-04-22T13:29:49.013397+0000","last_clean":"2024-04-22T13:29:49.013397+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:29:49.013397+0000","last_undegraded":"2024-04-22T13:29:49.013397+0000","last_fullsized":"2024-04-22T13:29:49.013397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012413+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:29:49.012413+0000","last_peered":"2024-04-22T13:29:49.012413+0000","last_clean":"2024-04-22T13:29:49.012413+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:29:49.012413+0000","last_undegraded":"2024-04-22T13:29:49.012413+0000","last_fullsized":"2024-04-22T13:29:49.012413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.013129+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:29:49.013129+0000","last_peered":"2024-04-22T13:29:49.013129+0000","last_clean":"2024-04-22T13:29:49.013129+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:29:49.013129+0000","last_undegraded":"2024-04-22T13:29:49.013129+0000","last_fullsized":"2024-04-22T13:29:49.013129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012001+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:29:49.012001+0000","last_peered":"2024-04-22T13:29:49.012001+0000","last_clean":"2024-04-22T13:29:49.012001+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:29:49.012001+0000","last_undegraded":"2024-04-22T13:29:49.012001+0000","last_fullsized":"2024-04-22T13:29:49.012001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.330042+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:29:50.330042+0000","last_peered":"2024-04-22T13:29:50.330042+0000","last_clean":"2024-04-22T13:29:50.330042+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:29:50.330042+0000","last_undegraded":"2024-04-22T13:29:50.330042+0000","last_fullsized":"2024-04-22T13:29:50.330042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339478+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:29:48.339478+0000","last_peered":"2024-04-22T13:29:48.339478+0000","last_clean":"2024-04-22T13:29:48.339478+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:29:48.339478+0000","last_undegraded":"2024-04-22T13:29:48.339478+0000","last_fullsized":"2024-04-22T13:29:48.339478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:51.328665+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:29:51.328665+0000","last_peered":"2024-04-22T13:29:51.328665+0000","last_clean":"2024-04-22T13:29:51.328665+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:29:51.328665+0000","last_undegraded":"2024-04-22T13:29:51.328665+0000","last_fullsized":"2024-04-22T13:29:51.328665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011987+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:29:49.011987+0000","last_peered":"2024-04-22T13:29:49.011987+0000","last_clean":"2024-04-22T13:29:49.011987+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:29:49.011987+0000","last_undegraded":"2024-04-22T13:29:49.011987+0000","last_fullsized":"2024-04-22T13:29:49.011987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316826+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:29:48.316826+0000","last_peered":"2024-04-22T13:29:48.316826+0000","last_clean":"2024-04-22T13:29:48.316826+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:29:48.316826+0000","last_undegraded":"2024-04-22T13:29:48.316826+0000","last_fullsized":"2024-04-22T13:29:48.316826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.326352+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:29:50.326352+0000","last_peered":"2024-04-22T13:29:50.326352+0000","last_clean":"2024-04-22T13:29:50.326352+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:29:50.326352+0000","last_undegraded":"2024-04-22T13:29:50.326352+0000","last_fullsized":"2024-04-22T13:29:50.326352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316288+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:29:49.316288+0000","last_peered":"2024-04-22T13:29:49.316288+0000","last_clean":"2024-04-22T13:29:49.316288+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:29:49.316288+0000","last_undegraded":"2024-04-22T13:29:49.316288+0000","last_fullsized":"2024-04-22T13:29:49.316288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317167+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:29:48.317167+0000","last_peered":"2024-04-22T13:29:48.317167+0000","last_clean":"2024-04-22T13:29:48.317167+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:29:48.317167+0000","last_undegraded":"2024-04-22T13:29:48.317167+0000","last_fullsized":"2024-04-22T13:29:48.317167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.328423+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:29:50.328423+0000","last_peered":"2024-04-22T13:29:50.328423+0000","last_clean":"2024-04-22T13:29:50.328423+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:29:50.328423+0000","last_undegraded":"2024-04-22T13:29:50.328423+0000","last_fullsized":"2024-04-22T13:29:50.328423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339589+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:29:48.339589+0000","last_peered":"2024-04-22T13:29:48.339589+0000","last_clean":"2024-04-22T13:29:48.339589+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:29:48.339589+0000","last_undegraded":"2024-04-22T13:29:48.339589+0000","last_fullsized":"2024-04-22T13:29:48.339589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.313997+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:29:49.313997+0000","last_peered":"2024-04-22T13:29:49.313997+0000","last_clean":"2024-04-22T13:29:49.313997+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:29:49.313997+0000","last_undegraded":"2024-04-22T13:29:49.313997+0000","last_fullsized":"2024-04-22T13:29:49.313997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346792+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:29:47.346792+0000","last_peered":"2024-04-22T13:29:47.346792+0000","last_clean":"2024-04-22T13:29:47.346792+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:29:47.346792+0000","last_undegraded":"2024-04-22T13:29:47.346792+0000","last_fullsized":"2024-04-22T13:29:47.346792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:31:33.682392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316852+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:29:49.316852+0000","last_peered":"2024-04-22T13:29:49.316852+0000","last_clean":"2024-04-22T13:29:49.316852+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:29:49.316852+0000","last_undegraded":"2024-04-22T13:29:49.316852+0000","last_fullsized":"2024-04-22T13:29:49.316852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339402+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:29:48.339402+0000","last_peered":"2024-04-22T13:29:48.339402+0000","last_clean":"2024-04-22T13:29:48.339402+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:29:48.339402+0000","last_undegraded":"2024-04-22T13:29:48.339402+0000","last_fullsized":"2024-04-22T13:29:48.339402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.319140+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:29:49.319140+0000","last_peered":"2024-04-22T13:29:49.319140+0000","last_clean":"2024-04-22T13:29:49.319140+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:29:49.319140+0000","last_undegraded":"2024-04-22T13:29:49.319140+0000","last_fullsized":"2024-04-22T13:29:49.319140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787619+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:29:48.787619+0000","last_peered":"2024-04-22T13:29:48.787619+0000","last_clean":"2024-04-22T13:29:48.787619+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:29:48.787619+0000","last_undegraded":"2024-04-22T13:29:48.787619+0000","last_fullsized":"2024-04-22T13:29:48.787619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315445+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:29:48.315445+0000","last_peered":"2024-04-22T13:29:48.315445+0000","last_clean":"2024-04-22T13:29:48.315445+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:29:48.315445+0000","last_undegraded":"2024-04-22T13:29:48.315445+0000","last_fullsized":"2024-04-22T13:29:48.315445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339347+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:29:48.339347+0000","last_peered":"2024-04-22T13:29:48.339347+0000","last_clean":"2024-04-22T13:29:48.339347+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:29:48.339347+0000","last_undegraded":"2024-04-22T13:29:48.339347+0000","last_fullsized":"2024-04-22T13:29:48.339347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.316800+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:29:49.316800+0000","last_peered":"2024-04-22T13:29:49.316800+0000","last_clean":"2024-04-22T13:29:49.316800+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:29:49.316800+0000","last_undegraded":"2024-04-22T13:29:49.316800+0000","last_fullsized":"2024-04-22T13:29:49.316800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012610+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:29:49.012610+0000","last_peered":"2024-04-22T13:29:49.012610+0000","last_clean":"2024-04-22T13:29:49.012610+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:29:49.012610+0000","last_undegraded":"2024-04-22T13:29:49.012610+0000","last_fullsized":"2024-04-22T13:29:49.012610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:50.327942+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:29:50.327942+0000","last_peered":"2024-04-22T13:29:50.327942+0000","last_clean":"2024-04-22T13:29:50.327942+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:29:50.327942+0000","last_undegraded":"2024-04-22T13:29:50.327942+0000","last_fullsized":"2024-04-22T13:29:50.327942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012940+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:29:49.012940+0000","last_peered":"2024-04-22T13:29:49.012940+0000","last_clean":"2024-04-22T13:29:49.012940+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:29:49.012940+0000","last_undegraded":"2024-04-22T13:29:49.012940+0000","last_fullsized":"2024-04-22T13:29:49.012940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012097+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:29:49.012097+0000","last_peered":"2024-04-22T13:29:49.012097+0000","last_clean":"2024-04-22T13:29:49.012097+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:29:49.012097+0000","last_undegraded":"2024-04-22T13:29:49.012097+0000","last_fullsized":"2024-04-22T13:29:49.012097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012674+0000","last_change":"2024-04-22T13:29:47.344012+0000","last_active":"2024-04-22T13:29:49.012674+0000","last_peered":"2024-04-22T13:29:49.012674+0000","last_clean":"2024-04-22T13:29:49.012674+0000","last_became_active":"2024-04-22T13:29:47.343615+0000","last_became_peered":"2024-04-22T13:29:47.343615+0000","last_unstale":"2024-04-22T13:29:49.012674+0000","last_undegraded":"2024-04-22T13:29:49.012674+0000","last_fullsized":"2024-04-22T13:29:49.012674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:33:50.110940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317099+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:29:48.317099+0000","last_peered":"2024-04-22T13:29:48.317099+0000","last_clean":"2024-04-22T13:29:48.317099+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:29:48.317099+0000","last_undegraded":"2024-04-22T13:29:48.317099+0000","last_fullsized":"2024-04-22T13:29:48.317099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.012934+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:29:49.012934+0000","last_peered":"2024-04-22T13:29:49.012934+0000","last_clean":"2024-04-22T13:29:49.012934+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:29:49.012934+0000","last_undegraded":"2024-04-22T13:29:49.012934+0000","last_fullsized":"2024-04-22T13:29:49.012934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787789+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:29:48.787789+0000","last_peered":"2024-04-22T13:29:48.787789+0000","last_clean":"2024-04-22T13:29:48.787789+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:29:48.787789+0000","last_undegraded":"2024-04-22T13:29:48.787789+0000","last_fullsized":"2024-04-22T13:29:48.787789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316058+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:29:48.316058+0000","last_peered":"2024-04-22T13:29:48.316058+0000","last_clean":"2024-04-22T13:29:48.316058+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:29:48.316058+0000","last_undegraded":"2024-04-22T13:29:48.316058+0000","last_fullsized":"2024-04-22T13:29:48.316058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.315735+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:29:48.315735+0000","last_peered":"2024-04-22T13:29:48.315735+0000","last_clean":"2024-04-22T13:29:48.315735+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:29:48.315735+0000","last_undegraded":"2024-04-22T13:29:48.315735+0000","last_fullsized":"2024-04-22T13:29:48.315735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T13:29:47.346945+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:29:47.346945+0000","last_peered":"2024-04-22T13:29:47.346945+0000","last_clean":"2024-04-22T13:29:47.346945+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:29:47.346945+0000","last_undegraded":"2024-04-22T13:29:47.346945+0000","last_fullsized":"2024-04-22T13:29:47.346945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:55:19.108638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317162+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:29:48.317162+0000","last_peered":"2024-04-22T13:29:48.317162+0000","last_clean":"2024-04-22T13:29:48.317162+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:29:48.317162+0000","last_undegraded":"2024-04-22T13:29:48.317162+0000","last_fullsized":"2024-04-22T13:29:48.317162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787867+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:29:48.787867+0000","last_peered":"2024-04-22T13:29:48.787867+0000","last_clean":"2024-04-22T13:29:48.787867+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:29:48.787867+0000","last_undegraded":"2024-04-22T13:29:48.787867+0000","last_fullsized":"2024-04-22T13:29:48.787867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316094+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:29:48.316094+0000","last_peered":"2024-04-22T13:29:48.316094+0000","last_clean":"2024-04-22T13:29:48.316094+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:29:48.316094+0000","last_undegraded":"2024-04-22T13:29:48.316094+0000","last_fullsized":"2024-04-22T13:29:48.316094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339510+0000","last_change":"2024-04-22T13:29:47.337619+0000","last_active":"2024-04-22T13:29:48.339510+0000","last_peered":"2024-04-22T13:29:48.339510+0000","last_clean":"2024-04-22T13:29:48.339510+0000","last_became_active":"2024-04-22T13:29:47.337247+0000","last_became_peered":"2024-04-22T13:29:47.337247+0000","last_unstale":"2024-04-22T13:29:48.339510+0000","last_undegraded":"2024-04-22T13:29:48.339510+0000","last_fullsized":"2024-04-22T13:29:48.339510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:26.843612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:49.011843+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:29:49.011843+0000","last_peered":"2024-04-22T13:29:49.011843+0000","last_clean":"2024-04-22T13:29:49.011843+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:29:49.011843+0000","last_undegraded":"2024-04-22T13:29:49.011843+0000","last_fullsized":"2024-04-22T13:29:49.011843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.339241+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:29:48.339241+0000","last_peered":"2024-04-22T13:29:48.339241+0000","last_clean":"2024-04-22T13:29:48.339241+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:29:48.339241+0000","last_undegraded":"2024-04-22T13:29:48.339241+0000","last_fullsized":"2024-04-22T13:29:48.339241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.316092+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:29:48.316092+0000","last_peered":"2024-04-22T13:29:48.316092+0000","last_clean":"2024-04-22T13:29:48.316092+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:29:48.316092+0000","last_undegraded":"2024-04-22T13:29:48.316092+0000","last_fullsized":"2024-04-22T13:29:48.316092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.317101+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:29:48.317101+0000","last_peered":"2024-04-22T13:29:48.317101+0000","last_clean":"2024-04-22T13:29:48.317101+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:29:48.317101+0000","last_undegraded":"2024-04-22T13:29:48.317101+0000","last_fullsized":"2024-04-22T13:29:48.317101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T13:29:48.787832+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:29:48.787832+0000","last_peered":"2024-04-22T13:29:48.787832+0000","last_clean":"2024-04-22T13:29:48.787832+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:29:48.787832+0000","last_undegraded":"2024-04-22T13:29:48.787832+0000","last_fullsized":"2024-04-22T13:29:48.787832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281019,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5110,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":116,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":13494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1030681,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":905216,"data_stored":430041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:29:54.358 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T13:29:54.358 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T13:29:54.358 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T13:29:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:29:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:54 smithi120 ceph-mon[31101]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:54 smithi120 ceph-mon[31101]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T13:29:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:54 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/892511023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:54 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3748148886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:54.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:54 smithi132 ceph-mon[31531]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:54 smithi132 ceph-mon[31531]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T13:29:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:54 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/892511023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:54 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3748148886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:54.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:54 smithi087 ceph-mon[27999]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:29:54.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:54 smithi087 ceph-mon[27999]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T13:29:54.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/892511023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:29:54.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3748148886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:29:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:55 smithi120 ceph-mon[31101]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:55 smithi120 ceph-mon[31101]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:55 smithi120 ceph-mon[31101]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-04-22T13:29:55.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:55 smithi132 ceph-mon[31531]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:55 smithi132 ceph-mon[31531]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:55 smithi132 ceph-mon[31531]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-04-22T13:29:55.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:55 smithi087 ceph-mon[27999]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:55 smithi087 ceph-mon[27999]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:29:55.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:55 smithi087 ceph-mon[27999]: pgmap v294: 97 pgs: 97 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- 172.21.15.87:0/394969379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101c40 msgr2=0x7f19e01020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 --2- 172.21.15.87:0/394969379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101c40 0x7f19e01020c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009990 tx=0x7f19d0009ca0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- 172.21.15.87:0/394969379 shutdown_connections 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 --2- 172.21.15.87:0/394969379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0102600 0x7f19e0109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 --2- 172.21.15.87:0/394969379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 0x7f19e0103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 --2- 172.21.15.87:0/394969379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101c40 0x7f19e01020c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- 172.21.15.87:0/394969379 >> 172.21.15.87:0/394969379 conn(0x7f19e00fc710 msgr2=0x7f19e00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- 172.21.15.87:0/394969379 shutdown_connections 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- 172.21.15.87:0/394969379 wait complete. 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 Processor -- start 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.381+0000 7f19e8438700 1 -- start start 2024-04-22T13:29:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101c40 0x7f19e01059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 0x7f19e0104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f19e010d480 con 0x7f19e0102600 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19e010d300 con 0x7f19e0101c40 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f19e010d600 con 0x7f19e0103070 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34084/0 (socket says 172.21.15.87:34084) 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 -- 172.21.15.87:0/688083760 learned_addr learned my addr 172.21.15.87:0/688083760 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e69d5700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 0x7f19e0104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e61d4700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101c40 0x7f19e01059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 -- 172.21.15.87:0/688083760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 msgr2=0x7f19e0104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 0x7f19e0104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 -- 172.21.15.87:0/688083760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101c40 msgr2=0x7f19e01059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101c40 0x7f19e01059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 -- 172.21.15.87:0/688083760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e0104e20 con 0x7f19e0102600 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e59d3700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f19d0006010 tx=0x7f19d00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0027070 con 0x7f19e0102600 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e01a2c20 con 0x7f19e0102600 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d00040c0 con 0x7f19e0102600 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a3070 con 0x7f19e0102600 2024-04-22T13:29:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.382+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0022d50 con 0x7f19e0102600 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.383+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e0050f80 con 0x7f19e0102600 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.384+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f19d002a070 con 0x7f19e0102600 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.385+0000 7f19d77fe700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 0x7f19cc0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.385+0000 7f19e61d4700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 0x7f19cc0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.385+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f19d00d11b0 con 0x7f19e0102600 2024-04-22T13:29:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.385+0000 7f19e61d4700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 0x7f19cc0a6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f19dc004630 tx=0x7f19dc00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.389+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19d0013070 con 0x7f19e0102600 2024-04-22T13:29:56.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.671+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f19e01a35a0 con 0x7f19e0102600 2024-04-22T13:29:56.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.673+0000 7f19d77fe700 1 -- 172.21.15.87:0/688083760 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f19e01a35a0 con 0x7f19e0102600 2024-04-22T13:29:56.668 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-04-22T13:29:56.669 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQBUZiZmOT4TKBAAdQIzwZPaoPpiygvK8L1R8w== 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 msgr2=0x7f19cc0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 0x7f19cc0a6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f19dc004630 tx=0x7f19dc00b3c0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 msgr2=0x7f19e0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f19d0006010 tx=0x7f19d00047b0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 shutdown_connections 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f19cc0a3d80 0x7f19cc0a6240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0103070 0x7f19e0104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101c40 0x7f19e01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 --2- 172.21.15.87:0/688083760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0102600 0x7f19e0104030 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:56.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 >> 172.21.15.87:0/688083760 conn(0x7f19e00fc710 msgr2=0x7f19e00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:56.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 shutdown_connections 2024-04-22T13:29:56.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:56.675+0000 7f19e8438700 1 -- 172.21.15.87:0/688083760 wait complete. 2024-04-22T13:29:56.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T13:29:56.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T13:29:57.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T13:29:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T13:29:57.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T13:29:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/688083760' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T13:29:57.365 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:29:57.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T13:29:57.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T13:29:57.400 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T13:29:57.400 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T13:29:57.401 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T13:29:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:29:57.978 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:57 smithi087 ceph-mon[27999]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.0 KiB/s wr, 20 op/s 2024-04-22T13:29:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:57 smithi120 ceph-mon[31101]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.0 KiB/s wr, 20 op/s 2024-04-22T13:29:58.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:57 smithi132 ceph-mon[31531]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.0 KiB/s wr, 20 op/s 2024-04-22T13:29:59.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.548+0000 7f3a5c59e700 1 -- 172.21.15.87:0/659069372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 msgr2=0x7f3a5410f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:59.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.548+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/659069372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a5410f920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00b0d0 tx=0x7f3a4c00b3e0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- 172.21.15.87:0/659069372 shutdown_connections 2024-04-22T13:29:59.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/659069372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a5410f920 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/659069372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a54107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/659069372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 0x7f3a54102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- 172.21.15.87:0/659069372 >> 172.21.15.87:0/659069372 conn(0x7f3a54079ba0 msgr2=0x7f3a54079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:59.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- 172.21.15.87:0/659069372 shutdown_connections 2024-04-22T13:29:59.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- 172.21.15.87:0/659069372 wait complete. 2024-04-22T13:29:59.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 Processor -- start 2024-04-22T13:29:59.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- start start 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 0x7f3a54198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a5419d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54112c30 con 0x7f3a541034b0 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54112ab0 con 0x7f3a54107940 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.549+0000 7f3a5c59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54112db0 con 0x7f3a54102af0 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34108/0 (socket says 172.21.15.87:34108) 2024-04-22T13:29:59.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 -- 172.21.15.87:0/267730986 learned_addr learned my addr 172.21.15.87:0/267730986 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a5a33a700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 0x7f3a54198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a5ab3b700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a5419d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 -- 172.21.15.87:0/267730986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 msgr2=0x7f3a54198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 0x7f3a54198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 -- 172.21.15.87:0/267730986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 msgr2=0x7f3a5419d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a5419d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 -- 172.21.15.87:0/267730986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5419dfa0 con 0x7f3a541034b0 2024-04-22T13:29:59.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a5ab3b700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a5419d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a59b39700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c9b0 tx=0x7f3a4400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44007b00 con 0x7f3a541034b0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5419e290 con 0x7f3a541034b0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a44007ca0 con 0x7f3a541034b0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44018660 con 0x7f3a541034b0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.550+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a541a5b10 con 0x7f3a541034b0 2024-04-22T13:29:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.551+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a54069730 con 0x7f3a541034b0 2024-04-22T13:29:59.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.552+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a4400f450 con 0x7f3a541034b0 2024-04-22T13:29:59.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.553+0000 7f3a4b7fe700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 0x7f3a400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:29:59.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.553+0000 7f3a5a33a700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 0x7f3a400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:29:59.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.553+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3a440d0640 con 0x7f3a541034b0 2024-04-22T13:29:59.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.553+0000 7f3a5a33a700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 0x7f3a400a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005950 tx=0x7f3a5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:29:59.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.557+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a44014030 con 0x7f3a541034b0 2024-04-22T13:29:59.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.816+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3a5419a2f0 con 0x7f3a541034b0 2024-04-22T13:29:59.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.821+0000 7f3a4b7fe700 1 -- 172.21.15.87:0/267730986 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264996 (secure 0 0 0) 0x7f3a5419a2f0 con 0x7f3a541034b0 2024-04-22T13:29:59.820 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:29:59.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 msgr2=0x7f3a400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:59.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 0x7f3a400a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005950 tx=0x7f3a5000b410 comp rx=0 tx=0).stop 2024-04-22T13:29:59.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 msgr2=0x7f3a54199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:29:59.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c9b0 tx=0x7f3a4400ccc0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 shutdown_connections 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a400a3cd0 0x7f3a400a6190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a54107940 0x7f3a5419d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a54102af0 0x7f3a54198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 --2- 172.21.15.87:0/267730986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a541034b0 0x7f3a54199520 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 >> 172.21.15.87:0/267730986 conn(0x7f3a54079ba0 msgr2=0x7f3a54100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 shutdown_connections 2024-04-22T13:29:59.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:29:59.824+0000 7f3a5c59e700 1 -- 172.21.15.87:0/267730986 wait complete. 2024-04-22T13:30:00.100 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:29:59 smithi087 ceph-mon[27999]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T13:30:00.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:29:59 smithi120 ceph-mon[31101]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T13:30:00.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:29:59 smithi132 ceph-mon[31531]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T13:30:00.699 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1421536657},{"type":"v1","addr":"172.21.15.87:6801","nonce":1421536657}]},"active_addr":"172.21.15.87:6801/1421536657","active_change":"2024-04-22T13:22:36.668336+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.87:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":194272199}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3207002517}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":837668825}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":4211868253}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2292675537}]}]} 2024-04-22T13:30:00.706 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T13:30:00.706 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T13:30:00.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T13:30:00.954 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:00.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:00 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/267730986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T13:30:00.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:00 smithi087 ceph-mon[27999]: overall HEALTH_OK 2024-04-22T13:30:01.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:00 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/267730986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T13:30:01.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:00 smithi120 ceph-mon[31101]: overall HEALTH_OK 2024-04-22T13:30:01.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:00 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/267730986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T13:30:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:00 smithi132 ceph-mon[31531]: overall HEALTH_OK 2024-04-22T13:30:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:01 smithi120 ceph-mon[31101]: pgmap v297: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-22T13:30:02.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:01 smithi087 ceph-mon[27999]: pgmap v297: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-22T13:30:02.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:01 smithi132 ceph-mon[31531]: pgmap v297: 97 pgs: 97 active+clean; 623 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-22T13:30:02.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 -- 172.21.15.87:0/2346479540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05681019d0 msgr2=0x7f0568101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:02.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 --2- 172.21.15.87:0/2346479540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05681019d0 0x7f0568101e50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0550009990 tx=0x7f0550009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 -- 172.21.15.87:0/2346479540 shutdown_connections 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 --2- 172.21.15.87:0/2346479540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568102390 0x7f056810e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 --2- 172.21.15.87:0/2346479540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0568107940 0x7f0568107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 --2- 172.21.15.87:0/2346479540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05681019d0 0x7f0568101e50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 -- 172.21.15.87:0/2346479540 >> 172.21.15.87:0/2346479540 conn(0x7f0568077990 msgr2=0x7f0568077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 -- 172.21.15.87:0/2346479540 shutdown_connections 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 -- 172.21.15.87:0/2346479540 wait complete. 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.713+0000 7f056dd91700 1 Processor -- start 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 -- start start 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681019d0 0x7f0568198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:02.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 0x7f056819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111be0 con 0x7f0568102390 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111a60 con 0x7f05681019d0 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f056dd91700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111d60 con 0x7f0568107940 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:02.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47586/0 (socket says 172.21.15.87:47586) 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 -- 172.21.15.87:0/1146959886 learned_addr learned my addr 172.21.15.87:0/1146959886 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f05677fe700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681019d0 0x7f0568198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f0567fff700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 0x7f056819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 -- 172.21.15.87:0/1146959886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 msgr2=0x7f056819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 0x7f056819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 -- 172.21.15.87:0/1146959886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681019d0 msgr2=0x7f0568198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:02.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681019d0 0x7f0568198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f055effd700 1 -- 172.21.15.87:0/1146959886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056819d140 con 0x7f0568102390 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.714+0000 7f0567fff700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 0x7f056819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f055effd700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f05500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550027070 con 0x7f0568102390 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05500040e0 con 0x7f0568102390 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056819d3d0 con 0x7f0568102390 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055000f460 con 0x7f0568102390 2024-04-22T13:30:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.715+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05681a4cb0 con 0x7f0568102390 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.716+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0568103080 con 0x7f0568102390 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.717+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f055002a070 con 0x7f0568102390 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.717+0000 7f05657fa700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 0x7f05540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.718+0000 7f05677fe700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 0x7f05540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.718+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f05500d17f0 con 0x7f0568102390 2024-04-22T13:30:02.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.718+0000 7f05677fe700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 0x7f05540a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0558009a10 tx=0x7f0558009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:02.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.721+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0550031070 con 0x7f0568102390 2024-04-22T13:30:02.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.923+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f056802df80 con 0x7f0568102390 2024-04-22T13:30:02.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.924+0000 7f05657fa700 1 -- 172.21.15.87:0/1146959886 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22238 (secure 0 0 0) 0x7f0550013070 con 0x7f0568102390 2024-04-22T13:30:02.920 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:30:02.921 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":85,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:29:48.311419+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:30:02.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 msgr2=0x7f05540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 0x7f05540a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0558009a10 tx=0x7f0558009190 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 msgr2=0x7f0568198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f05500047b0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 shutdown_connections 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f05540a3cd0 0x7f05540a6190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0568107940 0x7f056819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681019d0 0x7f0568198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 --2- 172.21.15.87:0/1146959886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0568102390 0x7f0568198670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 >> 172.21.15.87:0/1146959886 conn(0x7f0568077990 msgr2=0x7f05680ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 shutdown_connections 2024-04-22T13:30:02.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:02.927+0000 7f056dd91700 1 -- 172.21.15.87:0/1146959886 wait complete. 2024-04-22T13:30:03.772 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T13:30:03.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T13:30:04.015 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:03 smithi087 ceph-mon[27999]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-22T13:30:04.137 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:03 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1146959886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:03 smithi120 ceph-mon[31101]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-22T13:30:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:03 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1146959886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:03 smithi132 ceph-mon[31531]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.5 KiB/s wr, 17 op/s 2024-04-22T13:30:04.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:03 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1146959886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:05.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.780+0000 7fadadc0b700 1 -- 172.21.15.87:0/1365690996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8107990 msgr2=0x7fada8107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.780+0000 7fadadc0b700 1 --2- 172.21.15.87:0/1365690996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8107990 0x7fada8107d70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fad98009990 tx=0x7fad98009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 -- 172.21.15.87:0/1365690996 shutdown_connections 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 --2- 172.21.15.87:0/1365690996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8102300 0x7fada810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 --2- 172.21.15.87:0/1365690996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8107990 0x7fada8107d70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 --2- 172.21.15.87:0/1365690996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 -- 172.21.15.87:0/1365690996 >> 172.21.15.87:0/1365690996 conn(0x7fada80788f0 msgr2=0x7fada8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 -- 172.21.15.87:0/1365690996 shutdown_connections 2024-04-22T13:30:05.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 -- 172.21.15.87:0/1365690996 wait complete. 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 Processor -- start 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.781+0000 7fadadc0b700 1 -- start start 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada81990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8107990 0x7fada819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fada8111b00 con 0x7fada8102300 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fada8111980 con 0x7fada8101940 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fada8111c80 con 0x7fada8107990 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47612/0 (socket says 172.21.15.87:47612) 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 -- 172.21.15.87:0/3211309712 learned_addr learned my addr 172.21.15.87:0/3211309712 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada7fff700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8107990 0x7fada819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada77fe700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada81990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:05.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 -- 172.21.15.87:0/3211309712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8107990 msgr2=0x7fada819d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8107990 0x7fada819d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 -- 172.21.15.87:0/3211309712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 msgr2=0x7fada81990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada81990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 -- 172.21.15.87:0/3211309712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada819e0c0 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada77fe700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada81990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada6ffd700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c9b0 tx=0x7fad9c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c007b00 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada819e3b0 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad9c007ca0 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c018730 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.782+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada81a5c30 con 0x7fada8102300 2024-04-22T13:30:05.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.783+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada8069730 con 0x7fada8102300 2024-04-22T13:30:05.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.785+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fad9c00f450 con 0x7fada8102300 2024-04-22T13:30:05.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.785+0000 7fada4ff9700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 0x7fad900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:05.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.785+0000 7fada77fe700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 0x7fad900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:05.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.785+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fad9c0d0790 con 0x7fada8102300 2024-04-22T13:30:05.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.786+0000 7fada77fe700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 0x7fad900a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad98000c00 tx=0x7fad98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:05.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.789+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad9c014030 con 0x7fada8102300 2024-04-22T13:30:05.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.989+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fada819a410 con 0x7fada8102300 2024-04-22T13:30:05.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.990+0000 7fada4ff9700 1 -- 172.21.15.87:0/3211309712 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22238 (secure 0 0 0) 0x7fada819a410 con 0x7fada8102300 2024-04-22T13:30:05.986 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:30:05.986 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":85,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:29:48.311419+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 msgr2=0x7fad900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 0x7fad900a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad98000c00 tx=0x7fad98005fb0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 msgr2=0x7fada81995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00c9b0 tx=0x7fad9c00cd70 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 shutdown_connections 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fad900a3cd0 0x7fad900a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada8107990 0x7fada819d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada8101940 0x7fada81990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 --2- 172.21.15.87:0/3211309712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada8102300 0x7fada81995f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:05.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.992+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 >> 172.21.15.87:0/3211309712 conn(0x7fada80788f0 msgr2=0x7fada806a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:05.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.993+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 shutdown_connections 2024-04-22T13:30:05.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:05.993+0000 7fadadc0b700 1 -- 172.21.15.87:0/3211309712 wait complete. 2024-04-22T13:30:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:05 smithi120 ceph-mon[31101]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 9 op/s 2024-04-22T13:30:06.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:05 smithi132 ceph-mon[31531]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 9 op/s 2024-04-22T13:30:06.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:05 smithi087 ceph-mon[27999]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 9 op/s 2024-04-22T13:30:06.728 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T13:30:06.728 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T13:30:06.729 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T13:30:06.729 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T13:30:06.729 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T13:30:06.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T13:30:06.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T13:30:06.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T13:30:06.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T13:30:06.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T13:30:06.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T13:30:06.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T13:30:06.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3211309712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:07.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:06 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3211309712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:07.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:06 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3211309712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:30:07.441 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:07.449 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:07.449 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:07.458 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:07.567 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:08.147 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:07 smithi087 ceph-mon[27999]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-22T13:30:08.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:07 smithi120 ceph-mon[31101]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-22T13:30:08.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:07 smithi132 ceph-mon[31531]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-04-22T13:30:08.998 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:08.999 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:08.999 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:09.537 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:09.538 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:09.553 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:10.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:09 smithi120 ceph-mon[31101]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:30:10.195 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:09 smithi087 ceph-mon[27999]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:30:10.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:09 smithi132 ceph-mon[31531]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:30:11.985 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:12.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:11 smithi087 ceph-mon[27999]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T13:30:12.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:11 smithi120 ceph-mon[31101]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T13:30:12.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:11 smithi132 ceph-mon[31531]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T13:30:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:13 smithi120 ceph-mon[31101]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-22T13:30:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:13 smithi132 ceph-mon[31531]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-22T13:30:14.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:13 smithi087 ceph-mon[27999]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-22T13:30:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 -- 172.21.15.87:0/2117158213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c1019d0 msgr2=0x7fa91c10f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2117158213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c1019d0 0x7fa91c10f910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa91400b0a0 tx=0x7fa91400b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 -- 172.21.15.87:0/2117158213 shutdown_connections 2024-04-22T13:30:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2117158213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c1019d0 0x7fa91c10f910 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2117158213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c108ad0 0x7fa91c108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2117158213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa91c101030 0x7fa91c101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 -- 172.21.15.87:0/2117158213 >> 172.21.15.87:0/2117158213 conn(0x7fa91c077fb0 msgr2=0x7fa91c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 -- 172.21.15.87:0/2117158213 shutdown_connections 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.783+0000 7fa9213e1700 1 -- 172.21.15.87:0/2117158213 wait complete. 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 Processor -- start 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 -- start start 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 0x7fa91c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c1019d0 0x7fa91c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa91c108ad0 0x7fa91c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c112c20 con 0x7fa91c101030 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c112aa0 con 0x7fa91c1019d0 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa9213e1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c112da0 con 0x7fa91c108ad0 2024-04-22T13:30:15.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 0x7fa91c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 0x7fa91c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54372/0 (socket says 172.21.15.87:54372) 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 -- 172.21.15.87:0/2492187293 learned_addr learned my addr 172.21.15.87:0/2492187293 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91a7fc700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c1019d0 0x7fa91c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91b7fe700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa91c108ad0 0x7fa91c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 -- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa91c108ad0 msgr2=0x7fa91c19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa91c108ad0 0x7fa91c19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 -- 172.21.15.87:0/2492187293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c1019d0 msgr2=0x7fa91c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c1019d0 0x7fa91c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91affd700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c19e0a0 con 0x7fa91c101030 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.784+0000 7fa91a7fc700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa91c1019d0 0x7fa91c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa91affd700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 0x7fa91c199090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c920 tx=0x7fa91000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910007b00 con 0x7fa91c101030 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c19e390 con 0x7fa91c101030 2024-04-22T13:30:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa910004ce0 con 0x7fa91c101030 2024-04-22T13:30:15.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910005650 con 0x7fa91c101030 2024-04-22T13:30:15.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a5c10 con 0x7fa91c101030 2024-04-22T13:30:15.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.785+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa91c0631d0 con 0x7fa91c101030 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.787+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa910007ca0 con 0x7fa91c101030 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.787+0000 7fa903fff700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa9040a3cd0 0x7fa9040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.787+0000 7fa91a7fc700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa9040a3cd0 0x7fa9040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.787+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa9100d04d0 con 0x7fa91c101030 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa903fff700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] conn(0x7fa9040ad4b0 0x7fa9040af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa910012ec0 con 0x7fa9040ad4b0 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa9100d09b0 con 0x7fa91c101030 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa91a7fc700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa9040a3cd0 0x7fa9040a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0ff060 tx=0x7fa90c00b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:15.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa91b7fe700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] conn(0x7fa9040ad4b0 0x7fa9040af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:15.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.788+0000 7fa91b7fe700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] conn(0x7fa9040ad4b0 0x7fa9040af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:15.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.789+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== osd.8 v2:172.21.15.132:6816/1871032509 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa910012ec0 con 0x7fa9040ad4b0 2024-04-22T13:30:15.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.805+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 --> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa91c0635d0 con 0x7fa9040ad4b0 2024-04-22T13:30:15.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa903fff700 1 -- 172.21.15.87:0/2492187293 <== osd.8 v2:172.21.15.132:6816/1871032509 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa91c0635d0 con 0x7fa9040ad4b0 2024-04-22T13:30:15.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] conn(0x7fa9040ad4b0 msgr2=0x7fa9040af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] conn(0x7fa9040ad4b0 0x7fa9040af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa9040a3cd0 msgr2=0x7fa9040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa9040a3cd0 0x7fa9040a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0ff060 tx=0x7fa90c00b2c0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 msgr2=0x7fa91c199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 --2- 172.21.15.87:0/2492187293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa91c101030 0x7fa91c199090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c920 tx=0x7fa91000cce0 comp rx=0 tx=0).stop 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa91b7fe700 1 -- 172.21.15.87:0/2492187293 reap_dead start 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 shutdown_connections 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.806+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 >> 172.21.15.87:0/2492187293 conn(0x7fa91c077fb0 msgr2=0x7fa91c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.807+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 shutdown_connections 2024-04-22T13:30:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:15.807+0000 7fa9213e1700 1 -- 172.21.15.87:0/2492187293 wait complete. 2024-04-22T13:30:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:15 smithi120 ceph-mon[31101]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:16.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:15 smithi132 ceph-mon[31531]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:16.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:15 smithi087 ceph-mon[27999]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:17.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 -- 172.21.15.87:0/3241150633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 msgr2=0x7f20f810f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 --2- 172.21.15.87:0/3241150633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f810f8e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b0a0 tx=0x7f20f000b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 -- 172.21.15.87:0/3241150633 shutdown_connections 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 --2- 172.21.15.87:0/3241150633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f810f8e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 --2- 172.21.15.87:0/3241150633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8108a80 0x7f20f8108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 --2- 172.21.15.87:0/3241150633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f8102af0 0x7f20f8102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 -- 172.21.15.87:0/3241150633 >> 172.21.15.87:0/3241150633 conn(0x7f20f8079b20 msgr2=0x7f20f8079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 -- 172.21.15.87:0/3241150633 shutdown_connections 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.219+0000 7f20ffefa700 1 -- 172.21.15.87:0/3241150633 wait complete. 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 Processor -- start 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 -- start start 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8102af0 0x7f20f8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f8108a80 0x7f20f80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8112c10 con 0x7f20f81034b0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8112a90 con 0x7f20f8102af0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20ffefa700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8112d90 con 0x7f20f8108a80 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f8075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54380/0 (socket says 172.21.15.87:54380) 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 -- 172.21.15.87:0/2140012685 learned_addr learned my addr 172.21.15.87:0/2140012685 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fdc96700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8102af0 0x7f20f8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 -- 172.21.15.87:0/2140012685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f8108a80 msgr2=0x7f20f80757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fe497700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f8108a80 0x7f20f80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f8108a80 0x7f20f80757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 -- 172.21.15.87:0/2140012685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8102af0 msgr2=0x7f20f8074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8102af0 0x7f20f8074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f81a7490 con 0x7f20f81034b0 2024-04-22T13:30:17.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fdc96700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20f8102af0 0x7f20f8074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20fd495700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f8075280 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f20f400c9b0 tx=0x7f20f400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.220+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4007b00 con 0x7f20f81034b0 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.221+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20f4007ca0 con 0x7f20f81034b0 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.221+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f81a7780 con 0x7f20f81034b0 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.221+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4018730 con 0x7f20f81034b0 2024-04-22T13:30:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.221+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f81a7cc0 con 0x7f20f81034b0 2024-04-22T13:30:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.221+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f20f8101ee0 con 0x7f20f81034b0 2024-04-22T13:30:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f20f400f450 con 0x7f20f81034b0 2024-04-22T13:30:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20e40a3cd0 0x7f20e40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20fdc96700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20e40a3cd0 0x7f20e40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f20f40d0910 con 0x7f20f81034b0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] conn(0x7f20e40ad4b0 0x7f20e40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20f4007e50 con 0x7f20e40ad4b0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.223+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f20f40d0df0 con 0x7f20f81034b0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.224+0000 7f20fe497700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] conn(0x7f20e40ad4b0 0x7f20e40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.224+0000 7f20fdc96700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20e40a3cd0 0x7f20e40a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f20e8005fd0 tx=0x7f20e800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.224+0000 7f20fe497700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] conn(0x7f20e40ad4b0 0x7f20e40af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.225+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== osd.6 v2:172.21.15.87:6818/3192057229 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f20f4007e50 con 0x7f20e40ad4b0 2024-04-22T13:30:17.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 --> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f20f81041b0 con 0x7f20e40ad4b0 2024-04-22T13:30:17.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20eeffd700 1 -- 172.21.15.87:0/2140012685 <== osd.6 v2:172.21.15.87:6818/3192057229 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f20f81041b0 con 0x7f20e40ad4b0 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] conn(0x7f20e40ad4b0 msgr2=0x7f20e40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] conn(0x7f20e40ad4b0 0x7f20e40af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20e40a3cd0 msgr2=0x7f20e40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f20e40a3cd0 0x7f20e40a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f20e8005fd0 tx=0x7f20e800a5e0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 msgr2=0x7f20f8075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 --2- 172.21.15.87:0/2140012685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20f81034b0 0x7f20f8075280 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f20f400c9b0 tx=0x7f20f400cd70 comp rx=0 tx=0).stop 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20fe497700 1 -- 172.21.15.87:0/2140012685 reap_dead start 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 shutdown_connections 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 >> 172.21.15.87:0/2140012685 conn(0x7f20f8079b20 msgr2=0x7f20f8101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 shutdown_connections 2024-04-22T13:30:17.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.241+0000 7f20ffefa700 1 -- 172.21.15.87:0/2140012685 wait complete. 2024-04-22T13:30:17.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 -- 172.21.15.87:0/1175677753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 msgr2=0x7fcd6c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 --2- 172.21.15.87:0/1175677753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c102f10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009990 tx=0x7fcd60009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 -- 172.21.15.87:0/1175677753 shutdown_connections 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 --2- 172.21.15.87:0/1175677753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd6c103450 0x7fcd6c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 --2- 172.21.15.87:0/1175677753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd6c108a50 0x7fcd6c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 --2- 172.21.15.87:0/1175677753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c102f10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 -- 172.21.15.87:0/1175677753 >> 172.21.15.87:0/1175677753 conn(0x7fcd6c079ba0 msgr2=0x7fcd6c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 -- 172.21.15.87:0/1175677753 shutdown_connections 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 -- 172.21.15.87:0/1175677753 wait complete. 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.974+0000 7fcd72527700 1 Processor -- start 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 -- start start 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd6c103450 0x7fcd6c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd6c108a50 0x7fcd6c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6c112c80 con 0x7fcd6c102a90 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6c112b00 con 0x7fcd6c108a50 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd72527700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6c112e00 con 0x7fcd6c103450 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54390/0 (socket says 172.21.15.87:54390) 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 -- 172.21.15.87:0/185288596 learned_addr learned my addr 172.21.15.87:0/185288596 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6b7fe700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd6c103450 0x7fcd6c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd70ac4700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd6c108a50 0x7fcd6c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 -- 172.21.15.87:0/185288596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd6c103450 msgr2=0x7fcd6c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd6c103450 0x7fcd6c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 -- 172.21.15.87:0/185288596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd6c108a50 msgr2=0x7fcd6c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd6c108a50 0x7fcd6c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.975+0000 7fcd6bfff700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6c1a7720 con 0x7fcd6c102a90 2024-04-22T13:30:17.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd6bfff700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c074d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c920 tx=0x7fcd5c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c004a30 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6c1a7a10 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5c0054d0 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c007bd0 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd6c1a7f70 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.976+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcd6c0630d0 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.978+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcd5c004bd0 con 0x7fcd6c102a90 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.978+0000 7fcd697fa700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcd540a3cd0 0x7fcd540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.978+0000 7fcd6b7fe700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcd540a3cd0 0x7fcd540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.978+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcd5c0d02c0 con 0x7fcd6c102a90 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd697fa700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] conn(0x7fcd540ad4b0 0x7fcd540af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd540aff80 con 0x7fcd540ad4b0 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fcd5c0d07a0 con 0x7fcd6c102a90 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd70ac4700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] conn(0x7fcd540ad4b0 0x7fcd540af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd6b7fe700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcd540a3cd0 0x7fcd540a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000b5c0 tx=0x7fcd60005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.979+0000 7fcd70ac4700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] conn(0x7fcd540ad4b0 0x7fcd540af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:17.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.980+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== osd.9 v2:172.21.15.87:6826/2036314844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcd540aff80 con 0x7fcd540ad4b0 2024-04-22T13:30:17.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.996+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 --> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcd6c063550 con 0x7fcd540ad4b0 2024-04-22T13:30:17.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.996+0000 7fcd697fa700 1 -- 172.21.15.87:0/185288596 <== osd.9 v2:172.21.15.87:6826/2036314844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcd6c063550 con 0x7fcd540ad4b0 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] conn(0x7fcd540ad4b0 msgr2=0x7fcd540af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] conn(0x7fcd540ad4b0 0x7fcd540af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcd540a3cd0 msgr2=0x7fcd540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcd540a3cd0 0x7fcd540a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000b5c0 tx=0x7fcd60005960 comp rx=0 tx=0).stop 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 msgr2=0x7fcd6c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:17.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 --2- 172.21.15.87:0/185288596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd6c102a90 0x7fcd6c074d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c920 tx=0x7fcd5c00cc30 comp rx=0 tx=0).stop 2024-04-22T13:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd70ac4700 1 -- 172.21.15.87:0/185288596 reap_dead start 2024-04-22T13:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 shutdown_connections 2024-04-22T13:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 >> 172.21.15.87:0/185288596 conn(0x7fcd6c079ba0 msgr2=0x7fcd6c101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 shutdown_connections 2024-04-22T13:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:17.997+0000 7fcd72527700 1 -- 172.21.15.87:0/185288596 wait complete. 2024-04-22T13:30:18.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:17 smithi087 ceph-mon[27999]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:17 smithi120 ceph-mon[31101]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:18.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:17 smithi132 ceph-mon[31531]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:19.735 INFO:teuthology.orchestra.run.smithi087.stdout:287762808863 2024-04-22T13:30:19.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T13:30:19.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:19 smithi087 ceph-mon[27999]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:20.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:19 smithi120 ceph-mon[31101]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:20.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:19 smithi132 ceph-mon[31531]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:21.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:21 smithi087 ceph-mon[27999]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:22.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:21 smithi120 ceph-mon[31101]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:22.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:21 smithi132 ceph-mon[31531]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:30:24.105 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:23 smithi087 ceph-mon[27999]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:24.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:23 smithi120 ceph-mon[31101]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:24.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:23 smithi132 ceph-mon[31531]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:30:24.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.849+0000 7f18f4d90700 1 -- 172.21.15.87:0/650164914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f0107940 msgr2=0x7f18f0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.849+0000 7f18f4d90700 1 --2- 172.21.15.87:0/650164914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f0107940 0x7f18f0107d20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009990 tx=0x7f18e0009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 -- 172.21.15.87:0/650164914 shutdown_connections 2024-04-22T13:30:24.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 --2- 172.21.15.87:0/650164914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18f0102390 0x7f18f010e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 --2- 172.21.15.87:0/650164914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f0107940 0x7f18f0107d20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 --2- 172.21.15.87:0/650164914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f01019d0 0x7f18f0101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 -- 172.21.15.87:0/650164914 >> 172.21.15.87:0/650164914 conn(0x7f18f0077990 msgr2=0x7f18f0077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:24.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 -- 172.21.15.87:0/650164914 shutdown_connections 2024-04-22T13:30:24.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 -- 172.21.15.87:0/650164914 wait complete. 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 Processor -- start 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.850+0000 7f18f4d90700 1 -- start start 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 0x7f18f0199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f0102390 0x7f18f0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18f0107940 0x7f18f019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0111be0 con 0x7f18f01019d0 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0111a60 con 0x7f18f0102390 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0111d60 con 0x7f18f0107940 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 0x7f18f0199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:24.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 0x7f18f0199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36822/0 (socket says 172.21.15.87:36822) 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 -- 172.21.15.87:0/923985770 learned_addr learned my addr 172.21.15.87:0/923985770 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18edd9b700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f0102390 0x7f18f0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 -- 172.21.15.87:0/923985770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18f0107940 msgr2=0x7f18f019dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18f0107940 0x7f18f019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 -- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f0102390 msgr2=0x7f18f0199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f0102390 0x7f18f0199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f019e210 con 0x7f18f01019d0 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18edd9b700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f0102390 0x7f18f0199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18ee59c700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 0x7f18f0199250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000c00 tx=0x7f18e00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0027070 con 0x7f18f01019d0 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f019e4a0 con 0x7f18f01019d0 2024-04-22T13:30:24.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e00041d0 con 0x7f18f01019d0 2024-04-22T13:30:24.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e000f460 con 0x7f18f01019d0 2024-04-22T13:30:24.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.851+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f01a5d80 con 0x7f18f01019d0 2024-04-22T13:30:24.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.852+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18f00631d0 con 0x7f18f01019d0 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.853+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f18e002a070 con 0x7f18f01019d0 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18df7fe700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f18d80a3d50 0x7f18d80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18edd9b700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f18d80a3d50 0x7f18d80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f18e00d2020 con 0x7f18f01019d0 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18df7fe700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] conn(0x7f18d80ad530 0x7f18d80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:24.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18e00043d0 con 0x7f18d80ad530 2024-04-22T13:30:24.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.854+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f18e00d2500 con 0x7f18f01019d0 2024-04-22T13:30:24.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.855+0000 7f18edd9b700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f18d80a3d50 0x7f18d80a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18f0078140 tx=0x7f18e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:24.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.855+0000 7f18eed9d700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] conn(0x7f18d80ad530 0x7f18d80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:24.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.855+0000 7f18eed9d700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] conn(0x7f18d80ad530 0x7f18d80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:24.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.856+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== osd.10 v2:172.21.15.120:6824/2160182398 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18e00043d0 con 0x7f18d80ad530 2024-04-22T13:30:24.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.874+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 --> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f18f00635d0 con 0x7f18d80ad530 2024-04-22T13:30:24.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.874+0000 7f18df7fe700 1 -- 172.21.15.87:0/923985770 <== osd.10 v2:172.21.15.120:6824/2160182398 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f18f00635d0 con 0x7f18d80ad530 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] conn(0x7f18d80ad530 msgr2=0x7f18d80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] conn(0x7f18d80ad530 0x7f18d80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f18d80a3d50 msgr2=0x7f18d80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f18d80a3d50 0x7f18d80a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18f0078140 tx=0x7f18e4008040 comp rx=0 tx=0).stop 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 msgr2=0x7f18f0199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 --2- 172.21.15.87:0/923985770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18f01019d0 0x7f18f0199250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000c00 tx=0x7f18e00047b0 comp rx=0 tx=0).stop 2024-04-22T13:30:24.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18eed9d700 1 -- 172.21.15.87:0/923985770 reap_dead start 2024-04-22T13:30:24.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 shutdown_connections 2024-04-22T13:30:24.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 >> 172.21.15.87:0/923985770 conn(0x7f18f0077990 msgr2=0x7f18f006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:24.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 shutdown_connections 2024-04-22T13:30:24.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:24.875+0000 7f18f4d90700 1 -- 172.21.15.87:0/923985770 wait complete. 2024-04-22T13:30:25.679 INFO:teuthology.orchestra.run.smithi087.stdout:253403070500 2024-04-22T13:30:25.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T13:30:25.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:25 smithi087 ceph-mon[27999]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T13:30:26.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:25 smithi120 ceph-mon[31101]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T13:30:26.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:25 smithi132 ceph-mon[31531]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 -- 172.21.15.87:0/1230150447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 msgr2=0x7fa49c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 --2- 172.21.15.87:0/1230150447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c101dc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa490009990 tx=0x7fa490009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 -- 172.21.15.87:0/1230150447 shutdown_connections 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 --2- 172.21.15.87:0/1230150447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa49c102300 0x7fa49c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 --2- 172.21.15.87:0/1230150447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c107990 0x7fa49c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 --2- 172.21.15.87:0/1230150447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c101dc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 -- 172.21.15.87:0/1230150447 >> 172.21.15.87:0/1230150447 conn(0x7fa49c0788f0 msgr2=0x7fa49c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 -- 172.21.15.87:0/1230150447 shutdown_connections 2024-04-22T13:30:26.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.383+0000 7fa4a088f700 1 -- 172.21.15.87:0/1230150447 wait complete. 2024-04-22T13:30:26.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 Processor -- start 2024-04-22T13:30:26.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 -- start start 2024-04-22T13:30:26.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:26.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c102300 0x7fa49c103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:26.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa49c107990 0x7fa49c103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c111af0 con 0x7fa49c101940 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c111970 con 0x7fa49c102300 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa4a088f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c111c70 con 0x7fa49c107990 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c1034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36838/0 (socket says 172.21.15.87:36838) 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 -- 172.21.15.87:0/3204985820 learned_addr learned my addr 172.21.15.87:0/3204985820 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa499d9b700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c102300 0x7fa49c103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49ad9d700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa49c107990 0x7fa49c103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 -- 172.21.15.87:0/3204985820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa49c107990 msgr2=0x7fa49c103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa49c107990 0x7fa49c103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 -- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c102300 msgr2=0x7fa49c103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c102300 0x7fa49c103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.384+0000 7fa49a59c700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49c1a7690 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa49a59c700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c1034f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9a0 tx=0x7fa48c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00ca10 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa48c00cbb0 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49c1a7980 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c018680 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.385+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49c1a7e90 con 0x7fa49c101940 2024-04-22T13:30:26.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.386+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa49c0630d0 con 0x7fa49c101940 2024-04-22T13:30:26.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.387+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa48c010430 con 0x7fa49c101940 2024-04-22T13:30:26.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.387+0000 7fa48b7fe700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa4840a3cd0 0x7fa4840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:26.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa499d9b700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa4840a3cd0 0x7fa4840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:26.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa48c0d0a40 con 0x7fa49c101940 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa48b7fe700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] conn(0x7fa4840ad4b0 0x7fa4840af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa48c019ec0 con 0x7fa4840ad4b0 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa48c0d0f20 con 0x7fa49c101940 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa499d9b700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa4840a3cd0 0x7fa4840a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa49000b2e0 tx=0x7fa490005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa49ad9d700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] conn(0x7fa4840ad4b0 0x7fa4840af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.388+0000 7fa49ad9d700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] conn(0x7fa4840ad4b0 0x7fa4840af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:26.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.389+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== osd.1 v2:172.21.15.120:6800/3769829822 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa48c019ec0 con 0x7fa4840ad4b0 2024-04-22T13:30:26.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.406+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 --> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa49c0634d0 con 0x7fa4840ad4b0 2024-04-22T13:30:26.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.406+0000 7fa48b7fe700 1 -- 172.21.15.87:0/3204985820 <== osd.1 v2:172.21.15.120:6800/3769829822 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa49c0634d0 con 0x7fa4840ad4b0 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] conn(0x7fa4840ad4b0 msgr2=0x7fa4840af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] conn(0x7fa4840ad4b0 0x7fa4840af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa4840a3cd0 msgr2=0x7fa4840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa4840a3cd0 0x7fa4840a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa49000b2e0 tx=0x7fa490005960 comp rx=0 tx=0).stop 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 msgr2=0x7fa49c1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 --2- 172.21.15.87:0/3204985820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa49c101940 0x7fa49c1034f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9a0 tx=0x7fa48c00ecb0 comp rx=0 tx=0).stop 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa49ad9d700 1 -- 172.21.15.87:0/3204985820 reap_dead start 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 shutdown_connections 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 >> 172.21.15.87:0/3204985820 conn(0x7fa49c0788f0 msgr2=0x7fa49c06ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 shutdown_connections 2024-04-22T13:30:26.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:26.407+0000 7fa4a088f700 1 -- 172.21.15.87:0/3204985820 wait complete. 2024-04-22T13:30:26.829 INFO:teuthology.orchestra.run.smithi087.stdout:197568495662 2024-04-22T13:30:26.829 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T13:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 -- 172.21.15.87:0/3461189723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 msgr2=0x7f8738101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3461189723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738101df0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f87280099c0 tx=0x7f8728009cd0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 -- 172.21.15.87:0/3461189723 shutdown_connections 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3461189723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8738102330 0x7f873810e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3461189723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87381079c0 0x7f8738107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.031+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3461189723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738101df0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.032+0000 7f873f9fd700 1 -- 172.21.15.87:0/3461189723 >> 172.21.15.87:0/3461189723 conn(0x7f8738078980 msgr2=0x7f8738078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.032+0000 7f873f9fd700 1 -- 172.21.15.87:0/3461189723 shutdown_connections 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.032+0000 7f873f9fd700 1 -- 172.21.15.87:0/3461189723 wait complete. 2024-04-22T13:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.032+0000 7f873f9fd700 1 Processor -- start 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.032+0000 7f873f9fd700 1 -- start start 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8738102330 0x7f8738198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87381079c0 0x7f873819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8738111bf0 con 0x7f8738101970 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8738111a70 con 0x7f87381079c0 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873f9fd700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8738111d70 con 0x7f8738102330 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36864/0 (socket says 172.21.15.87:36864) 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 -- 172.21.15.87:0/3775896760 learned_addr learned my addr 172.21.15.87:0/3775896760 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873cf98700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8738102330 0x7f8738198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873df9a700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87381079c0 0x7f873819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 -- 172.21.15.87:0/3775896760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8738102330 msgr2=0x7f8738198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8738102330 0x7f8738198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 -- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87381079c0 msgr2=0x7f873819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87381079c0 0x7f873819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873819d0c0 con 0x7f8738101970 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.033+0000 7f873d799700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738198100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f873400b530 tx=0x7f873400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.034+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734015020 con 0x7f8738101970 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.034+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873819d3b0 con 0x7f8738101970 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.034+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8734011ce0 con 0x7f8738101970 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.034+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734010dd0 con 0x7f8738101970 2024-04-22T13:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.034+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87381a4c30 con 0x7f8738101970 2024-04-22T13:30:27.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.035+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8738063190 con 0x7f8738101970 2024-04-22T13:30:27.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.036+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8734018020 con 0x7f8738101970 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f872e7fc700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f87240a3cd0 0x7f87240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f87340cf990 con 0x7f8738101970 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f873cf98700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f87240a3cd0 0x7f87240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f872e7fc700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] conn(0x7f87240ad4b0 0x7f87240af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87240aff80 con 0x7f87240ad4b0 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.037+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f87340d42c0 con 0x7f8738101970 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.038+0000 7f873df9a700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] conn(0x7f87240ad4b0 0x7f87240af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.038+0000 7f873cf98700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f87240a3cd0 0x7f87240a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8728009990 tx=0x7f872800b870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.038+0000 7f873df9a700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] conn(0x7f87240ad4b0 0x7f87240af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.039+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== osd.7 v2:172.21.15.120:6816/1140009169 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87240aff80 con 0x7f87240ad4b0 2024-04-22T13:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.056+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 --> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8738063610 con 0x7f87240ad4b0 2024-04-22T13:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.056+0000 7f872e7fc700 1 -- 172.21.15.87:0/3775896760 <== osd.7 v2:172.21.15.120:6816/1140009169 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8738063610 con 0x7f87240ad4b0 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] conn(0x7f87240ad4b0 msgr2=0x7f87240af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] conn(0x7f87240ad4b0 0x7f87240af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f87240a3cd0 msgr2=0x7f87240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f87240a3cd0 0x7f87240a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8728009990 tx=0x7f872800b870 comp rx=0 tx=0).stop 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 msgr2=0x7f8738198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 --2- 172.21.15.87:0/3775896760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8738101970 0x7f8738198100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f873400b530 tx=0x7f873400b840 comp rx=0 tx=0).stop 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873df9a700 1 -- 172.21.15.87:0/3775896760 reap_dead start 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 shutdown_connections 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 >> 172.21.15.87:0/3775896760 conn(0x7f8738078980 msgr2=0x7f87380ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 shutdown_connections 2024-04-22T13:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:27.057+0000 7f873f9fd700 1 -- 172.21.15.87:0/3775896760 wait complete. 2024-04-22T13:30:27.934 INFO:teuthology.orchestra.run.smithi087.stdout:219043332138 2024-04-22T13:30:27.934 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T13:30:28.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:27 smithi087 ceph-mon[27999]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-22T13:30:28.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:27 smithi120 ceph-mon[31101]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-22T13:30:28.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:27 smithi132 ceph-mon[31531]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-22T13:30:29.070 INFO:teuthology.orchestra.run.smithi087.stdout:55834574920 2024-04-22T13:30:29.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/1208045464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 msgr2=0x7f9e84101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/1208045464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84101df0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800b0a0 tx=0x7f9e7800b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/1208045464 shutdown_connections 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/1208045464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e84102330 0x7f9e8410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/1208045464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e84107920 0x7f9e84107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/1208045464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84101df0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/1208045464 >> 172.21.15.87:0/1208045464 conn(0x7f9e84078910 msgr2=0x7f9e84078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/1208045464 shutdown_connections 2024-04-22T13:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.341+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/1208045464 wait complete. 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 Processor -- start 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 -- start start 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e84102330 0x7f9e84199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e84107920 0x7f9e8419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84111b60 con 0x7f9e84101970 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841119e0 con 0x7f9e84107920 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8bd1d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84111ce0 con 0x7f9e84102330 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36876/0 (socket says 172.21.15.87:36876) 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 -- 172.21.15.87:0/3164304068 learned_addr learned my addr 172.21.15.87:0/3164304068 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e8a2ba700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e84107920 0x7f9e8419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:29.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e892b8700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e84102330 0x7f9e84199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 -- 172.21.15.87:0/3164304068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e84102330 msgr2=0x7f9e84199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e84102330 0x7f9e84199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 -- 172.21.15.87:0/3164304068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e84107920 msgr2=0x7f9e8419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e84107920 0x7f9e8419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.342+0000 7f9e89ab9700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8419e160 con 0x7f9e84101970 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e89ab9700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84199150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400b530 tx=0x7f9e7400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74015020 con 0x7f9e84101970 2024-04-22T13:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e74011ce0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8419e450 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74010dd0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e841a5cd0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.343+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e840631d0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.345+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9e74018020 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.345+0000 7f9e82ffd700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9e6c0a3cd0 0x7f9e6c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.345+0000 7f9e892b8700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9e6c0a3cd0 0x7f9e6c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9e740d00f0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e82ffd700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] conn(0x7f9e6c0ad4b0 0x7f9e6c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e74018ec0 con 0x7f9e6c0ad4b0 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f9e740d05d0 con 0x7f9e84101970 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e8a2ba700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] conn(0x7f9e6c0ad4b0 0x7f9e6c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e892b8700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9e6c0a3cd0 0x7f9e6c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e84078f60 tx=0x7f9e7800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.346+0000 7f9e8a2ba700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] conn(0x7f9e6c0ad4b0 0x7f9e6c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.347+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== osd.0 v2:172.21.15.87:6802/4104861709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e74018ec0 con 0x7f9e6c0ad4b0 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.363+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 --> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9e84063650 con 0x7f9e6c0ad4b0 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e82ffd700 1 -- 172.21.15.87:0/3164304068 <== osd.0 v2:172.21.15.87:6802/4104861709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9e84063650 con 0x7f9e6c0ad4b0 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] conn(0x7f9e6c0ad4b0 msgr2=0x7f9e6c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] conn(0x7f9e6c0ad4b0 0x7f9e6c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9e6c0a3cd0 msgr2=0x7f9e6c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9e6c0a3cd0 0x7f9e6c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e84078f60 tx=0x7f9e7800b7b0 comp rx=0 tx=0).stop 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 msgr2=0x7f9e84199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 --2- 172.21.15.87:0/3164304068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e84101970 0x7f9e84199150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400b530 tx=0x7f9e7400b840 comp rx=0 tx=0).stop 2024-04-22T13:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8a2ba700 1 -- 172.21.15.87:0/3164304068 reap_dead start 2024-04-22T13:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 shutdown_connections 2024-04-22T13:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 >> 172.21.15.87:0/3164304068 conn(0x7f9e84078910 msgr2=0x7f9e8406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 shutdown_connections 2024-04-22T13:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.364+0000 7f9e8bd1d700 1 -- 172.21.15.87:0/3164304068 wait complete. 2024-04-22T13:30:30.158 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:29 smithi087 ceph-mon[27999]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:30:30.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:29 smithi120 ceph-mon[31101]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:30:30.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:29 smithi132 ceph-mon[31531]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:30:30.484 INFO:teuthology.orchestra.run.smithi087.stdout:309237645338 2024-04-22T13:30:30.484 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.938+0000 7f3d6535a700 1 -- 172.21.15.87:0/262203965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 msgr2=0x7f3d6010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.938+0000 7f3d6535a700 1 --2- 172.21.15.87:0/262203965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d6010f8e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009990 tx=0x7f3d58009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.938+0000 7f3d6535a700 1 -- 172.21.15.87:0/262203965 shutdown_connections 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 --2- 172.21.15.87:0/262203965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d6010f8e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 --2- 172.21.15.87:0/262203965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60108a10 0x7f3d60108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 --2- 172.21.15.87:0/262203965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d60102b00 0x7f3d60102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 -- 172.21.15.87:0/262203965 >> 172.21.15.87:0/262203965 conn(0x7f3d60079c00 msgr2=0x7f3d6007a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 -- 172.21.15.87:0/262203965 shutdown_connections 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 -- 172.21.15.87:0/262203965 wait complete. 2024-04-22T13:30:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 Processor -- start 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.939+0000 7f3d6535a700 1 -- start start 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d60102b00 0x7f3d60199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d60199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60108a10 0x7f3d6019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3d60112c50 con 0x7f3d601034c0 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d60112ad0 con 0x7f3d60108a10 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d60112dd0 con 0x7f3d60102b00 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d60199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d60199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41224/0 (socket says 172.21.15.87:41224) 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 -- 172.21.15.87:0/1881551537 learned_addr learned my addr 172.21.15.87:0/1881551537 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5f7fe700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60108a10 0x7f3d6019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5effd700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d60102b00 0x7f3d60199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:30.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 -- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d60102b00 msgr2=0x7f3d60199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d60102b00 0x7f3d60199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 -- 172.21.15.87:0/1881551537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60108a10 msgr2=0x7f3d6019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60108a10 0x7f3d6019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6019e0e0 con 0x7f3d601034c0 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d5e7fc700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d60199660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b530 tx=0x7f3d4800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48015020 con 0x7f3d601034c0 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6019e3d0 con 0x7f3d601034c0 2024-04-22T13:30:30.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d48011ce0 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48010dd0 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.940+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d601a5c50 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.941+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d600631d0 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.942+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3d48018020 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d57fff700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3d4c0a3cd0 0x7f3d4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d5effd700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3d4c0a3cd0 0x7f3d4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3d480d00f0 con 0x7f3d601034c0 2024-04-22T13:30:30.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d57fff700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] conn(0x7f3d4c0ad4b0 0x7f3d4c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d48018ec0 con 0x7f3d4c0ad4b0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f3d480d05d0 con 0x7f3d601034c0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.943+0000 7f3d5effd700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3d4c0a3cd0 0x7f3d4c0a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009550 tx=0x7f3d500091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.944+0000 7f3d5f7fe700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] conn(0x7f3d4c0ad4b0 0x7f3d4c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.944+0000 7f3d5f7fe700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] conn(0x7f3d4c0ad4b0 0x7f3d4c0af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.945+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== osd.11 v2:172.21.15.132:6824/3735146249 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d48018ec0 con 0x7f3d4c0ad4b0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.961+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 --> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d60063650 con 0x7f3d4c0ad4b0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d57fff700 1 -- 172.21.15.87:0/1881551537 <== osd.11 v2:172.21.15.132:6824/3735146249 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3d60063650 con 0x7f3d4c0ad4b0 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] conn(0x7f3d4c0ad4b0 msgr2=0x7f3d4c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] conn(0x7f3d4c0ad4b0 0x7f3d4c0af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3d4c0a3cd0 msgr2=0x7f3d4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3d4c0a3cd0 0x7f3d4c0a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009550 tx=0x7f3d500091b0 comp rx=0 tx=0).stop 2024-04-22T13:30:30.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 msgr2=0x7f3d60199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:30.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 --2- 172.21.15.87:0/1881551537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d601034c0 0x7f3d60199660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b530 tx=0x7f3d4800b840 comp rx=0 tx=0).stop 2024-04-22T13:30:30.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d5f7fe700 1 -- 172.21.15.87:0/1881551537 reap_dead start 2024-04-22T13:30:30.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 shutdown_connections 2024-04-22T13:30:30.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 >> 172.21.15.87:0/1881551537 conn(0x7f3d60079c00 msgr2=0x7f3d60100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:30.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 shutdown_connections 2024-04-22T13:30:30.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:29.962+0000 7f3d6535a700 1 -- 172.21.15.87:0/1881551537 wait complete. 2024-04-22T13:30:31.487 INFO:teuthology.orchestra.run.smithi087.stdout:343597383703 2024-04-22T13:30:31.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T13:30:31.938 INFO:teuthology.orchestra.run.smithi087.stdout:34359738446 2024-04-22T13:30:31.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 -- 172.21.15.87:0/588593390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43441019d0 msgr2=0x7f4344101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 --2- 172.21.15.87:0/588593390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43441019d0 0x7f4344101e50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4334009990 tx=0x7f4334009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 -- 172.21.15.87:0/588593390 shutdown_connections 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 --2- 172.21.15.87:0/588593390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4344102390 0x7f434410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 --2- 172.21.15.87:0/588593390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4344107940 0x7f4344107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 --2- 172.21.15.87:0/588593390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43441019d0 0x7f4344101e50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 -- 172.21.15.87:0/588593390 >> 172.21.15.87:0/588593390 conn(0x7f4344077990 msgr2=0x7f4344077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 -- 172.21.15.87:0/588593390 shutdown_connections 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.451+0000 7f434c3ae700 1 -- 172.21.15.87:0/588593390 wait complete. 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 Processor -- start 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 -- start start 2024-04-22T13:30:32.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43441019d0 0x7f4344103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 0x7f4344103a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4344107940 0x7f4344103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4344111b50 con 0x7f4344102390 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43441119d0 con 0x7f43441019d0 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434c3ae700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4344111cd0 con 0x7f4344107940 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f4349949700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 0x7f4344103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f4349949700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 0x7f4344103a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36888/0 (socket says 172.21.15.87:36888) 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f4349949700 1 -- 172.21.15.87:0/3396473470 learned_addr learned my addr 172.21.15.87:0/3396473470 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434a94b700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4344107940 0x7f4344103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.452+0000 7f434a14a700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43441019d0 0x7f4344103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 -- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4344107940 msgr2=0x7f4344103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4344107940 0x7f4344103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 -- 172.21.15.87:0/3396473470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43441019d0 msgr2=0x7f4344103510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43441019d0 0x7f4344103510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43441a7700 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f4349949700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 0x7f4344103a50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f433400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334027070 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4334022470 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43441a7990 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334017650 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.453+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43441a7de0 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.454+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f43440630d0 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.455+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f433402a070 con 0x7f4344102390 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.455+0000 7f433b7fe700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f43300a3cd0 0x7f43300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f434a14a700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f43300a3cd0 0x7f43300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f43340d1a10 con 0x7f4344102390 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f433b7fe700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] conn(0x7f43300ad4b0 0x7f43300af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4334009030 con 0x7f43300ad4b0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f43340d1ef0 con 0x7f4344102390 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f434a14a700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f43300a3cd0 0x7f43300a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4340009bd0 tx=0x7f4340009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f434a94b700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] conn(0x7f43300ad4b0 0x7f43300af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.456+0000 7f434a94b700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] conn(0x7f43300ad4b0 0x7f43300af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.458+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== osd.2 v2:172.21.15.132:6800/2448340418 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4334009030 con 0x7f43300ad4b0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.473+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 --> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f43440634d0 con 0x7f43300ad4b0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.474+0000 7f433b7fe700 1 -- 172.21.15.87:0/3396473470 <== osd.2 v2:172.21.15.132:6800/2448340418 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f43440634d0 con 0x7f43300ad4b0 2024-04-22T13:30:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.474+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] conn(0x7f43300ad4b0 msgr2=0x7f43300af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] conn(0x7f43300ad4b0 0x7f43300af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f43300a3cd0 msgr2=0x7f43300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f43300a3cd0 0x7f43300a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4340009bd0 tx=0x7f4340009260 comp rx=0 tx=0).stop 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 msgr2=0x7f4344103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 --2- 172.21.15.87:0/3396473470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4344102390 0x7f4344103a50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f433400bfd0 comp rx=0 tx=0).stop 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434a94b700 1 -- 172.21.15.87:0/3396473470 reap_dead start 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 shutdown_connections 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 >> 172.21.15.87:0/3396473470 conn(0x7f4344077990 msgr2=0x7f434406b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 shutdown_connections 2024-04-22T13:30:32.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:28.475+0000 7f434c3ae700 1 -- 172.21.15.87:0/3396473470 wait complete. 2024-04-22T13:30:32.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:31 smithi120 ceph-mon[31101]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:32.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:31 smithi132 ceph-mon[31531]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:32.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:31 smithi087 ceph-mon[27999]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:32.732 INFO:teuthology.orchestra.run.smithi087.stdout:77309411397 2024-04-22T13:30:32.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/683327976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081022a0 msgr2=0x7f6e0810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/683327976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081022a0 0x7f6e0810e7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b0a0 tx=0x7f6e0000b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/683327976 shutdown_connections 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/683327976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081022a0 0x7f6e0810e7b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/683327976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e081078e0 0x7f6e08107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/683327976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e081018e0 0x7f6e08101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/683327976 >> 172.21.15.87:0/683327976 conn(0x7f6e080fd3f0 msgr2=0x7f6e080ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:33.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/683327976 shutdown_connections 2024-04-22T13:30:33.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/683327976 wait complete. 2024-04-22T13:30:33.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 Processor -- start 2024-04-22T13:30:33.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.844+0000 7f6e0d3a1700 1 -- start start 2024-04-22T13:30:33.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 0x7f6e08199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:33.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e081022a0 0x7f6e08199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:33.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e081078e0 0x7f6e0819d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:33.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08112c00 con 0x7f6e081018e0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08112a80 con 0x7f6e081078e0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0d3a1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08112d80 con 0x7f6e081022a0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 0x7f6e08199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 0x7f6e08199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41244/0 (socket says 172.21.15.87:41244) 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 -- 172.21.15.87:0/2536441645 learned_addr learned my addr 172.21.15.87:0/2536441645 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e0cba0700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e081078e0 0x7f6e0819d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e077fe700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e081022a0 0x7f6e08199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 -- 172.21.15.87:0/2536441645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e081022a0 msgr2=0x7f6e08199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e081022a0 0x7f6e08199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 -- 172.21.15.87:0/2536441645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e081078e0 msgr2=0x7f6e0819d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e081078e0 0x7f6e0819d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0819e060 con 0x7f6e081018e0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.845+0000 7f6e07fff700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 0x7f6e08199050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b530 tx=0x7f6df800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:33.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8015020 con 0x7f6e081018e0 2024-04-22T13:30:33.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0819e350 con 0x7f6e081018e0 2024-04-22T13:30:33.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df8011ce0 con 0x7f6e081018e0 2024-04-22T13:30:33.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8010dd0 con 0x7f6e081018e0 2024-04-22T13:30:33.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a5bd0 con 0x7f6e081018e0 2024-04-22T13:30:33.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.846+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e08063150 con 0x7f6e081018e0 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.848+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6df8018020 con 0x7f6e081018e0 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.848+0000 7f6e057fa700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6df00a3cd0 0x7f6df00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.848+0000 7f6e077fe700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6df00a3cd0 0x7f6df00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.848+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6df80d00f0 con 0x7f6e081018e0 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e057fa700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] conn(0x7f6df00ad4b0 0x7f6df00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6df8018ec0 con 0x7f6df00ad4b0 2024-04-22T13:30:33.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f6df80d05d0 con 0x7f6e081018e0 2024-04-22T13:30:33.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e0cba0700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] conn(0x7f6df00ad4b0 0x7f6df00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:33.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e077fe700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6df00a3cd0 0x7f6df00a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6e08106f00 tx=0x7f6dfc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:33.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.849+0000 7f6e0cba0700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] conn(0x7f6df00ad4b0 0x7f6df00af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:33.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.850+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== osd.3 v2:172.21.15.87:6810/2474923327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6df8018ec0 con 0x7f6df00ad4b0 2024-04-22T13:30:33.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 --> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e080635d0 con 0x7f6df00ad4b0 2024-04-22T13:30:33.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e057fa700 1 -- 172.21.15.87:0/2536441645 <== osd.3 v2:172.21.15.87:6810/2474923327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e080635d0 con 0x7f6df00ad4b0 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] conn(0x7f6df00ad4b0 msgr2=0x7f6df00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] conn(0x7f6df00ad4b0 0x7f6df00af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6df00a3cd0 msgr2=0x7f6df00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f6df00a3cd0 0x7f6df00a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6e08106f00 tx=0x7f6dfc00b410 comp rx=0 tx=0).stop 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 msgr2=0x7f6e08199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0d3a1700 1 --2- 172.21.15.87:0/2536441645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e081018e0 0x7f6e08199050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6df800b530 tx=0x7f6df800b840 comp rx=0 tx=0).stop 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.866+0000 7f6e0cba0700 1 -- 172.21.15.87:0/2536441645 reap_dead start 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.867+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 shutdown_connections 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.867+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 >> 172.21.15.87:0/2536441645 conn(0x7f6e080fd3f0 msgr2=0x7f6e080fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.867+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 shutdown_connections 2024-04-22T13:30:33.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:33.867+0000 7f6e0d3a1700 1 -- 172.21.15.87:0/2536441645 wait complete. 2024-04-22T13:30:34.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:33 smithi120 ceph-mon[31101]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:34.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:33 smithi132 ceph-mon[31531]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:34.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:33 smithi087 ceph-mon[27999]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:34.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 -- 172.21.15.87:0/980716487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1304107940 msgr2=0x7f1304107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 --2- 172.21.15.87:0/980716487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1304107940 0x7f1304107d20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f12f400b0d0 tx=0x7f12f400b3e0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 -- 172.21.15.87:0/980716487 shutdown_connections 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 --2- 172.21.15.87:0/980716487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13041034b0 0x7f130410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 --2- 172.21.15.87:0/980716487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1304107940 0x7f1304107d20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 --2- 172.21.15.87:0/980716487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304102af0 0x7f1304102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 -- 172.21.15.87:0/980716487 >> 172.21.15.87:0/980716487 conn(0x7f1304079ba0 msgr2=0x7f1304079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 -- 172.21.15.87:0/980716487 shutdown_connections 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 -- 172.21.15.87:0/980716487 wait complete. 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.814+0000 7f130900e700 1 Processor -- start 2024-04-22T13:30:34.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 -- start start 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1304102af0 0x7f130419b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 0x7f130419b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304107940 0x7f130419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1304112d30 con 0x7f13041034b0 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1304112bb0 con 0x7f1304102af0 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130900e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1304112eb0 con 0x7f1304107940 2024-04-22T13:30:34.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 0x7f130419b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 0x7f130419b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41268/0 (socket says 172.21.15.87:41268) 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f130259c700 1 -- 172.21.15.87:0/291811244 learned_addr learned my addr 172.21.15.87:0/291811244 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.815+0000 7f1302d9d700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1304102af0 0x7f130419b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130359e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304107940 0x7f130419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 -- 172.21.15.87:0/291811244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304107940 msgr2=0x7f130419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304107940 0x7f130419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 -- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1304102af0 msgr2=0x7f130419b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1304102af0 0x7f130419b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13041a0490 con 0x7f13041034b0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130359e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1304107940 0x7f130419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130259c700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 0x7f130419b9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00b530 tx=0x7f12ec00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec015020 con 0x7f13041034b0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12ec011ce0 con 0x7f13041034b0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec010e40 con 0x7f13041034b0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.816+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13041a0780 con 0x7f13041034b0 2024-04-22T13:30:34.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.817+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13041a8000 con 0x7f13041034b0 2024-04-22T13:30:34.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.818+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13041013c0 con 0x7f13041034b0 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.820+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f12ec018020 con 0x7f13041034b0 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.821+0000 7f12fbfff700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12f00a3ea0 0x7f12f00a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.821+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f12ec0d02b0 con 0x7f13041034b0 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.821+0000 7f12fbfff700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] conn(0x7f12f00ad680 0x7f12f00afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.821+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12f00b0150 con 0x7f12f00ad680 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.821+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f12ec0d06a0 con 0x7f13041034b0 2024-04-22T13:30:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.822+0000 7f130359e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] conn(0x7f12f00ad680 0x7f12f00afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.822+0000 7f130359e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] conn(0x7f12f00ad680 0x7f12f00afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.823+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== osd.4 v2:172.21.15.120:6808/172450130 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12f00b0150 con 0x7f12f00ad680 2024-04-22T13:30:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.825+0000 7f1302d9d700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12f00a3ea0 0x7f12f00a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.825+0000 7f1302d9d700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12f00a3ea0 0x7f12f00a6360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009fd0 tx=0x7f12f400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.856+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 --> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13041041b0 con 0x7f12f00ad680 2024-04-22T13:30:34.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.857+0000 7f12fbfff700 1 -- 172.21.15.87:0/291811244 <== osd.4 v2:172.21.15.120:6808/172450130 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f13041041b0 con 0x7f12f00ad680 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.859+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] conn(0x7f12f00ad680 msgr2=0x7f12f00afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.859+0000 7f130900e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] conn(0x7f12f00ad680 0x7f12f00afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12f00a3ea0 msgr2=0x7f12f00a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130900e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f12f00a3ea0 0x7f12f00a6360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009fd0 tx=0x7f12f400b040 comp rx=0 tx=0).stop 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 msgr2=0x7f130419b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130900e700 1 --2- 172.21.15.87:0/291811244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13041034b0 0x7f130419b9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00b530 tx=0x7f12ec00b840 comp rx=0 tx=0).stop 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130359e700 1 -- 172.21.15.87:0/291811244 reap_dead start 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.860+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 shutdown_connections 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.861+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 >> 172.21.15.87:0/291811244 conn(0x7f1304079ba0 msgr2=0x7f1304100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.861+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 shutdown_connections 2024-04-22T13:30:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:34.861+0000 7f130900e700 1 -- 172.21.15.87:0/291811244 wait complete. 2024-04-22T13:30:35.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 -- 172.21.15.87:0/15073025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a241023a0 msgr2=0x7f3a24102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 --2- 172.21.15.87:0/15073025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a241023a0 0x7f3a24102780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400b0d0 tx=0x7f3a1400b3e0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 -- 172.21.15.87:0/15073025 shutdown_connections 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 --2- 172.21.15.87:0/15073025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 0x7f3a2410e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 --2- 172.21.15.87:0/15073025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a241023a0 0x7f3a24102780 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 --2- 172.21.15.87:0/15073025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a24102d50 0x7f3a241065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 -- 172.21.15.87:0/15073025 >> 172.21.15.87:0/15073025 conn(0x7f3a24077fb0 msgr2=0x7f3a240ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 -- 172.21.15.87:0/15073025 shutdown_connections 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 -- 172.21.15.87:0/15073025 wait complete. 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.597+0000 7f3a29688700 1 Processor -- start 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 -- start start 2024-04-22T13:30:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a241023a0 0x7f3a240fd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 0x7f3a240fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3a24111c40 con 0x7f3a24102d50 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a24111ac0 con 0x7f3a24106cc0 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a29688700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a24111dc0 con 0x7f3a241023a0 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41296/0 (socket says 172.21.15.87:41296) 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 -- 172.21.15.87:0/190502425 learned_addr learned my addr 172.21.15.87:0/190502425 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a22ffd700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a241023a0 0x7f3a240fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a237fe700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 0x7f3a240fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 -- 172.21.15.87:0/190502425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a241023a0 msgr2=0x7f3a240fd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a241023a0 0x7f3a240fd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 -- 172.21.15.87:0/190502425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 msgr2=0x7f3a240fdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 0x7f3a240fdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 -- 172.21.15.87:0/190502425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a241a77b0 con 0x7f3a24102d50 2024-04-22T13:30:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.598+0000 7f3a227fc700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00c970 tx=0x7f3a0c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.599+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004a30 con 0x7f3a24102d50 2024-04-22T13:30:35.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.599+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a241a7aa0 con 0x7f3a24102d50 2024-04-22T13:30:35.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.599+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c0054d0 con 0x7f3a24102d50 2024-04-22T13:30:35.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.599+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c007c10 con 0x7f3a24102d50 2024-04-22T13:30:35.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.599+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a241a7fe0 con 0x7f3a24102d50 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.600+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a24069730 con 0x7f3a24102d50 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.601+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a0c004bd0 con 0x7f3a24102d50 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.601+0000 7f3a1bfff700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 0x7f3a100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.602+0000 7f3a22ffd700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 0x7f3a100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.602+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3a0c0d0300 con 0x7f3a24102d50 2024-04-22T13:30:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.602+0000 7f3a22ffd700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 0x7f3a100a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009fd0 tx=0x7f3a1400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.606+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a0c014030 con 0x7f3a24102d50 2024-04-22T13:30:35.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.812+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f3a241a8210 con 0x7f3a24102d50 2024-04-22T13:30:35.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.813+0000 7f3a1bfff700 1 -- 172.21.15.87:0/190502425 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3a0c09cd30 con 0x7f3a24102d50 2024-04-22T13:30:35.809 INFO:teuthology.orchestra.run.smithi087.stdout:287762808866 2024-04-22T13:30:35.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 msgr2=0x7f3a100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 0x7f3a100a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009fd0 tx=0x7f3a1400b040 comp rx=0 tx=0).stop 2024-04-22T13:30:35.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 msgr2=0x7f3a240fd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00c970 tx=0x7f3a0c00cc80 comp rx=0 tx=0).stop 2024-04-22T13:30:35.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 shutdown_connections 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f3a100a3cd0 0x7f3a100a6190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a24106cc0 0x7f3a240fdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a241023a0 0x7f3a240fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 --2- 172.21.15.87:0/190502425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a24102d50 0x7f3a240fd960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 >> 172.21.15.87:0/190502425 conn(0x7f3a24077fb0 msgr2=0x7f3a24101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:35.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 shutdown_connections 2024-04-22T13:30:35.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.815+0000 7f3a29688700 1 -- 172.21.15.87:0/190502425 wait complete. 2024-04-22T13:30:35.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 -- 172.21.15.87:0/571152225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708101940 msgr2=0x7f2708101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 --2- 172.21.15.87:0/571152225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708101940 0x7f2708101dc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009990 tx=0x7f26fc009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 -- 172.21.15.87:0/571152225 shutdown_connections 2024-04-22T13:30:35.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 --2- 172.21.15.87:0/571152225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2708102300 0x7f270810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 --2- 172.21.15.87:0/571152225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 0x7f2708107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 --2- 172.21.15.87:0/571152225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708101940 0x7f2708101dc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 -- 172.21.15.87:0/571152225 >> 172.21.15.87:0/571152225 conn(0x7f27080788f0 msgr2=0x7f2708078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:35.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 -- 172.21.15.87:0/571152225 shutdown_connections 2024-04-22T13:30:35.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.843+0000 7f270d0b9700 1 -- 172.21.15.87:0/571152225 wait complete. 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.844+0000 7f270d0b9700 1 Processor -- start 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.844+0000 7f270d0b9700 1 -- start start 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2708101940 0x7f2708199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 0x7f270819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 0x7f27081a5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 0x7f270819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2708111b60 con 0x7f2708102300 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f27081119e0 con 0x7f2708101940 2024-04-22T13:30:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270d0b9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2708111ce0 con 0x7f2708107990 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 0x7f270819a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41308/0 (socket says 172.21.15.87:41308) 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 -- 172.21.15.87:0/3611802428 learned_addr learned my addr 172.21.15.87:0/3611802428 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f2706d9d700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2708101940 0x7f2708199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270759e700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 0x7f27081a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 -- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 msgr2=0x7f27081a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 0x7f27081a5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 -- 172.21.15.87:0/3611802428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2708101940 msgr2=0x7f2708199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2708101940 0x7f2708199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270659c700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27081a55a0 con 0x7f2708102300 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.845+0000 7f270759e700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2708107990 0x7f27081a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f270659c700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 0x7f270819a520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26fc000c00 tx=0x7f26fc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc027070 con 0x7f2708102300 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc0040e0 con 0x7f2708102300 2024-04-22T13:30:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27081a5770 con 0x7f2708102300 2024-04-22T13:30:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc00f460 con 0x7f2708102300 2024-04-22T13:30:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.846+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27081a5b60 con 0x7f2708102300 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.847+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2708063110 con 0x7f2708102300 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.848+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f26fc02a070 con 0x7f2708102300 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.848+0000 7f26effff700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26f00a3d50 0x7f26f00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f2706d9d700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26f00a3d50 0x7f26f00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f26fc0d19f0 con 0x7f2708102300 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f26effff700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] conn(0x7f26f00ad530 0x7f26f00af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26fc009030 con 0x7f26f00ad530 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f26fc0d1ed0 con 0x7f2708102300 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f2706d9d700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26f00a3d50 0x7f26f00a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009a10 tx=0x7f26f8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.849+0000 7f270759e700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] conn(0x7f26f00ad530 0x7f26f00af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.850+0000 7f270759e700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] conn(0x7f26f00ad530 0x7f26f00af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.850+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== osd.5 v2:172.21.15.132:6808/1166176702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f26fc009030 con 0x7f26f00ad530 2024-04-22T13:30:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.866+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 --> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2708063540 con 0x7f26f00ad530 2024-04-22T13:30:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f26effff700 1 -- 172.21.15.87:0/3611802428 <== osd.5 v2:172.21.15.132:6808/1166176702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2708063540 con 0x7f26f00ad530 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] conn(0x7f26f00ad530 msgr2=0x7f26f00af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] conn(0x7f26f00ad530 0x7f26f00af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26f00a3d50 msgr2=0x7f26f00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f26f00a3d50 0x7f26f00a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009a10 tx=0x7f26f8009190 comp rx=0 tx=0).stop 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 msgr2=0x7f270819a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 --2- 172.21.15.87:0/3611802428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2708102300 0x7f270819a520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26fc000c00 tx=0x7f26fc0047b0 comp rx=0 tx=0).stop 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270759e700 1 -- 172.21.15.87:0/3611802428 reap_dead start 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 shutdown_connections 2024-04-22T13:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.867+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 >> 172.21.15.87:0/3611802428 conn(0x7f27080788f0 msgr2=0x7f27080ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.868+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 shutdown_connections 2024-04-22T13:30:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:35.868+0000 7f270d0b9700 1 -- 172.21.15.87:0/3611802428 wait complete. 2024-04-22T13:30:36.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:35 smithi120 ceph-mon[31101]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:35 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/190502425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T13:30:36.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:35 smithi087 ceph-mon[27999]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:36.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:35 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/190502425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T13:30:36.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:35 smithi132 ceph-mon[31531]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:36.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:35 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/190502425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T13:30:36.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808863 got 287762808866 for osd.9 2024-04-22T13:30:36.555 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T13:30:37.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:36 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-22T13:30:37.205 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T13:30:37.205 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T13:30:37.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T13:30:37.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-22T13:30:37.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T13:30:37.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:36 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T13:30:37.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:36 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-22T13:30:37.880 INFO:teuthology.orchestra.run.smithi087.stdout:141733920828 2024-04-22T13:30:37.881 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T13:30:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-22T13:30:38.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:37 smithi120 ceph-mon[31101]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-22T13:30:38.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T13:30:38.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-22T13:30:38.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:37 smithi132 ceph-mon[31531]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T13:30:38.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-22T13:30:38.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:37 smithi087 ceph-mon[27999]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T13:30:38.311 INFO:teuthology.orchestra.run.smithi087.stdout:107374182464 2024-04-22T13:30:38.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T13:30:38.973 INFO:teuthology.orchestra.run.smithi087.stdout:163208757303 2024-04-22T13:30:38.973 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T13:30:39.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:38 smithi120 ceph-mon[31101]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T13:30:39.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:38 smithi132 ceph-mon[31531]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T13:30:39.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:38 smithi087 ceph-mon[27999]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T13:30:39.452 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:39.912 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:39 smithi087 ceph-mon[27999]: pgmap v318: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:40.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:39 smithi120 ceph-mon[31101]: pgmap v318: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:40.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:39 smithi132 ceph-mon[31531]: pgmap v318: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 -- 172.21.15.87:0/235303914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c101030 msgr2=0x7fba4c101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 --2- 172.21.15.87:0/235303914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c101030 0x7fba4c101490 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fba48009990 tx=0x7fba48009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 -- 172.21.15.87:0/235303914 shutdown_connections 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 --2- 172.21.15.87:0/235303914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 --2- 172.21.15.87:0/235303914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c108ad0 0x7fba4c108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 --2- 172.21.15.87:0/235303914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c101030 0x7fba4c101490 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 -- 172.21.15.87:0/235303914 >> 172.21.15.87:0/235303914 conn(0x7fba4c077fb0 msgr2=0x7fba4c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 -- 172.21.15.87:0/235303914 shutdown_connections 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.872+0000 7fba538a1700 1 -- 172.21.15.87:0/235303914 wait complete. 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 Processor -- start 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 -- start start 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c101030 0x7fba4c1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c112c20 con 0x7fba4c108ad0 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c112aa0 con 0x7fba4c101030 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba538a1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c112da0 con 0x7fba4c1019d0 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba51e3e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba51e3e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47048/0 (socket says 172.21.15.87:47048) 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba51e3e700 1 -- 172.21.15.87:0/2519530645 learned_addr learned my addr 172.21.15.87:0/2519530645 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.873+0000 7fba5163d700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c101030 0x7fba4c1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 -- 172.21.15.87:0/2519530645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 msgr2=0x7fba4c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba50e3c700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 -- 172.21.15.87:0/2519530645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c101030 msgr2=0x7fba4c1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:41.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c101030 0x7fba4c1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 -- 172.21.15.87:0/2519530645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c19e100 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba50e3c700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba51e3e700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fba4400ada0 tx=0x7fba4400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44017020 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba4400ce30 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c19e3f0 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44005350 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.874+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c1a5c70 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.875+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c104350 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.876+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fba4400c980 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.877+0000 7fba427fc700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 0x7fba380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.877+0000 7fba5163d700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 0x7fba380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.877+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fba440d0140 con 0x7fba4c108ad0 2024-04-22T13:30:41.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.877+0000 7fba5163d700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 0x7fba380a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005950 tx=0x7fba3c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:41.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:41.880+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba44021020 con 0x7fba4c108ad0 2024-04-22T13:30:42.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.087+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fba4c02e140 con 0x7fba4c108ad0 2024-04-22T13:30:42.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.087+0000 7fba427fc700 1 -- 172.21.15.87:0/2519530645 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fba4400a1f0 con 0x7fba4c108ad0 2024-04-22T13:30:42.083 INFO:teuthology.orchestra.run.smithi087.stdout:219043332141 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 msgr2=0x7fba380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 0x7fba380a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005950 tx=0x7fba3c00a300 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 msgr2=0x7fba4c19d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fba4400ada0 tx=0x7fba4400c330 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 shutdown_connections 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fba380a3cd0 0x7fba380a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba4c108ad0 0x7fba4c19d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba4c101030 0x7fba4c1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 --2- 172.21.15.87:0/2519530645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba4c1019d0 0x7fba4c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 >> 172.21.15.87:0/2519530645 conn(0x7fba4c077fb0 msgr2=0x7fba4c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 shutdown_connections 2024-04-22T13:30:42.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.089+0000 7fba538a1700 1 -- 172.21.15.87:0/2519530645 wait complete. 2024-04-22T13:30:42.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:41 smithi120 ceph-mon[31101]: pgmap v319: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:42.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:41 smithi132 ceph-mon[31531]: pgmap v319: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:42.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:41 smithi087 ceph-mon[27999]: pgmap v319: 97 pgs: 2 peering, 95 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 -- 172.21.15.87:0/1599475940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 msgr2=0x7f7594108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 --2- 172.21.15.87:0/1599475940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f7594108e40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7584009990 tx=0x7f7584009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 -- 172.21.15.87:0/1599475940 shutdown_connections 2024-04-22T13:30:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 --2- 172.21.15.87:0/1599475940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f759410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 --2- 172.21.15.87:0/1599475940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f7594108e40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 --2- 172.21.15.87:0/1599475940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 0x7f759406b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 -- 172.21.15.87:0/1599475940 >> 172.21.15.87:0/1599475940 conn(0x7f7594078980 msgr2=0x7f7594078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:42.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 -- 172.21.15.87:0/1599475940 shutdown_connections 2024-04-22T13:30:42.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.241+0000 7f759a5d6700 1 -- 172.21.15.87:0/1599475940 wait complete. 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.242+0000 7f759a5d6700 1 Processor -- start 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.242+0000 7f759a5d6700 1 -- start start 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 0x7f75941990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f7594199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7594112c00 con 0x7f7594108a60 2024-04-22T13:30:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7594112a80 con 0x7f759406b140 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f759a5d6700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7594112d80 con 0x7f759406bae0 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47066/0 (socket says 172.21.15.87:47066) 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 -- 172.21.15.87:0/3169026571 learned_addr learned my addr 172.21.15.87:0/3169026571 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7593fff700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 0x7f75941990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f75937fe700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f7594199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 -- 172.21.15.87:0/3169026571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 msgr2=0x7f7594199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f7594199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 -- 172.21.15.87:0/3169026571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 msgr2=0x7f75941990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 0x7f75941990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f7598b73700 1 -- 172.21.15.87:0/3169026571 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759419e0e0 con 0x7f7594108a60 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.243+0000 7f75937fe700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f7594199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:42.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f7598b73700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f758c007ee0 tx=0x7f758c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:42.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c00dbd0 con 0x7f7594108a60 2024-04-22T13:30:42.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f758c00dd70 con 0x7f7594108a60 2024-04-22T13:30:42.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759419e3d0 con 0x7f7594108a60 2024-04-22T13:30:42.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c005990 con 0x7f7594108a60 2024-04-22T13:30:42.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.244+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75941a5c50 con 0x7f7594108a60 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.245+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75941041a0 con 0x7f7594108a60 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.246+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f758c01a020 con 0x7f7594108a60 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.246+0000 7f75917fa700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 0x7f757c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.247+0000 7f7593fff700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 0x7f757c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.247+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f758c0cfb50 con 0x7f7594108a60 2024-04-22T13:30:42.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.247+0000 7f7593fff700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 0x7f757c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f758400b5c0 tx=0x7f7584005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:42.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.251+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f758c010070 con 0x7f7594108a60 2024-04-22T13:30:42.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.456+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f759402df80 con 0x7f7594108a60 2024-04-22T13:30:42.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.457+0000 7f75917fa700 1 -- 172.21.15.87:0/3169026571 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f758c09c4d0 con 0x7f7594108a60 2024-04-22T13:30:42.453 INFO:teuthology.orchestra.run.smithi087.stdout:197568495666 2024-04-22T13:30:42.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 msgr2=0x7f757c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 0x7f757c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f758400b5c0 tx=0x7f7584005960 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 msgr2=0x7f759419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f758c007ee0 tx=0x7f758c00d3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 shutdown_connections 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f757c0a3cd0 0x7f757c0a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7594108a60 0x7f759419d9a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f759406b140 0x7f75941990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 --2- 172.21.15.87:0/3169026571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f759406bae0 0x7f7594199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 >> 172.21.15.87:0/3169026571 conn(0x7f7594078980 msgr2=0x7f75940ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 shutdown_connections 2024-04-22T13:30:42.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:42.459+0000 7f759a5d6700 1 -- 172.21.15.87:0/3169026571 wait complete. 2024-04-22T13:30:42.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2519530645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T13:30:42.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3169026571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T13:30:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:42 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2519530645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T13:30:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:42 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3169026571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T13:30:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:42 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2519530645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T13:30:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:42 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3169026571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T13:30:43.358 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332141 for osd.7 2024-04-22T13:30:43.358 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:43.389 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.602 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.602 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:43.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:43 smithi087 ceph-mon[27999]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:44.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495662 got 197568495666 for osd.6 2024-04-22T13:30:44.075 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:44.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:43 smithi120 ceph-mon[31101]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:43 smithi132 ceph-mon[31531]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:46.023 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:45 smithi087 ceph-mon[27999]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:46.023 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:45 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:30:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:45 smithi120 ceph-mon[31101]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:45 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:30:46.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:45 smithi132 ceph-mon[31531]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:46.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:45 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:30:47.971 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:47 smithi087 ceph-mon[27999]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 15 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:47.971 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:47.971 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:47.972 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:47.972 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:47 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:47 smithi120 ceph-mon[31101]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 15 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:48.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:47 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:47 smithi132 ceph-mon[31531]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 15 B/s, 2 keys/s, 0 objects/s recovering 2024-04-22T13:30:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:47 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:30:48.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 -- 172.21.15.87:0/2598739050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78101940 msgr2=0x7f1a78101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:48.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/2598739050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78101940 0x7f1a78101dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00b0a0 tx=0x7f1a6c00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 -- 172.21.15.87:0/2598739050 shutdown_connections 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/2598739050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78102300 0x7f1a7810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/2598739050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78107990 0x7f1a78107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/2598739050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78101940 0x7f1a78101dc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 -- 172.21.15.87:0/2598739050 >> 172.21.15.87:0/2598739050 conn(0x7f1a780788f0 msgr2=0x7f1a78078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 -- 172.21.15.87:0/2598739050 shutdown_connections 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.683+0000 7f1a7db1f700 1 -- 172.21.15.87:0/2598739050 wait complete. 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 Processor -- start 2024-04-22T13:30:48.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 -- start start 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 0x7f1a78199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78107990 0x7f1a781a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78111b60 con 0x7f1a78102300 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781119e0 con 0x7f1a78101940 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a7db1f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78111ce0 con 0x7f1a78107990 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a76ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47072/0 (socket says 172.21.15.87:47072) 2024-04-22T13:30:48.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.684+0000 7f1a76ffd700 1 -- 172.21.15.87:0/4100080156 learned_addr learned my addr 172.21.15.87:0/4100080156 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a777fe700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 0x7f1a78199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a77fff700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78107990 0x7f1a781a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 -- 172.21.15.87:0/4100080156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78107990 msgr2=0x7f1a781a5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78107990 0x7f1a781a5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 -- 172.21.15.87:0/4100080156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 msgr2=0x7f1a78199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 0x7f1a78199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 -- 172.21.15.87:0/4100080156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a781a5580 con 0x7f1a78102300 2024-04-22T13:30:48.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a777fe700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 0x7f1a78199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:48.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a76ffd700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c006070 tx=0x7f1a6c004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:48.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c021070 con 0x7f1a78102300 2024-04-22T13:30:48.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a6c007df0 con 0x7f1a78102300 2024-04-22T13:30:48.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a781a5750 con 0x7f1a78102300 2024-04-22T13:30:48.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c01d450 con 0x7f1a78102300 2024-04-22T13:30:48.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.685+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a781a5ba0 con 0x7f1a78102300 2024-04-22T13:30:48.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.686+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a78103060 con 0x7f1a78102300 2024-04-22T13:30:48.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.687+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1a6c0040b0 con 0x7f1a78102300 2024-04-22T13:30:48.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.688+0000 7f1a74ff9700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 0x7f1a600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:48.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.688+0000 7f1a777fe700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 0x7f1a600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:48.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.688+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f1a6c00f020 con 0x7f1a78102300 2024-04-22T13:30:48.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.688+0000 7f1a777fe700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 0x7f1a600a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007900 tx=0x7f1a70008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:48.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.691+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a6c02a020 con 0x7f1a78102300 2024-04-22T13:30:48.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.900+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1a78109eb0 con 0x7f1a78102300 2024-04-22T13:30:48.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.900+0000 7f1a74ff9700 1 -- 172.21.15.87:0/4100080156 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1a6c098e20 con 0x7f1a78102300 2024-04-22T13:30:48.896 INFO:teuthology.orchestra.run.smithi087.stdout:141733920831 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 msgr2=0x7f1a600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 0x7f1a600a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007900 tx=0x7f1a70008040 comp rx=0 tx=0).stop 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 msgr2=0x7f1a7819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c006070 tx=0x7f1a6c004ef0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 shutdown_connections 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1a600a3cd0 0x7f1a600a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a78107990 0x7f1a781a5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a78101940 0x7f1a78199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 --2- 172.21.15.87:0/4100080156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a78102300 0x7f1a7819a500 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 >> 172.21.15.87:0/4100080156 conn(0x7f1a780788f0 msgr2=0x7f1a780ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 shutdown_connections 2024-04-22T13:30:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.903+0000 7f1a7db1f700 1 -- 172.21.15.87:0/4100080156 wait complete. 2024-04-22T13:30:49.155 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4100080156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T13:30:49.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4100080156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T13:30:49.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4100080156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T13:30:50.120 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:49 smithi132 ceph-mon[31531]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 13 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/966068184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T13:30:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3496661435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T13:30:50.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:49 smithi087 ceph-mon[27999]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 13 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:50.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/966068184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T13:30:50.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3496661435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T13:30:50.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:49 smithi120 ceph-mon[31101]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 13 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:50.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/966068184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T13:30:50.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3496661435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T13:30:51.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 -- 172.21.15.87:0/986227953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c103480 msgr2=0x7f7c4c10f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:51.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 --2- 172.21.15.87:0/986227953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c103480 0x7f7c4c10f8d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400b0a0 tx=0x7f7c4400b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 -- 172.21.15.87:0/986227953 shutdown_connections 2024-04-22T13:30:51.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 --2- 172.21.15.87:0/986227953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c103480 0x7f7c4c10f8d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 --2- 172.21.15.87:0/986227953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c108a70 0x7f7c4c108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 --2- 172.21.15.87:0/986227953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.521+0000 7f7c509c2700 1 -- 172.21.15.87:0/986227953 >> 172.21.15.87:0/986227953 conn(0x7f7c4c079bb0 msgr2=0x7f7c4c079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 -- 172.21.15.87:0/986227953 shutdown_connections 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 -- 172.21.15.87:0/986227953 wait complete. 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 Processor -- start 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 -- start start 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 0x7f7c4c199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c4c108a70 0x7f7c4c19daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c112ca0 con 0x7f7c4c102ac0 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.522+0000 7f7c509c2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c112b20 con 0x7f7c4c103480 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c509c2700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c112e20 con 0x7f7c4c108a70 2024-04-22T13:30:51.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:51.753 INFO:teuthology.orchestra.run.smithi087.stdout:55834574924 2024-04-22T13:30:51.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47102/0 (socket says 172.21.15.87:47102) 2024-04-22T13:30:51.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 -- 172.21.15.87:0/3496661435 learned_addr learned my addr 172.21.15.87:0/3496661435 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:51.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 -- 172.21.15.87:0/3496661435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c4c108a70 msgr2=0x7f7c4c19daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:51.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c4c108a70 0x7f7c4c19daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c49d9b700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 0x7f7c4c199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 -- 172.21.15.87:0/3496661435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 msgr2=0x7f7c4c199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 0x7f7c4c199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 -- 172.21.15.87:0/3496661435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c19e1e0 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c49d9b700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 0x7f7c4c199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c4a59c700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00b530 tx=0x7f7c3c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c015020 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c19e4d0 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c3c011ce0 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c010d20 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.523+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c1a5d50 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.525+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c069730 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.526+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7c3c018020 con 0x7f7c4c102ac0 2024-04-22T13:30:51.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.527+0000 7f7c3b7fe700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 0x7f7c340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.527+0000 7f7c49d9b700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 0x7f7c340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.527+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7c3c0cfcc0 con 0x7f7c4c102ac0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.527+0000 7f7c49d9b700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 0x7f7c340a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005fd0 tx=0x7f7c40005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.532+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c3c01f020 con 0x7f7c4c102ac0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.744+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7c4c02df80 con 0x7f7c4c102ac0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.745+0000 7f7c3b7fe700 1 -- 172.21.15.87:0/3496661435 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7c3c09c650 con 0x7f7c4c102ac0 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 msgr2=0x7f7c340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 0x7f7c340a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005fd0 tx=0x7f7c40005ee0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 msgr2=0x7f7c4c1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00b530 tx=0x7f7c3c00b8f0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 shutdown_connections 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7c340a3d50 0x7f7c340a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c4c108a70 0x7f7c4c19daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c4c102ac0 0x7f7c4c1991d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 --2- 172.21.15.87:0/3496661435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4c103480 0x7f7c4c199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:51.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 >> 172.21.15.87:0/3496661435 conn(0x7f7c4c079bb0 msgr2=0x7f7c4c101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:51.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 shutdown_connections 2024-04-22T13:30:51.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.747+0000 7f7c509c2700 1 -- 172.21.15.87:0/3496661435 wait complete. 2024-04-22T13:30:52.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:51 smithi132 ceph-mon[31531]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:52.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:51 smithi120 ceph-mon[31101]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:52.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:51 smithi087 ceph-mon[27999]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 12 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T13:30:52.408 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574924 for osd.1 2024-04-22T13:30:52.408 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 -- 172.21.15.87:0/1098906728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f774006afe0 msgr2=0x7f774006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 --2- 172.21.15.87:0/1098906728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f774006afe0 0x7f774006b440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7730009990 tx=0x7f7730009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 -- 172.21.15.87:0/1098906728 shutdown_connections 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 --2- 172.21.15.87:0/1098906728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 0x7f774010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 --2- 172.21.15.87:0/1098906728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7740108a20 0x7f7740108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 --2- 172.21.15.87:0/1098906728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f774006afe0 0x7f774006b440 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.932+0000 7f7747821700 1 -- 172.21.15.87:0/1098906728 >> 172.21.15.87:0/1098906728 conn(0x7f77400788b0 msgr2=0x7f7740078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.933+0000 7f7747821700 1 -- 172.21.15.87:0/1098906728 shutdown_connections 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.933+0000 7f7747821700 1 -- 172.21.15.87:0/1098906728 wait complete. 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.933+0000 7f7747821700 1 Processor -- start 2024-04-22T13:30:53.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.933+0000 7f7747821700 1 -- start start 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 0x7f77401993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f774019dc60 0x7f774019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7740112be0 con 0x7f7740108a20 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7740112a60 con 0x7f774019dc60 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7740112d60 con 0x7f774006b980 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47084/0 (socket says 172.21.15.87:47084) 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 -- 172.21.15.87:0/966068184 learned_addr learned my addr 172.21.15.87:0/966068184 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f77455bd700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 0x7f77401993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7745dbe700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f774019dc60 0x7f774019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 -- 172.21.15.87:0/966068184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 msgr2=0x7f77401993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 0x7f77401993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 -- 172.21.15.87:0/966068184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f774019dc60 msgr2=0x7f774019e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f774019dc60 0x7f774019e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 -- 172.21.15.87:0/966068184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774019e780 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7744dbc700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7730006010 tx=0x7f77300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730026070 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7730003680 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774019ea10 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.934+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773000fa10 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.935+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77401a6130 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.935+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7740069730 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.937+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7730029070 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.937+0000 7f77367fc700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 0x7f772c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.938+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f77300d10e0 con 0x7f7740108a20 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.938+0000 7f77455bd700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 0x7f772c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.939+0000 7f77455bd700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 0x7f772c0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f773c005950 tx=0x7f773c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:48.942+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7730030020 con 0x7f7740108a20 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.147+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f774002e140 con 0x7f7740108a20 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.147+0000 7f77367fc700 1 -- 172.21.15.87:0/966068184 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7730013070 con 0x7f7740108a20 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stdout:309237645343 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 msgr2=0x7f772c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:53.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 0x7f772c0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f773c005950 tx=0x7f773c00b410 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 msgr2=0x7f7740199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7730006010 tx=0x7f77300047b0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 shutdown_connections 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f772c0a3cd0 0x7f772c0a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f774019dc60 0x7f774019e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f774006b980 0x7f77401993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 --2- 172.21.15.87:0/966068184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7740108a20 0x7f7740199920 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 >> 172.21.15.87:0/966068184 conn(0x7f77400788b0 msgr2=0x7f77400ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 shutdown_connections 2024-04-22T13:30:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:49.149+0000 7f7747821700 1 -- 172.21.15.87:0/966068184 wait complete. 2024-04-22T13:30:53.898 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:30:53.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:53 smithi087 ceph-mon[27999]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-04-22T13:30:54.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:53 smithi132 ceph-mon[31531]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-04-22T13:30:54.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645343 for osd.10 2024-04-22T13:30:54.343 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:54.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:53 smithi120 ceph-mon[31101]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-04-22T13:30:54.461 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920831 for osd.4 2024-04-22T13:30:54.461 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 -- 172.21.15.87:0/1483548669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174102a90 msgr2=0x7f7174102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 --2- 172.21.15.87:0/1483548669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174102a90 0x7f7174102f10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f715c009990 tx=0x7f715c009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 -- 172.21.15.87:0/1483548669 shutdown_connections 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 --2- 172.21.15.87:0/1483548669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174103450 0x7f717410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 --2- 172.21.15.87:0/1483548669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 0x7f7174108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 --2- 172.21.15.87:0/1483548669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174102a90 0x7f7174102f10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 -- 172.21.15.87:0/1483548669 >> 172.21.15.87:0/1483548669 conn(0x7f7174079ba0 msgr2=0x7f7174079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 -- 172.21.15.87:0/1483548669 shutdown_connections 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.982+0000 7f71789ab700 1 -- 172.21.15.87:0/1483548669 wait complete. 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 Processor -- start 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 -- start start 2024-04-22T13:30:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174102a90 0x7f7174199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 0x7f717419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7174112be0 con 0x7f7174103450 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7174112a60 con 0x7f7174102a90 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f71789ab700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7174112d60 con 0x7f7174108a50 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f7169d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f7169d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40030/0 (socket says 172.21.15.87:40030) 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f7169d9b700 1 -- 172.21.15.87:0/3483128888 learned_addr learned my addr 172.21.15.87:0/3483128888 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f7172d9d700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 0x7f717419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.983+0000 7f717259c700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174102a90 0x7f7174199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 -- 172.21.15.87:0/3483128888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 msgr2=0x7f717419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 0x7f717419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 -- 172.21.15.87:0/3483128888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174102a90 msgr2=0x7f7174199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174102a90 0x7f7174199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 -- 172.21.15.87:0/3483128888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717419e160 con 0x7f7174103450 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f7169d9b700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f715c000c00 tx=0x7f715c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c027070 con 0x7f7174103450 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f715c003f20 con 0x7f7174103450 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c022ca0 con 0x7f7174103450 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717419e3f0 con 0x7f7174103450 2024-04-22T13:30:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.984+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71741a5cd0 con 0x7f7174103450 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.985+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7174069730 con 0x7f7174103450 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.986+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f715c02a070 con 0x7f7174103450 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.987+0000 7f716bfff700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 0x7f71600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.987+0000 7f717259c700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 0x7f71600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.987+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f715c0d13c0 con 0x7f7174103450 2024-04-22T13:30:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.988+0000 7f717259c700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 0x7f71600a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7164005950 tx=0x7f716400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:54.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:54.992+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f715c05eff0 con 0x7f7174103450 2024-04-22T13:30:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.195+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f717402df80 con 0x7f7174103450 2024-04-22T13:30:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.195+0000 7f716bfff700 1 -- 172.21.15.87:0/3483128888 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f715c013070 con 0x7f7174103450 2024-04-22T13:30:55.191 INFO:teuthology.orchestra.run.smithi087.stdout:107374182468 2024-04-22T13:30:55.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 msgr2=0x7f71600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:55.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 0x7f71600a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7164005950 tx=0x7f716400b410 comp rx=0 tx=0).stop 2024-04-22T13:30:55.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 msgr2=0x7f7174199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:55.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f715c000c00 tx=0x7f715c0047b0 comp rx=0 tx=0).stop 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 shutdown_connections 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f71600a3cd0 0x7f71600a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7174108a50 0x7f717419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7174102a90 0x7f7174199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 --2- 172.21.15.87:0/3483128888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7174103450 0x7f7174199690 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 >> 172.21.15.87:0/3483128888 conn(0x7f7174079ba0 msgr2=0x7f7174101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 shutdown_connections 2024-04-22T13:30:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:55.197+0000 7f71789ab700 1 -- 172.21.15.87:0/3483128888 wait complete. 2024-04-22T13:30:56.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:55 smithi132 ceph-mon[31531]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3483128888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T13:30:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:55 smithi087 ceph-mon[27999]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:56.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3483128888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T13:30:56.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:55 smithi120 ceph-mon[31101]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:30:56.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3483128888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T13:30:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/3403985053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 msgr2=0x7ff5d4101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/3403985053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4101860 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009990 tx=0x7ff5bc009ca0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/3403985053 shutdown_connections 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/3403985053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4101da0 0x7ff5d410f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/3403985053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4107b00 0x7ff5d4107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/3403985053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4101860 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/3403985053 >> 172.21.15.87:0/3403985053 conn(0x7ff5d4076c60 msgr2=0x7ff5d4077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/3403985053 shutdown_connections 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.163+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/3403985053 wait complete. 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 Processor -- start 2024-04-22T13:30:57.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 -- start start 2024-04-22T13:30:57.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 0x7ff5d41996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4107b00 0x7ff5d419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4112c50 con 0x7ff5d41013e0 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4112ad0 con 0x7ff5d4101da0 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d9d6d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4112dd0 con 0x7ff5d4107b00 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40034/0 (socket says 172.21.15.87:40034) 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 -- 172.21.15.87:0/2641520324 learned_addr learned my addr 172.21.15.87:0/2641520324 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5caffd700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 0x7ff5d41996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d3fff700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4107b00 0x7ff5d419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 -- 172.21.15.87:0/2641520324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4107b00 msgr2=0x7ff5d419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4107b00 0x7ff5d419da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 -- 172.21.15.87:0/2641520324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 msgr2=0x7ff5d41996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 0x7ff5d41996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.164+0000 7ff5d37fe700 1 -- 172.21.15.87:0/2641520324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d419e140 con 0x7ff5d41013e0 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5caffd700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 0x7ff5d41996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d37fe700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400c970 tx=0x7ff5c400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4004220 con 0x7ff5d41013e0 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d419e430 con 0x7ff5d41013e0 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c4007d50 con 0x7ff5d41013e0 2024-04-22T13:30:57.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4003d10 con 0x7ff5d41013e0 2024-04-22T13:30:57.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.165+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d41a5cb0 con 0x7ff5d41013e0 2024-04-22T13:30:57.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.166+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d4069730 con 0x7ff5d41013e0 2024-04-22T13:30:57.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.167+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff5c40043c0 con 0x7ff5d41013e0 2024-04-22T13:30:57.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.168+0000 7ff5d17fa700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 0x7ff5c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.168+0000 7ff5caffd700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 0x7ff5c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.168+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ff5c40d0410 con 0x7ff5d41013e0 2024-04-22T13:30:57.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.168+0000 7ff5caffd700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 0x7ff5c00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc006010 tx=0x7ff5bc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:57.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.172+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5c4014030 con 0x7ff5d41013e0 2024-04-22T13:30:57.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.376+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff5d402df80 con 0x7ff5d41013e0 2024-04-22T13:30:57.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.376+0000 7ff5d17fa700 1 -- 172.21.15.87:0/2641520324 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff5c409cd90 con 0x7ff5d41013e0 2024-04-22T13:30:57.372 INFO:teuthology.orchestra.run.smithi087.stdout:77309411403 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 msgr2=0x7ff5c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 0x7ff5c00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc006010 tx=0x7ff5bc00b540 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 msgr2=0x7ff5d4199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400c970 tx=0x7ff5c400cd30 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 shutdown_connections 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff5c00a3cd0 0x7ff5c00a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4107b00 0x7ff5d419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5d41013e0 0x7ff5d4199180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 --2- 172.21.15.87:0/2641520324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5d4101da0 0x7ff5d41996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 >> 172.21.15.87:0/2641520324 conn(0x7ff5d4076c60 msgr2=0x7ff5d40ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:57.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 shutdown_connections 2024-04-22T13:30:57.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.378+0000 7ff5d9d6d700 1 -- 172.21.15.87:0/2641520324 wait complete. 2024-04-22T13:30:57.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 -- 172.21.15.87:0/19265005 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4103450 msgr2=0x7f57d410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 --2- 172.21.15.87:0/19265005 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4103450 0x7f57d410f8b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00b0a0 tx=0x7f57cc00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 -- 172.21.15.87:0/19265005 shutdown_connections 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 --2- 172.21.15.87:0/19265005 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4103450 0x7f57d410f8b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 --2- 172.21.15.87:0/19265005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4108a50 0x7f57d4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 --2- 172.21.15.87:0/19265005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 -- 172.21.15.87:0/19265005 >> 172.21.15.87:0/19265005 conn(0x7f57d4079ba0 msgr2=0x7f57d4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.819+0000 7f57da2f3700 1 -- 172.21.15.87:0/19265005 shutdown_connections 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.820+0000 7f57da2f3700 1 -- 172.21.15.87:0/19265005 wait complete. 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.820+0000 7f57da2f3700 1 Processor -- start 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.820+0000 7f57da2f3700 1 -- start start 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d41990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4103450 0x7f57d41995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f57d4112c00 con 0x7f57d4108a50 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57d4112a80 con 0x7f57d4102a90 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57d4112d80 con 0x7f57d4103450 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40052/0 (socket says 172.21.15.87:40052) 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 -- 172.21.15.87:0/3241983114 learned_addr learned my addr 172.21.15.87:0/3241983114 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d37fe700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4103450 0x7f57d41995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d3fff700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d41990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 -- 172.21.15.87:0/3241983114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4103450 msgr2=0x7f57d41995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4103450 0x7f57d41995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 -- 172.21.15.87:0/3241983114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 msgr2=0x7f57d41990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d41990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 -- 172.21.15.87:0/3241983114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d419e0c0 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d3fff700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d41990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d8890700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0062a0 tx=0x7f57cc004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc021070 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d419e350 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57cc007d00 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc01d450 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.821+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d41a5c30 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.822+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d4069730 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.823+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f57cc004000 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.824+0000 7f57d17fa700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 0x7f57bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.824+0000 7f57d3fff700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 0x7f57bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.824+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f57cc00f020 con 0x7f57d4108a50 2024-04-22T13:30:57.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.824+0000 7f57d3fff700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 0x7f57bc0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005fd0 tx=0x7f57c4005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:57.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:57.828+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57cc02a020 con 0x7f57d4108a50 2024-04-22T13:30:57.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:57 smithi087 ceph-mon[27999]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:30:57.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:57 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2641520324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T13:30:58.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.031+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f57d402df80 con 0x7f57d4108a50 2024-04-22T13:30:58.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.032+0000 7f57d17fa700 1 -- 172.21.15.87:0/3241983114 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57cc09e960 con 0x7f57d4108a50 2024-04-22T13:30:58.028 INFO:teuthology.orchestra.run.smithi087.stdout:34359738452 2024-04-22T13:30:58.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 msgr2=0x7f57bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:58.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 0x7f57bc0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005fd0 tx=0x7f57c4005d40 comp rx=0 tx=0).stop 2024-04-22T13:30:58.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 msgr2=0x7f57d419d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:58.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0062a0 tx=0x7f57cc004ed0 comp rx=0 tx=0).stop 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 shutdown_connections 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f57bc0a3cd0 0x7f57bc0a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d4108a50 0x7f57d419d980 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4102a90 0x7f57d41990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 --2- 172.21.15.87:0/3241983114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57d4103450 0x7f57d41995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 >> 172.21.15.87:0/3241983114 conn(0x7f57d4079ba0 msgr2=0x7f57d4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 shutdown_connections 2024-04-22T13:30:58.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:58.034+0000 7f57da2f3700 1 -- 172.21.15.87:0/3241983114 wait complete. 2024-04-22T13:30:58.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:57 smithi132 ceph-mon[31531]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:30:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:57 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2641520324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T13:30:58.425 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:57 smithi120 ceph-mon[31101]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:30:58.425 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:57 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2641520324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T13:30:58.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182468 for osd.3 2024-04-22T13:30:58.695 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:59.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3241983114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T13:30:59.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3241983114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T13:30:59.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738452 for osd.0 2024-04-22T13:30:59.355 DEBUG:teuthology.parallel:result is None 2024-04-22T13:30:59.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3241983114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 -- 172.21.15.87:0/244245286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 msgr2=0x7fd38410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/244245286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38410e7f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00b0a0 tx=0x7fd37c00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 -- 172.21.15.87:0/244245286 shutdown_connections 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/244245286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38410e7f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/244245286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd384107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/244245286 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 0x7fd384101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.558+0000 7fd38bcd9700 1 -- 172.21.15.87:0/244245286 >> 172.21.15.87:0/244245286 conn(0x7fd3840788f0 msgr2=0x7fd384078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.559+0000 7fd38bcd9700 1 -- 172.21.15.87:0/244245286 shutdown_connections 2024-04-22T13:30:59.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.559+0000 7fd38bcd9700 1 -- 172.21.15.87:0/244245286 wait complete. 2024-04-22T13:30:59.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 Processor -- start 2024-04-22T13:30:59.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 -- start start 2024-04-22T13:30:59.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 0x7fd38419b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd38419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd384110a60 con 0x7fd384102300 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841108e0 con 0x7fd384107990 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd38bcd9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd384110be0 con 0x7fd384101940 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd389274700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd389274700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40072/0 (socket says 172.21.15.87:40072) 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.560+0000 7fd389274700 1 -- 172.21.15.87:0/3101339048 learned_addr learned my addr 172.21.15.87:0/3101339048 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389a75700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 0x7fd38419b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd38a276700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd38419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 -- 172.21.15.87:0/3101339048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 msgr2=0x7fd38419b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 0x7fd38419b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 -- 172.21.15.87:0/3101339048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 msgr2=0x7fd38419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:59.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd38419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 -- 172.21.15.87:0/3101339048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3841a02b0 con 0x7fd384102300 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd38a276700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd38419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd389274700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd37400b530 tx=0x7fd37400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374015020 con 0x7fd384102300 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd374011ce0 con 0x7fd384102300 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3841a05a0 con 0x7fd384102300 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374010e40 con 0x7fd384102300 2024-04-22T13:30:59.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.561+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3841a7e20 con 0x7fd384102300 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.562+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd384069730 con 0x7fd384102300 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.563+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd374018020 con 0x7fd384102300 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.564+0000 7fd37affd700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 0x7fd3700a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.564+0000 7fd389a75700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 0x7fd3700a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.564+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd3740cfb50 con 0x7fd384102300 2024-04-22T13:30:59.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.564+0000 7fd389a75700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 0x7fd3700a61f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd38000ac50 tx=0x7fd38000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:30:59.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.568+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd37401f020 con 0x7fd384102300 2024-04-22T13:30:59.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.774+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd38419c600 con 0x7fd384102300 2024-04-22T13:30:59.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.774+0000 7fd37affd700 1 -- 172.21.15.87:0/3101339048 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd37409c5e0 con 0x7fd384102300 2024-04-22T13:30:59.770 INFO:teuthology.orchestra.run.smithi087.stdout:343597383708 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 msgr2=0x7fd3700a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 0x7fd3700a61f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd38000ac50 tx=0x7fd38000a380 comp rx=0 tx=0).stop 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 msgr2=0x7fd38419b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd37400b530 tx=0x7fd37400b840 comp rx=0 tx=0).stop 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 shutdown_connections 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd3700a3d30 0x7fd3700a61f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd384107990 0x7fd38419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd384101940 0x7fd38419b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 --2- 172.21.15.87:0/3101339048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd384102300 0x7fd38419b7e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:30:59.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 >> 172.21.15.87:0/3101339048 conn(0x7fd3840788f0 msgr2=0x7fd3840ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:30:59.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 shutdown_connections 2024-04-22T13:30:59.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:30:59.776+0000 7fd38bcd9700 1 -- 172.21.15.87:0/3101339048 wait complete. 2024-04-22T13:30:59.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411397 got 77309411403 for osd.2 2024-04-22T13:31:00.000 DEBUG:teuthology.parallel:result is None 2024-04-22T13:31:00.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:59 smithi087 ceph-mon[27999]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:00.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:30:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3101339048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T13:31:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:59 smithi132 ceph-mon[31531]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:30:59 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3101339048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T13:31:00.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:59 smithi120 ceph-mon[31101]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:30:59 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3101339048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T13:31:01.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:00 smithi132 ceph-mon[31531]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:01.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:00 smithi087 ceph-mon[27999]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:01.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:00 smithi120 ceph-mon[31101]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:01.613 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383708 for osd.11 2024-04-22T13:31:01.613 DEBUG:teuthology.parallel:result is None 2024-04-22T13:31:02.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 -- 172.21.15.87:0/4021833622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f045406b180 msgr2=0x7f045406b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 --2- 172.21.15.87:0/4021833622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f045406b180 0x7f045406b630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f044c00b0a0 tx=0x7f044c00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 -- 172.21.15.87:0/4021833622 shutdown_connections 2024-04-22T13:31:02.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 --2- 172.21.15.87:0/4021833622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f045406b180 0x7f045406b630 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 --2- 172.21.15.87:0/4021833622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454107b30 0x7f0454105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 --2- 172.21.15.87:0/4021833622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0454105a70 0x7f0454105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.319+0000 7f045bd21700 1 -- 172.21.15.87:0/4021833622 >> 172.21.15.87:0/4021833622 conn(0x7f0454078740 msgr2=0x7f0454078b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- 172.21.15.87:0/4021833622 shutdown_connections 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- 172.21.15.87:0/4021833622 wait complete. 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 Processor -- start 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- start start 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 0x7f045419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 0x7f045419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0454119320 con 0x7f0454107b30 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04541191a0 con 0x7f045406b180 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045bd21700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f04541194a0 con 0x7f0454105a70 2024-04-22T13:31:02.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.320+0000 7f045a2be700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60048/0 (socket says 172.21.15.87:60048) 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 -- 172.21.15.87:0/1129612334 learned_addr learned my addr 172.21.15.87:0/1129612334 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f0459abd700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 0x7f045419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f04592bc700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 0x7f045419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 -- 172.21.15.87:0/1129612334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 msgr2=0x7f045419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 0x7f045419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 -- 172.21.15.87:0/1129612334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 msgr2=0x7f045419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 0x7f045419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 -- 172.21.15.87:0/1129612334 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04541a4760 con 0x7f0454107b30 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f0459abd700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 0x7f045419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f04592bc700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 0x7f045419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045a2be700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f044c00b830 tx=0x7f044c004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:02.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c021070 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04541a49f0 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f044c003c00 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c004750 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.321+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04541ac2d0 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.322+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0454069730 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.323+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f044c007a00 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.324+0000 7f044affd700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 0x7f04400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.324+0000 7f0459abd700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 0x7f04400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.324+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f044c00f020 con 0x7f0454107b30 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.324+0000 7f0459abd700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 0x7f04400a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0454079bd0 tx=0x7f0450008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:02.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.328+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f044c02a020 con 0x7f0454107b30 2024-04-22T13:31:02.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.534+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f04541ac4a0 con 0x7f0454107b30 2024-04-22T13:31:02.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.534+0000 7f044affd700 1 -- 172.21.15.87:0/1129612334 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f044c09e8a0 con 0x7f0454107b30 2024-04-22T13:31:02.531 INFO:teuthology.orchestra.run.smithi087.stdout:163208757308 2024-04-22T13:31:02.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.536+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 msgr2=0x7f04400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.536+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 0x7f04400a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0454079bd0 tx=0x7f0450008040 comp rx=0 tx=0).stop 2024-04-22T13:31:02.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 msgr2=0x7f04541a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f044c00b830 tx=0x7f044c004d90 comp rx=0 tx=0).stop 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 shutdown_connections 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04400a3cd0 0x7f04400a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0454107b30 0x7f04541a4020 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f045406b180 0x7f045419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 --2- 172.21.15.87:0/1129612334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0454105a70 0x7f045419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 >> 172.21.15.87:0/1129612334 conn(0x7f0454078740 msgr2=0x7f04540ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 shutdown_connections 2024-04-22T13:31:02.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.537+0000 7f045bd21700 1 -- 172.21.15.87:0/1129612334 wait complete. 2024-04-22T13:31:02.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1129612334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 -- 172.21.15.87:0/3232141263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084108a10 msgr2=0x7fb084108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/3232141263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084108a10 0x7fb084108df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009990 tx=0x7fb07c009ca0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 -- 172.21.15.87:0/3232141263 shutdown_connections 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/3232141263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 0x7fb08410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/3232141263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084108a10 0x7fb084108df0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/3232141263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084102b00 0x7fb084102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 -- 172.21.15.87:0/3232141263 >> 172.21.15.87:0/3232141263 conn(0x7fb084079c00 msgr2=0x7fb08407a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 -- 172.21.15.87:0/3232141263 shutdown_connections 2024-04-22T13:31:02.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.833+0000 7fb08c1a8700 1 -- 172.21.15.87:0/3232141263 wait complete. 2024-04-22T13:31:02.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 Processor -- start 2024-04-22T13:31:02.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 -- start start 2024-04-22T13:31:02.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 0x7fb084199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 0x7fb08419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb084112c40 con 0x7fb084102b00 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb084112ac0 con 0x7fb084108a10 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08c1a8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb084112dc0 con 0x7fb0841034c0 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60070/0 (socket says 172.21.15.87:60070) 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 -- 172.21.15.87:0/897188720 learned_addr learned my addr 172.21.15.87:0/897188720 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089743700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 0x7fb084199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08a745700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 0x7fb08419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 -- 172.21.15.87:0/897188720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 msgr2=0x7fb084199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 0x7fb084199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 -- 172.21.15.87:0/897188720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 msgr2=0x7fb08419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 0x7fb08419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb089f44700 1 -- 172.21.15.87:0/897188720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08419e160 con 0x7fb084102b00 2024-04-22T13:31:02.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.834+0000 7fb08a745700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 0x7fb08419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb089f44700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005910 tx=0x7fb07c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c027070 con 0x7fb084102b00 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08419e3f0 con 0x7fb084102b00 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb07c004140 con 0x7fb084102b00 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0841a5cd0 con 0x7fb084102b00 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.835+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c00f460 con 0x7fb084102b00 2024-04-22T13:31:02.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.836+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb084104150 con 0x7fb084102b00 2024-04-22T13:31:02.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.837+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb07c02a070 con 0x7fb084102b00 2024-04-22T13:31:02.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.838+0000 7fb082ffd700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 0x7fb06c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:02.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.838+0000 7fb089743700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 0x7fb06c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:02.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.838+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fb07c0d1b60 con 0x7fb084102b00 2024-04-22T13:31:02.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.838+0000 7fb089743700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 0x7fb06c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb078005ea0 tx=0x7fb078005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:02.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:02.841+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb07c031070 con 0x7fb084102b00 2024-04-22T13:31:02.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1129612334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T13:31:02.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1129612334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T13:31:03.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.042+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb08419a4b0 con 0x7fb084102b00 2024-04-22T13:31:03.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.043+0000 7fb082ffd700 1 -- 172.21.15.87:0/897188720 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb07c013070 con 0x7fb084102b00 2024-04-22T13:31:03.040 INFO:teuthology.orchestra.run.smithi087.stdout:253403070509 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 msgr2=0x7fb06c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 0x7fb06c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb078005ea0 tx=0x7fb078005e30 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 msgr2=0x7fb084199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005910 tx=0x7fb07c003730 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 shutdown_connections 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb06c0a3cd0 0x7fb06c0a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb084108a10 0x7fb08419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb084102b00 0x7fb084199150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 --2- 172.21.15.87:0/897188720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0841034c0 0x7fb084199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 >> 172.21.15.87:0/897188720 conn(0x7fb084079c00 msgr2=0x7fb084100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:03.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 shutdown_connections 2024-04-22T13:31:03.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:03.045+0000 7fb08c1a8700 1 -- 172.21.15.87:0/897188720 wait complete. 2024-04-22T13:31:03.871 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:03 smithi087 ceph-mon[27999]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:03.871 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:03 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/897188720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T13:31:03.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:03 smithi120 ceph-mon[31101]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:03.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:03 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/897188720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T13:31:03.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:03 smithi132 ceph-mon[31531]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:03.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:03 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/897188720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T13:31:04.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070500 got 253403070509 for osd.8 2024-04-22T13:31:04.227 DEBUG:teuthology.parallel:result is None 2024-04-22T13:31:04.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757303 got 163208757308 for osd.5 2024-04-22T13:31:04.775 DEBUG:teuthology.parallel:result is None 2024-04-22T13:31:04.775 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T13:31:04.776 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T13:31:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:05 smithi120 ceph-mon[31101]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:05 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:05 smithi087 ceph-mon[27999]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:06.188 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.188 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:05 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:05 smithi132 ceph-mon[31531]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:05 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 -- 172.21.15.87:0/3603948924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 msgr2=0x7f5ad406b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/3603948924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad406b660 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5abc0099c0 tx=0x7f5abc009cd0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 -- 172.21.15.87:0/3603948924 shutdown_connections 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/3603948924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 0x7f5ad410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/3603948924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 0x7f5ad4108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/3603948924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad406b660 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 -- 172.21.15.87:0/3603948924 >> 172.21.15.87:0/3603948924 conn(0x7f5ad40779b0 msgr2=0x7f5ad4077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 -- 172.21.15.87:0/3603948924 shutdown_connections 2024-04-22T13:31:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.895+0000 7f5ad9da8700 1 -- 172.21.15.87:0/3603948924 wait complete. 2024-04-22T13:31:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 Processor -- start 2024-04-22T13:31:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 -- start start 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 0x7f5ad41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 0x7f5ad419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4112c70 con 0x7f5ad406b1e0 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4112af0 con 0x7f5ad406bba0 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad9da8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4112df0 con 0x7f5ad4108a20 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60086/0 (socket says 172.21.15.87:60086) 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 -- 172.21.15.87:0/676433320 learned_addr learned my addr 172.21.15.87:0/676433320 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad3fff700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 0x7f5ad419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5acaffd700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 0x7f5ad41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 -- 172.21.15.87:0/676433320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 msgr2=0x7f5ad419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 0x7f5ad419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 -- 172.21.15.87:0/676433320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 msgr2=0x7f5ad41996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 0x7f5ad41996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.896+0000 7f5ad37fe700 1 -- 172.21.15.87:0/676433320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad419e160 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad37fe700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400b530 tx=0x7f5ac400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4015020 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad419e450 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac4011ce0 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4010d20 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.897+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a5cd0 con 0x7f5ad406b1e0 2024-04-22T13:31:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.898+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad41041a0 con 0x7f5ad406b1e0 2024-04-22T13:31:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.899+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5ac4018020 con 0x7f5ad406b1e0 2024-04-22T13:31:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.900+0000 7f5ad17fa700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 0x7f5ac00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.900+0000 7f5acaffd700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 0x7f5ac00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.900+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f5ac40d0150 con 0x7f5ad406b1e0 2024-04-22T13:31:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.900+0000 7f5acaffd700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 0x7f5ac00a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4078020 tx=0x7f5abc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:06.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:06.904+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ac401f020 con 0x7f5ad406b1e0 2024-04-22T13:31:07.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.110+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ad419a4b0 con 0x7f5ac00a3cd0 2024-04-22T13:31:07.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.118+0000 7f5ad17fa700 1 -- 172.21.15.87:0/676433320 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258947 (secure 0 0 0) 0x7f5ad419a4b0 con 0x7f5ac00a3cd0 2024-04-22T13:31:07.118 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 msgr2=0x7f5ac00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 0x7f5ac00a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4078020 tx=0x7f5abc01a040 comp rx=0 tx=0).stop 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 msgr2=0x7f5ad41991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400b530 tx=0x7f5ac400b8f0 comp rx=0 tx=0).stop 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 shutdown_connections 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f5ac00a3cd0 0x7f5ac00a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:07.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ad4108a20 0x7f5ad419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ad406b1e0 0x7f5ad41991a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 --2- 172.21.15.87:0/676433320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad406bba0 0x7f5ad41996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 >> 172.21.15.87:0/676433320 conn(0x7f5ad40779b0 msgr2=0x7f5ad41006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 shutdown_connections 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:07.121+0000 7f5ad9da8700 1 -- 172.21.15.87:0/676433320 wait complete. 2024-04-22T13:31:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:31:07.898 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-04-22T13:31:06.771170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653677,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":607,"num_read_kb":419,"num_write":337,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369108,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572716,"statfs":{"total":1151940427776,"available":1151562461184,"internally_reserved":0,"allocated":11833344,"data_stored":7426131,"data_compressed":112761,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002928"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888428+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:30:37.888428+0000","last_peered":"2024-04-22T13:30:37.888428+0000","last_clean":"2024-04-22T13:30:37.888428+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:30:37.888428+0000","last_undegraded":"2024-04-22T13:30:37.888428+0000","last_fullsized":"2024-04-22T13:30:37.888428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888027+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:30:37.888027+0000","last_peered":"2024-04-22T13:30:37.888027+0000","last_clean":"2024-04-22T13:30:37.888027+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:30:37.888027+0000","last_undegraded":"2024-04-22T13:30:37.888027+0000","last_fullsized":"2024-04-22T13:30:37.888027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888000+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:30:37.888000+0000","last_peered":"2024-04-22T13:30:37.888000+0000","last_clean":"2024-04-22T13:30:37.888000+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:30:37.888000+0000","last_undegraded":"2024-04-22T13:30:37.888000+0000","last_fullsized":"2024-04-22T13:30:37.888000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984199+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:30:37.984199+0000","last_peered":"2024-04-22T13:30:37.984199+0000","last_clean":"2024-04-22T13:30:37.984199+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:30:37.984199+0000","last_undegraded":"2024-04-22T13:30:37.984199+0000","last_fullsized":"2024-04-22T13:30:37.984199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886068+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:30:37.886068+0000","last_peered":"2024-04-22T13:30:37.886068+0000","last_clean":"2024-04-22T13:30:37.886068+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:30:37.886068+0000","last_undegraded":"2024-04-22T13:30:37.886068+0000","last_fullsized":"2024-04-22T13:30:37.886068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890783+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:30:37.890783+0000","last_peered":"2024-04-22T13:30:37.890783+0000","last_clean":"2024-04-22T13:30:37.890783+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:30:37.890783+0000","last_undegraded":"2024-04-22T13:30:37.890783+0000","last_fullsized":"2024-04-22T13:30:37.890783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886449+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:30:37.886449+0000","last_peered":"2024-04-22T13:30:37.886449+0000","last_clean":"2024-04-22T13:30:37.886449+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:30:37.886449+0000","last_undegraded":"2024-04-22T13:30:37.886449+0000","last_fullsized":"2024-04-22T13:30:37.886449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888427+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:30:37.888427+0000","last_peered":"2024-04-22T13:30:37.888427+0000","last_clean":"2024-04-22T13:30:37.888427+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:30:37.888427+0000","last_undegraded":"2024-04-22T13:30:37.888427+0000","last_fullsized":"2024-04-22T13:30:37.888427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984547+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:30:37.984547+0000","last_peered":"2024-04-22T13:30:37.984547+0000","last_clean":"2024-04-22T13:30:37.984547+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:30:37.984547+0000","last_undegraded":"2024-04-22T13:30:37.984547+0000","last_fullsized":"2024-04-22T13:30:37.984547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891044+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:30:37.891044+0000","last_peered":"2024-04-22T13:30:37.891044+0000","last_clean":"2024-04-22T13:30:37.891044+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:30:37.891044+0000","last_undegraded":"2024-04-22T13:30:37.891044+0000","last_fullsized":"2024-04-22T13:30:37.891044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886866+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:30:37.886866+0000","last_peered":"2024-04-22T13:30:37.886866+0000","last_clean":"2024-04-22T13:30:37.886866+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:30:37.886866+0000","last_undegraded":"2024-04-22T13:30:37.886866+0000","last_fullsized":"2024-04-22T13:30:37.886866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984141+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:30:37.984141+0000","last_peered":"2024-04-22T13:30:37.984141+0000","last_clean":"2024-04-22T13:30:37.984141+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:30:37.984141+0000","last_undegraded":"2024-04-22T13:30:37.984141+0000","last_fullsized":"2024-04-22T13:30:37.984141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984040+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:30:37.984040+0000","last_peered":"2024-04-22T13:30:37.984040+0000","last_clean":"2024-04-22T13:30:37.984040+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:30:37.984040+0000","last_undegraded":"2024-04-22T13:30:37.984040+0000","last_fullsized":"2024-04-22T13:30:37.984040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888174+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:30:37.888174+0000","last_peered":"2024-04-22T13:30:37.888174+0000","last_clean":"2024-04-22T13:30:37.888174+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:30:37.888174+0000","last_undegraded":"2024-04-22T13:30:37.888174+0000","last_fullsized":"2024-04-22T13:30:37.888174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.887878+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:30:37.887878+0000","last_peered":"2024-04-22T13:30:37.887878+0000","last_clean":"2024-04-22T13:30:37.887878+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:30:37.887878+0000","last_undegraded":"2024-04-22T13:30:37.887878+0000","last_fullsized":"2024-04-22T13:30:37.887878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983907+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:30:37.983907+0000","last_peered":"2024-04-22T13:30:37.983907+0000","last_clean":"2024-04-22T13:30:37.983907+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:30:37.983907+0000","last_undegraded":"2024-04-22T13:30:37.983907+0000","last_fullsized":"2024-04-22T13:30:37.983907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888509+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:30:37.888509+0000","last_peered":"2024-04-22T13:30:37.888509+0000","last_clean":"2024-04-22T13:30:37.888509+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:30:37.888509+0000","last_undegraded":"2024-04-22T13:30:37.888509+0000","last_fullsized":"2024-04-22T13:30:37.888509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984742+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:30:37.984742+0000","last_peered":"2024-04-22T13:30:37.984742+0000","last_clean":"2024-04-22T13:30:37.984742+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:30:37.984742+0000","last_undegraded":"2024-04-22T13:30:37.984742+0000","last_fullsized":"2024-04-22T13:30:37.984742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892259+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:30:37.892259+0000","last_peered":"2024-04-22T13:30:37.892259+0000","last_clean":"2024-04-22T13:30:37.892259+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:30:37.892259+0000","last_undegraded":"2024-04-22T13:30:37.892259+0000","last_fullsized":"2024-04-22T13:30:37.892259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984668+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:30:37.984668+0000","last_peered":"2024-04-22T13:30:37.984668+0000","last_clean":"2024-04-22T13:30:37.984668+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:30:37.984668+0000","last_undegraded":"2024-04-22T13:30:37.984668+0000","last_fullsized":"2024-04-22T13:30:37.984668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888601+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:30:37.888601+0000","last_peered":"2024-04-22T13:30:37.888601+0000","last_clean":"2024-04-22T13:30:37.888601+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:30:37.888601+0000","last_undegraded":"2024-04-22T13:30:37.888601+0000","last_fullsized":"2024-04-22T13:30:37.888601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888812+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:30:37.888812+0000","last_peered":"2024-04-22T13:30:37.888812+0000","last_clean":"2024-04-22T13:30:37.888812+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:30:37.888812+0000","last_undegraded":"2024-04-22T13:30:37.888812+0000","last_fullsized":"2024-04-22T13:30:37.888812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888089+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:30:37.888089+0000","last_peered":"2024-04-22T13:30:37.888089+0000","last_clean":"2024-04-22T13:30:37.888089+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:30:37.888089+0000","last_undegraded":"2024-04-22T13:30:37.888089+0000","last_fullsized":"2024-04-22T13:30:37.888089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.903934+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:30:37.903934+0000","last_peered":"2024-04-22T13:30:37.903934+0000","last_clean":"2024-04-22T13:30:37.903934+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:30:37.903934+0000","last_undegraded":"2024-04-22T13:30:37.903934+0000","last_fullsized":"2024-04-22T13:30:37.903934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984670+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:30:37.984670+0000","last_peered":"2024-04-22T13:30:37.984670+0000","last_clean":"2024-04-22T13:30:37.984670+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:30:37.984670+0000","last_undegraded":"2024-04-22T13:30:37.984670+0000","last_fullsized":"2024-04-22T13:30:37.984670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886291+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:30:37.886291+0000","last_peered":"2024-04-22T13:30:37.886291+0000","last_clean":"2024-04-22T13:30:37.886291+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:30:37.886291+0000","last_undegraded":"2024-04-22T13:30:37.886291+0000","last_fullsized":"2024-04-22T13:30:37.886291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889707+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:30:37.889707+0000","last_peered":"2024-04-22T13:30:37.889707+0000","last_clean":"2024-04-22T13:30:37.889707+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:30:37.889707+0000","last_undegraded":"2024-04-22T13:30:37.889707+0000","last_fullsized":"2024-04-22T13:30:37.889707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.432772+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:04.432772+0000","last_peered":"2024-04-22T13:31:04.432772+0000","last_clean":"2024-04-22T13:31:04.432772+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:04.432772+0000","last_undegraded":"2024-04-22T13:31:04.432772+0000","last_fullsized":"2024-04-22T13:31:04.432772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886115+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:30:37.886115+0000","last_peered":"2024-04-22T13:30:37.886115+0000","last_clean":"2024-04-22T13:30:37.886115+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:30:37.886115+0000","last_undegraded":"2024-04-22T13:30:37.886115+0000","last_fullsized":"2024-04-22T13:30:37.886115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890675+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:30:37.890675+0000","last_peered":"2024-04-22T13:30:37.890675+0000","last_clean":"2024-04-22T13:30:37.890675+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:30:37.890675+0000","last_undegraded":"2024-04-22T13:30:37.890675+0000","last_fullsized":"2024-04-22T13:30:37.890675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890205+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:30:37.890205+0000","last_peered":"2024-04-22T13:30:37.890205+0000","last_clean":"2024-04-22T13:30:37.890205+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:30:37.890205+0000","last_undegraded":"2024-04-22T13:30:37.890205+0000","last_fullsized":"2024-04-22T13:30:37.890205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983847+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:30:37.983847+0000","last_peered":"2024-04-22T13:30:37.983847+0000","last_clean":"2024-04-22T13:30:37.983847+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:30:37.983847+0000","last_undegraded":"2024-04-22T13:30:37.983847+0000","last_fullsized":"2024-04-22T13:30:37.983847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888633+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:30:37.888633+0000","last_peered":"2024-04-22T13:30:37.888633+0000","last_clean":"2024-04-22T13:30:37.888633+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:30:37.888633+0000","last_undegraded":"2024-04-22T13:30:37.888633+0000","last_fullsized":"2024-04-22T13:30:37.888633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889655+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:30:37.889655+0000","last_peered":"2024-04-22T13:30:37.889655+0000","last_clean":"2024-04-22T13:30:37.889655+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:30:37.889655+0000","last_undegraded":"2024-04-22T13:30:37.889655+0000","last_fullsized":"2024-04-22T13:30:37.889655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984351+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:30:37.984351+0000","last_peered":"2024-04-22T13:30:37.984351+0000","last_clean":"2024-04-22T13:30:37.984351+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:30:37.984351+0000","last_undegraded":"2024-04-22T13:30:37.984351+0000","last_fullsized":"2024-04-22T13:30:37.984351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.431923+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:04.431923+0000","last_peered":"2024-04-22T13:31:04.431923+0000","last_clean":"2024-04-22T13:31:04.431923+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:04.431923+0000","last_undegraded":"2024-04-22T13:31:04.431923+0000","last_fullsized":"2024-04-22T13:31:04.431923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984583+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:30:37.984583+0000","last_peered":"2024-04-22T13:30:37.984583+0000","last_clean":"2024-04-22T13:30:37.984583+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:30:37.984583+0000","last_undegraded":"2024-04-22T13:30:37.984583+0000","last_fullsized":"2024-04-22T13:30:37.984583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886884+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:30:37.886884+0000","last_peered":"2024-04-22T13:30:37.886884+0000","last_clean":"2024-04-22T13:30:37.886884+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:30:37.886884+0000","last_undegraded":"2024-04-22T13:30:37.886884+0000","last_fullsized":"2024-04-22T13:30:37.886884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888031+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:30:37.888031+0000","last_peered":"2024-04-22T13:30:37.888031+0000","last_clean":"2024-04-22T13:30:37.888031+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:30:37.888031+0000","last_undegraded":"2024-04-22T13:30:37.888031+0000","last_fullsized":"2024-04-22T13:30:37.888031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890023+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:30:37.890023+0000","last_peered":"2024-04-22T13:30:37.890023+0000","last_clean":"2024-04-22T13:30:37.890023+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:30:37.890023+0000","last_undegraded":"2024-04-22T13:30:37.890023+0000","last_fullsized":"2024-04-22T13:30:37.890023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984765+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:30:37.984765+0000","last_peered":"2024-04-22T13:30:37.984765+0000","last_clean":"2024-04-22T13:30:37.984765+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:30:37.984765+0000","last_undegraded":"2024-04-22T13:30:37.984765+0000","last_fullsized":"2024-04-22T13:30:37.984765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":203,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:05.449711+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:05.449711+0000","last_peered":"2024-04-22T13:31:05.449711+0000","last_clean":"2024-04-22T13:31:05.449711+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:05.449711+0000","last_undegraded":"2024-04-22T13:31:05.449711+0000","last_fullsized":"2024-04-22T13:31:05.449711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":157,"num_read_kb":136,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886245+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:30:37.886245+0000","last_peered":"2024-04-22T13:30:37.886245+0000","last_clean":"2024-04-22T13:30:37.886245+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:30:37.886245+0000","last_undegraded":"2024-04-22T13:30:37.886245+0000","last_fullsized":"2024-04-22T13:30:37.886245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.900267+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:30:37.900267+0000","last_peered":"2024-04-22T13:30:37.900267+0000","last_clean":"2024-04-22T13:30:37.900267+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:30:37.900267+0000","last_undegraded":"2024-04-22T13:30:37.900267+0000","last_fullsized":"2024-04-22T13:30:37.900267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984188+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:30:37.984188+0000","last_peered":"2024-04-22T13:30:37.984188+0000","last_clean":"2024-04-22T13:30:37.984188+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:30:37.984188+0000","last_undegraded":"2024-04-22T13:30:37.984188+0000","last_fullsized":"2024-04-22T13:30:37.984188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890400+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:30:37.890400+0000","last_peered":"2024-04-22T13:30:37.890400+0000","last_clean":"2024-04-22T13:30:37.890400+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:30:37.890400+0000","last_undegraded":"2024-04-22T13:30:37.890400+0000","last_fullsized":"2024-04-22T13:30:37.890400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.985927+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:30:37.985927+0000","last_peered":"2024-04-22T13:30:37.985927+0000","last_clean":"2024-04-22T13:30:37.985927+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:30:37.985927+0000","last_undegraded":"2024-04-22T13:30:37.985927+0000","last_fullsized":"2024-04-22T13:30:37.985927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.431910+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:04.431910+0000","last_peered":"2024-04-22T13:31:04.431910+0000","last_clean":"2024-04-22T13:31:04.431910+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:04.431910+0000","last_undegraded":"2024-04-22T13:31:04.431910+0000","last_fullsized":"2024-04-22T13:31:04.431910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886449+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:30:37.886449+0000","last_peered":"2024-04-22T13:30:37.886449+0000","last_clean":"2024-04-22T13:30:37.886449+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:30:37.886449+0000","last_undegraded":"2024-04-22T13:30:37.886449+0000","last_fullsized":"2024-04-22T13:30:37.886449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.885939+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:30:37.885939+0000","last_peered":"2024-04-22T13:30:37.885939+0000","last_clean":"2024-04-22T13:30:37.885939+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:30:37.885939+0000","last_undegraded":"2024-04-22T13:30:37.885939+0000","last_fullsized":"2024-04-22T13:30:37.885939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889574+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:30:37.889574+0000","last_peered":"2024-04-22T13:30:37.889574+0000","last_clean":"2024-04-22T13:30:37.889574+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:30:37.889574+0000","last_undegraded":"2024-04-22T13:30:37.889574+0000","last_fullsized":"2024-04-22T13:30:37.889574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886461+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:30:37.886461+0000","last_peered":"2024-04-22T13:30:37.886461+0000","last_clean":"2024-04-22T13:30:37.886461+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:30:37.886461+0000","last_undegraded":"2024-04-22T13:30:37.886461+0000","last_fullsized":"2024-04-22T13:30:37.886461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.985586+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:30:37.985586+0000","last_peered":"2024-04-22T13:30:37.985586+0000","last_clean":"2024-04-22T13:30:37.985586+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:30:37.985586+0000","last_undegraded":"2024-04-22T13:30:37.985586+0000","last_fullsized":"2024-04-22T13:30:37.985586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984254+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:30:37.984254+0000","last_peered":"2024-04-22T13:30:37.984254+0000","last_clean":"2024-04-22T13:30:37.984254+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:30:37.984254+0000","last_undegraded":"2024-04-22T13:30:37.984254+0000","last_fullsized":"2024-04-22T13:30:37.984254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891048+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:30:37.891048+0000","last_peered":"2024-04-22T13:30:37.891048+0000","last_clean":"2024-04-22T13:30:37.891048+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:30:37.891048+0000","last_undegraded":"2024-04-22T13:30:37.891048+0000","last_fullsized":"2024-04-22T13:30:37.891048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886760+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:30:37.886760+0000","last_peered":"2024-04-22T13:30:37.886760+0000","last_clean":"2024-04-22T13:30:37.886760+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:30:37.886760+0000","last_undegraded":"2024-04-22T13:30:37.886760+0000","last_fullsized":"2024-04-22T13:30:37.886760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890958+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:30:37.890958+0000","last_peered":"2024-04-22T13:30:37.890958+0000","last_clean":"2024-04-22T13:30:37.890958+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:30:37.890958+0000","last_undegraded":"2024-04-22T13:30:37.890958+0000","last_fullsized":"2024-04-22T13:30:37.890958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889975+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:30:37.889975+0000","last_peered":"2024-04-22T13:30:37.889975+0000","last_clean":"2024-04-22T13:30:37.889975+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:30:37.889975+0000","last_undegraded":"2024-04-22T13:30:37.889975+0000","last_fullsized":"2024-04-22T13:30:37.889975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:00.425757+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:00.425757+0000","last_peered":"2024-04-22T13:31:00.425757+0000","last_clean":"2024-04-22T13:31:00.425757+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:00.425757+0000","last_undegraded":"2024-04-22T13:31:00.425757+0000","last_fullsized":"2024-04-22T13:31:00.425757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889179+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:30:37.889179+0000","last_peered":"2024-04-22T13:30:37.889179+0000","last_clean":"2024-04-22T13:30:37.889179+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:30:37.889179+0000","last_undegraded":"2024-04-22T13:30:37.889179+0000","last_fullsized":"2024-04-22T13:30:37.889179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984201+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:30:37.984201+0000","last_peered":"2024-04-22T13:30:37.984201+0000","last_clean":"2024-04-22T13:30:37.984201+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:30:37.984201+0000","last_undegraded":"2024-04-22T13:30:37.984201+0000","last_fullsized":"2024-04-22T13:30:37.984201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886564+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:30:37.886564+0000","last_peered":"2024-04-22T13:30:37.886564+0000","last_clean":"2024-04-22T13:30:37.886564+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:30:37.886564+0000","last_undegraded":"2024-04-22T13:30:37.886564+0000","last_fullsized":"2024-04-22T13:30:37.886564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983629+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:30:37.983629+0000","last_peered":"2024-04-22T13:30:37.983629+0000","last_clean":"2024-04-22T13:30:37.983629+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:30:37.983629+0000","last_undegraded":"2024-04-22T13:30:37.983629+0000","last_fullsized":"2024-04-22T13:30:37.983629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886727+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:30:37.886727+0000","last_peered":"2024-04-22T13:30:37.886727+0000","last_clean":"2024-04-22T13:30:37.886727+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:30:37.886727+0000","last_undegraded":"2024-04-22T13:30:37.886727+0000","last_fullsized":"2024-04-22T13:30:37.886727+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886305+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:30:37.886305+0000","last_peered":"2024-04-22T13:30:37.886305+0000","last_clean":"2024-04-22T13:30:37.886305+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:30:37.886305+0000","last_undegraded":"2024-04-22T13:30:37.886305+0000","last_fullsized":"2024-04-22T13:30:37.886305+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984000+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:30:37.984000+0000","last_peered":"2024-04-22T13:30:37.984000+0000","last_clean":"2024-04-22T13:30:37.984000+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:30:37.984000+0000","last_undegraded":"2024-04-22T13:30:37.984000+0000","last_fullsized":"2024-04-22T13:30:37.984000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984799+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:30:37.984799+0000","last_peered":"2024-04-22T13:30:37.984799+0000","last_clean":"2024-04-22T13:30:37.984799+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:30:37.984799+0000","last_undegraded":"2024-04-22T13:30:37.984799+0000","last_fullsized":"2024-04-22T13:30:37.984799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889156+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:30:37.889156+0000","last_peered":"2024-04-22T13:30:37.889156+0000","last_clean":"2024-04-22T13:30:37.889156+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:30:37.889156+0000","last_undegraded":"2024-04-22T13:30:37.889156+0000","last_fullsized":"2024-04-22T13:30:37.889156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984536+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:30:37.984536+0000","last_peered":"2024-04-22T13:30:37.984536+0000","last_clean":"2024-04-22T13:30:37.984536+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:30:37.984536+0000","last_undegraded":"2024-04-22T13:30:37.984536+0000","last_fullsized":"2024-04-22T13:30:37.984536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888114+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:30:37.888114+0000","last_peered":"2024-04-22T13:30:37.888114+0000","last_clean":"2024-04-22T13:30:37.888114+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:30:37.888114+0000","last_undegraded":"2024-04-22T13:30:37.888114+0000","last_fullsized":"2024-04-22T13:30:37.888114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984228+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:30:37.984228+0000","last_peered":"2024-04-22T13:30:37.984228+0000","last_clean":"2024-04-22T13:30:37.984228+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:30:37.984228+0000","last_undegraded":"2024-04-22T13:30:37.984228+0000","last_fullsized":"2024-04-22T13:30:37.984228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.885945+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:30:37.885945+0000","last_peered":"2024-04-22T13:30:37.885945+0000","last_clean":"2024-04-22T13:30:37.885945+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:30:37.885945+0000","last_undegraded":"2024-04-22T13:30:37.885945+0000","last_fullsized":"2024-04-22T13:30:37.885945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888568+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:30:37.888568+0000","last_peered":"2024-04-22T13:30:37.888568+0000","last_clean":"2024-04-22T13:30:37.888568+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:30:37.888568+0000","last_undegraded":"2024-04-22T13:30:37.888568+0000","last_fullsized":"2024-04-22T13:30:37.888568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.887997+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:30:37.887997+0000","last_peered":"2024-04-22T13:30:37.887997+0000","last_clean":"2024-04-22T13:30:37.887997+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:30:37.887997+0000","last_undegraded":"2024-04-22T13:30:37.887997+0000","last_fullsized":"2024-04-22T13:30:37.887997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892071+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:30:37.892071+0000","last_peered":"2024-04-22T13:30:37.892071+0000","last_clean":"2024-04-22T13:30:37.892071+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:30:37.892071+0000","last_undegraded":"2024-04-22T13:30:37.892071+0000","last_fullsized":"2024-04-22T13:30:37.892071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888989+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:30:37.888989+0000","last_peered":"2024-04-22T13:30:37.888989+0000","last_clean":"2024-04-22T13:30:37.888989+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:30:37.888989+0000","last_undegraded":"2024-04-22T13:30:37.888989+0000","last_fullsized":"2024-04-22T13:30:37.888989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886070+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:30:37.886070+0000","last_peered":"2024-04-22T13:30:37.886070+0000","last_clean":"2024-04-22T13:30:37.886070+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:30:37.886070+0000","last_undegraded":"2024-04-22T13:30:37.886070+0000","last_fullsized":"2024-04-22T13:30:37.886070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886659+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:30:37.886659+0000","last_peered":"2024-04-22T13:30:37.886659+0000","last_clean":"2024-04-22T13:30:37.886659+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:30:37.886659+0000","last_undegraded":"2024-04-22T13:30:37.886659+0000","last_fullsized":"2024-04-22T13:30:37.886659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892621+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:30:37.892621+0000","last_peered":"2024-04-22T13:30:37.892621+0000","last_clean":"2024-04-22T13:30:37.892621+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:30:37.892621+0000","last_undegraded":"2024-04-22T13:30:37.892621+0000","last_fullsized":"2024-04-22T13:30:37.892621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.887009+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:30:37.887009+0000","last_peered":"2024-04-22T13:30:37.887009+0000","last_clean":"2024-04-22T13:30:37.887009+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:30:37.887009+0000","last_undegraded":"2024-04-22T13:30:37.887009+0000","last_fullsized":"2024-04-22T13:30:37.887009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889888+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:30:37.889888+0000","last_peered":"2024-04-22T13:30:37.889888+0000","last_clean":"2024-04-22T13:30:37.889888+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:30:37.889888+0000","last_undegraded":"2024-04-22T13:30:37.889888+0000","last_fullsized":"2024-04-22T13:30:37.889888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.988344+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:30:37.988344+0000","last_peered":"2024-04-22T13:30:37.988344+0000","last_clean":"2024-04-22T13:30:37.988344+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:30:37.988344+0000","last_undegraded":"2024-04-22T13:30:37.988344+0000","last_fullsized":"2024-04-22T13:30:37.988344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984277+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:30:37.984277+0000","last_peered":"2024-04-22T13:30:37.984277+0000","last_clean":"2024-04-22T13:30:37.984277+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:30:37.984277+0000","last_undegraded":"2024-04-22T13:30:37.984277+0000","last_fullsized":"2024-04-22T13:30:37.984277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886865+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:30:37.886865+0000","last_peered":"2024-04-22T13:30:37.886865+0000","last_clean":"2024-04-22T13:30:37.886865+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:30:37.886865+0000","last_undegraded":"2024-04-22T13:30:37.886865+0000","last_fullsized":"2024-04-22T13:30:37.886865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888152+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:30:37.888152+0000","last_peered":"2024-04-22T13:30:37.888152+0000","last_clean":"2024-04-22T13:30:37.888152+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:30:37.888152+0000","last_undegraded":"2024-04-22T13:30:37.888152+0000","last_fullsized":"2024-04-22T13:30:37.888152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891051+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:30:37.891051+0000","last_peered":"2024-04-22T13:30:37.891051+0000","last_clean":"2024-04-22T13:30:37.891051+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:30:37.891051+0000","last_undegraded":"2024-04-22T13:30:37.891051+0000","last_fullsized":"2024-04-22T13:30:37.891051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891826+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:30:37.891826+0000","last_peered":"2024-04-22T13:30:37.891826+0000","last_clean":"2024-04-22T13:30:37.891826+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:30:37.891826+0000","last_undegraded":"2024-04-22T13:30:37.891826+0000","last_fullsized":"2024-04-22T13:30:37.891826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888356+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:30:37.888356+0000","last_peered":"2024-04-22T13:30:37.888356+0000","last_clean":"2024-04-22T13:30:37.888356+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:30:37.888356+0000","last_undegraded":"2024-04-22T13:30:37.888356+0000","last_fullsized":"2024-04-22T13:30:37.888356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983895+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:30:37.983895+0000","last_peered":"2024-04-22T13:30:37.983895+0000","last_clean":"2024-04-22T13:30:37.983895+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:30:37.983895+0000","last_undegraded":"2024-04-22T13:30:37.983895+0000","last_fullsized":"2024-04-22T13:30:37.983895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888852+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:30:37.888852+0000","last_peered":"2024-04-22T13:30:37.888852+0000","last_clean":"2024-04-22T13:30:37.888852+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:30:37.888852+0000","last_undegraded":"2024-04-22T13:30:37.888852+0000","last_fullsized":"2024-04-22T13:30:37.888852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892395+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:30:37.892395+0000","last_peered":"2024-04-22T13:30:37.892395+0000","last_clean":"2024-04-22T13:30:37.892395+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:30:37.892395+0000","last_undegraded":"2024-04-22T13:30:37.892395+0000","last_fullsized":"2024-04-22T13:30:37.892395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.893674+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:30:37.893674+0000","last_peered":"2024-04-22T13:30:37.893674+0000","last_clean":"2024-04-22T13:30:37.893674+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:30:37.893674+0000","last_undegraded":"2024-04-22T13:30:37.893674+0000","last_fullsized":"2024-04-22T13:30:37.893674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889806+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:30:37.889806+0000","last_peered":"2024-04-22T13:30:37.889806+0000","last_clean":"2024-04-22T13:30:37.889806+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:30:37.889806+0000","last_undegraded":"2024-04-22T13:30:37.889806+0000","last_fullsized":"2024-04-22T13:30:37.889806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888742+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:30:37.888742+0000","last_peered":"2024-04-22T13:30:37.888742+0000","last_clean":"2024-04-22T13:30:37.888742+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:30:37.888742+0000","last_undegraded":"2024-04-22T13:30:37.888742+0000","last_fullsized":"2024-04-22T13:30:37.888742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984920+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:30:37.984920+0000","last_peered":"2024-04-22T13:30:37.984920+0000","last_clean":"2024-04-22T13:30:37.984920+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:30:37.984920+0000","last_undegraded":"2024-04-22T13:30:37.984920+0000","last_fullsized":"2024-04-22T13:30:37.984920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984141+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:30:37.984141+0000","last_peered":"2024-04-22T13:30:37.984141+0000","last_clean":"2024-04-22T13:30:37.984141+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:30:37.984141+0000","last_undegraded":"2024-04-22T13:30:37.984141+0000","last_fullsized":"2024-04-22T13:30:37.984141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888206+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:30:37.888206+0000","last_peered":"2024-04-22T13:30:37.888206+0000","last_clean":"2024-04-22T13:30:37.888206+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:30:37.888206+0000","last_undegraded":"2024-04-22T13:30:37.888206+0000","last_fullsized":"2024-04-22T13:30:37.888206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281019,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63075,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":471,"num_read_kb":300,"num_write":154,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189225,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":456041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106953,"data_compressed":37587,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":456360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":1047702,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":514890,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":456903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":516206,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":455826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1047386,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":59465,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60781,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61043,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:31:07.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T13:31:08.148 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:07 smithi120 ceph-mon[31101]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:07 smithi120 ceph-mon[31101]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:08.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:07 smithi132 ceph-mon[31531]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:07 smithi132 ceph-mon[31531]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:08.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:07 smithi087 ceph-mon[27999]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:08.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:07 smithi087 ceph-mon[27999]: from='mgr.14152 172.21.15.87:0/473920591' entity='mgr.x' 2024-04-22T13:31:09.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:08 smithi120 ceph-mon[31101]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:09.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:08 smithi120 ceph-mon[31101]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-22T13:31:09.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:08 smithi132 ceph-mon[31531]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:09.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:08 smithi132 ceph-mon[31531]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-22T13:31:09.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:08 smithi087 ceph-mon[27999]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:09.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:08 smithi087 ceph-mon[27999]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 -- 172.21.15.87:0/3789075854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 msgr2=0x7f9d6410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 --2- 172.21.15.87:0/3789075854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d6410f8b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b0a0 tx=0x7f9d5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 -- 172.21.15.87:0/3789075854 shutdown_connections 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 --2- 172.21.15.87:0/3789075854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d6410f8b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 --2- 172.21.15.87:0/3789075854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 0x7f9d64108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 --2- 172.21.15.87:0/3789075854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 0x7f9d64102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 -- 172.21.15.87:0/3789075854 >> 172.21.15.87:0/3789075854 conn(0x7f9d64079ba0 msgr2=0x7f9d64079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 -- 172.21.15.87:0/3789075854 shutdown_connections 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.921+0000 7f9d68a85700 1 -- 172.21.15.87:0/3789075854 wait complete. 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 Processor -- start 2024-04-22T13:31:09.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 -- start start 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 0x7f9d64199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 0x7f9d6419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64112c20 con 0x7f9d64103450 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64112aa0 con 0x7f9d64108a50 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d68a85700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64112da0 con 0x7f9d64102a90 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d61d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d61d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54430/0 (socket says 172.21.15.87:54430) 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d61d9b700 1 -- 172.21.15.87:0/2615361105 learned_addr learned my addr 172.21.15.87:0/2615361105 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d62d9d700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 0x7f9d6419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.922+0000 7f9d6259c700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 0x7f9d64199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 -- 172.21.15.87:0/2615361105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 msgr2=0x7f9d64199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 0x7f9d64199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 -- 172.21.15.87:0/2615361105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 msgr2=0x7f9d6419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:09.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 0x7f9d6419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 -- 172.21.15.87:0/2615361105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6419e0a0 con 0x7f9d64103450 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d61d9b700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800d980 tx=0x7f9d5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58017070 con 0x7f9d64103450 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6419e390 con 0x7f9d64103450 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d58005020 con 0x7f9d64103450 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58009b50 con 0x7f9d64103450 2024-04-22T13:31:09.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.923+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d641a5c10 con 0x7f9d64103450 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.924+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d64069730 con 0x7f9d64103450 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.925+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9d580051d0 con 0x7f9d64103450 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.925+0000 7f9d537fe700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 0x7f9d4c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.925+0000 7f9d6259c700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 0x7f9d4c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.926+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f9d580d02a0 con 0x7f9d64103450 2024-04-22T13:31:09.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.926+0000 7f9d6259c700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 0x7f9d4c0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9d54005950 tx=0x7f9d540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:09.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:09.930+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d58020020 con 0x7f9d64103450 2024-04-22T13:31:09.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:09 smithi087 ceph-mon[27999]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:10.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.124+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d64062e80 con 0x7f9d4c0a3c40 2024-04-22T13:31:10.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.131+0000 7f9d537fe700 1 -- 172.21.15.87:0/2615361105 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258997 (secure 0 0 0) 0x7f9d64062e80 con 0x7f9d4c0a3c40 2024-04-22T13:31:10.130 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:10.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 msgr2=0x7f9d4c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:10.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 0x7f9d4c0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9d54005950 tx=0x7f9d540058e0 comp rx=0 tx=0).stop 2024-04-22T13:31:10.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 msgr2=0x7f9d641995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:10.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800d980 tx=0x7f9d5800dc90 comp rx=0 tx=0).stop 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 shutdown_connections 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f9d4c0a3c40 0x7f9d4c0a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d64108a50 0x7f9d6419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d64102a90 0x7f9d64199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 --2- 172.21.15.87:0/2615361105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d64103450 0x7f9d641995d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 >> 172.21.15.87:0/2615361105 conn(0x7f9d64079ba0 msgr2=0x7f9d641017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 shutdown_connections 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:10.134+0000 7f9d68a85700 1 -- 172.21.15.87:0/2615361105 wait complete. 2024-04-22T13:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:31:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:09 smithi120 ceph-mon[31101]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:10.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:09 smithi132 ceph-mon[31531]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:10.898 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-22T13:31:08.771706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653677,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":612,"num_read_kb":424,"num_write":337,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":379,"ondisk_log_size":379,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369100,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572724,"statfs":{"total":1151940427776,"available":1151562469376,"internally_reserved":0,"allocated":11833344,"data_stored":7426131,"data_compressed":112761,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888428+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:30:37.888428+0000","last_peered":"2024-04-22T13:30:37.888428+0000","last_clean":"2024-04-22T13:30:37.888428+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:30:37.888428+0000","last_undegraded":"2024-04-22T13:30:37.888428+0000","last_fullsized":"2024-04-22T13:30:37.888428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888027+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:30:37.888027+0000","last_peered":"2024-04-22T13:30:37.888027+0000","last_clean":"2024-04-22T13:30:37.888027+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:30:37.888027+0000","last_undegraded":"2024-04-22T13:30:37.888027+0000","last_fullsized":"2024-04-22T13:30:37.888027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888000+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:30:37.888000+0000","last_peered":"2024-04-22T13:30:37.888000+0000","last_clean":"2024-04-22T13:30:37.888000+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:30:37.888000+0000","last_undegraded":"2024-04-22T13:30:37.888000+0000","last_fullsized":"2024-04-22T13:30:37.888000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984199+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:30:37.984199+0000","last_peered":"2024-04-22T13:30:37.984199+0000","last_clean":"2024-04-22T13:30:37.984199+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:30:37.984199+0000","last_undegraded":"2024-04-22T13:30:37.984199+0000","last_fullsized":"2024-04-22T13:30:37.984199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886068+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:30:37.886068+0000","last_peered":"2024-04-22T13:30:37.886068+0000","last_clean":"2024-04-22T13:30:37.886068+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:30:37.886068+0000","last_undegraded":"2024-04-22T13:30:37.886068+0000","last_fullsized":"2024-04-22T13:30:37.886068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890783+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:30:37.890783+0000","last_peered":"2024-04-22T13:30:37.890783+0000","last_clean":"2024-04-22T13:30:37.890783+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:30:37.890783+0000","last_undegraded":"2024-04-22T13:30:37.890783+0000","last_fullsized":"2024-04-22T13:30:37.890783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.780035+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:06.780035+0000","last_peered":"2024-04-22T13:31:06.780035+0000","last_clean":"2024-04-22T13:31:06.780035+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:06.780035+0000","last_undegraded":"2024-04-22T13:31:06.780035+0000","last_fullsized":"2024-04-22T13:31:06.780035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888427+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:30:37.888427+0000","last_peered":"2024-04-22T13:30:37.888427+0000","last_clean":"2024-04-22T13:30:37.888427+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:30:37.888427+0000","last_undegraded":"2024-04-22T13:30:37.888427+0000","last_fullsized":"2024-04-22T13:30:37.888427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984547+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:30:37.984547+0000","last_peered":"2024-04-22T13:30:37.984547+0000","last_clean":"2024-04-22T13:30:37.984547+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:30:37.984547+0000","last_undegraded":"2024-04-22T13:30:37.984547+0000","last_fullsized":"2024-04-22T13:30:37.984547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891044+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:30:37.891044+0000","last_peered":"2024-04-22T13:30:37.891044+0000","last_clean":"2024-04-22T13:30:37.891044+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:30:37.891044+0000","last_undegraded":"2024-04-22T13:30:37.891044+0000","last_fullsized":"2024-04-22T13:30:37.891044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.785351+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:06.785351+0000","last_peered":"2024-04-22T13:31:06.785351+0000","last_clean":"2024-04-22T13:31:06.785351+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:06.785351+0000","last_undegraded":"2024-04-22T13:31:06.785351+0000","last_fullsized":"2024-04-22T13:31:06.785351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984141+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:30:37.984141+0000","last_peered":"2024-04-22T13:30:37.984141+0000","last_clean":"2024-04-22T13:30:37.984141+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:30:37.984141+0000","last_undegraded":"2024-04-22T13:30:37.984141+0000","last_fullsized":"2024-04-22T13:30:37.984141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.773682+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:06.773682+0000","last_peered":"2024-04-22T13:31:06.773682+0000","last_clean":"2024-04-22T13:31:06.773682+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:06.773682+0000","last_undegraded":"2024-04-22T13:31:06.773682+0000","last_fullsized":"2024-04-22T13:31:06.773682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888174+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:30:37.888174+0000","last_peered":"2024-04-22T13:30:37.888174+0000","last_clean":"2024-04-22T13:30:37.888174+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:30:37.888174+0000","last_undegraded":"2024-04-22T13:30:37.888174+0000","last_fullsized":"2024-04-22T13:30:37.888174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.887878+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:30:37.887878+0000","last_peered":"2024-04-22T13:30:37.887878+0000","last_clean":"2024-04-22T13:30:37.887878+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:30:37.887878+0000","last_undegraded":"2024-04-22T13:30:37.887878+0000","last_fullsized":"2024-04-22T13:30:37.887878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983907+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:30:37.983907+0000","last_peered":"2024-04-22T13:30:37.983907+0000","last_clean":"2024-04-22T13:30:37.983907+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:30:37.983907+0000","last_undegraded":"2024-04-22T13:30:37.983907+0000","last_fullsized":"2024-04-22T13:30:37.983907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888509+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:30:37.888509+0000","last_peered":"2024-04-22T13:30:37.888509+0000","last_clean":"2024-04-22T13:30:37.888509+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:30:37.888509+0000","last_undegraded":"2024-04-22T13:30:37.888509+0000","last_fullsized":"2024-04-22T13:30:37.888509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984742+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:30:37.984742+0000","last_peered":"2024-04-22T13:30:37.984742+0000","last_clean":"2024-04-22T13:30:37.984742+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:30:37.984742+0000","last_undegraded":"2024-04-22T13:30:37.984742+0000","last_fullsized":"2024-04-22T13:30:37.984742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892259+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:30:37.892259+0000","last_peered":"2024-04-22T13:30:37.892259+0000","last_clean":"2024-04-22T13:30:37.892259+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:30:37.892259+0000","last_undegraded":"2024-04-22T13:30:37.892259+0000","last_fullsized":"2024-04-22T13:30:37.892259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984668+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:30:37.984668+0000","last_peered":"2024-04-22T13:30:37.984668+0000","last_clean":"2024-04-22T13:30:37.984668+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:30:37.984668+0000","last_undegraded":"2024-04-22T13:30:37.984668+0000","last_fullsized":"2024-04-22T13:30:37.984668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888601+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:30:37.888601+0000","last_peered":"2024-04-22T13:30:37.888601+0000","last_clean":"2024-04-22T13:30:37.888601+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:30:37.888601+0000","last_undegraded":"2024-04-22T13:30:37.888601+0000","last_fullsized":"2024-04-22T13:30:37.888601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888812+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:30:37.888812+0000","last_peered":"2024-04-22T13:30:37.888812+0000","last_clean":"2024-04-22T13:30:37.888812+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:30:37.888812+0000","last_undegraded":"2024-04-22T13:30:37.888812+0000","last_fullsized":"2024-04-22T13:30:37.888812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888089+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:30:37.888089+0000","last_peered":"2024-04-22T13:30:37.888089+0000","last_clean":"2024-04-22T13:30:37.888089+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:30:37.888089+0000","last_undegraded":"2024-04-22T13:30:37.888089+0000","last_fullsized":"2024-04-22T13:30:37.888089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.903934+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:30:37.903934+0000","last_peered":"2024-04-22T13:30:37.903934+0000","last_clean":"2024-04-22T13:30:37.903934+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:30:37.903934+0000","last_undegraded":"2024-04-22T13:30:37.903934+0000","last_fullsized":"2024-04-22T13:30:37.903934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984670+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:30:37.984670+0000","last_peered":"2024-04-22T13:30:37.984670+0000","last_clean":"2024-04-22T13:30:37.984670+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:30:37.984670+0000","last_undegraded":"2024-04-22T13:30:37.984670+0000","last_fullsized":"2024-04-22T13:30:37.984670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886291+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:30:37.886291+0000","last_peered":"2024-04-22T13:30:37.886291+0000","last_clean":"2024-04-22T13:30:37.886291+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:30:37.886291+0000","last_undegraded":"2024-04-22T13:30:37.886291+0000","last_fullsized":"2024-04-22T13:30:37.886291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889707+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:30:37.889707+0000","last_peered":"2024-04-22T13:30:37.889707+0000","last_clean":"2024-04-22T13:30:37.889707+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:30:37.889707+0000","last_undegraded":"2024-04-22T13:30:37.889707+0000","last_fullsized":"2024-04-22T13:30:37.889707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.432772+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:04.432772+0000","last_peered":"2024-04-22T13:31:04.432772+0000","last_clean":"2024-04-22T13:31:04.432772+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:04.432772+0000","last_undegraded":"2024-04-22T13:31:04.432772+0000","last_fullsized":"2024-04-22T13:31:04.432772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886115+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:30:37.886115+0000","last_peered":"2024-04-22T13:30:37.886115+0000","last_clean":"2024-04-22T13:30:37.886115+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:30:37.886115+0000","last_undegraded":"2024-04-22T13:30:37.886115+0000","last_fullsized":"2024-04-22T13:30:37.886115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890675+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:30:37.890675+0000","last_peered":"2024-04-22T13:30:37.890675+0000","last_clean":"2024-04-22T13:30:37.890675+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:30:37.890675+0000","last_undegraded":"2024-04-22T13:30:37.890675+0000","last_fullsized":"2024-04-22T13:30:37.890675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890205+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:30:37.890205+0000","last_peered":"2024-04-22T13:30:37.890205+0000","last_clean":"2024-04-22T13:30:37.890205+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:30:37.890205+0000","last_undegraded":"2024-04-22T13:30:37.890205+0000","last_fullsized":"2024-04-22T13:30:37.890205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.983847+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:30:37.983847+0000","last_peered":"2024-04-22T13:30:37.983847+0000","last_clean":"2024-04-22T13:30:37.983847+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:30:37.983847+0000","last_undegraded":"2024-04-22T13:30:37.983847+0000","last_fullsized":"2024-04-22T13:30:37.983847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888633+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:30:37.888633+0000","last_peered":"2024-04-22T13:30:37.888633+0000","last_clean":"2024-04-22T13:30:37.888633+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:30:37.888633+0000","last_undegraded":"2024-04-22T13:30:37.888633+0000","last_fullsized":"2024-04-22T13:30:37.888633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889655+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:30:37.889655+0000","last_peered":"2024-04-22T13:30:37.889655+0000","last_clean":"2024-04-22T13:30:37.889655+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:30:37.889655+0000","last_undegraded":"2024-04-22T13:30:37.889655+0000","last_fullsized":"2024-04-22T13:30:37.889655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984351+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:30:37.984351+0000","last_peered":"2024-04-22T13:30:37.984351+0000","last_clean":"2024-04-22T13:30:37.984351+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:30:37.984351+0000","last_undegraded":"2024-04-22T13:30:37.984351+0000","last_fullsized":"2024-04-22T13:30:37.984351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.431923+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:04.431923+0000","last_peered":"2024-04-22T13:31:04.431923+0000","last_clean":"2024-04-22T13:31:04.431923+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:04.431923+0000","last_undegraded":"2024-04-22T13:31:04.431923+0000","last_fullsized":"2024-04-22T13:31:04.431923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984583+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:30:37.984583+0000","last_peered":"2024-04-22T13:30:37.984583+0000","last_clean":"2024-04-22T13:30:37.984583+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:30:37.984583+0000","last_undegraded":"2024-04-22T13:30:37.984583+0000","last_fullsized":"2024-04-22T13:30:37.984583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886884+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:30:37.886884+0000","last_peered":"2024-04-22T13:30:37.886884+0000","last_clean":"2024-04-22T13:30:37.886884+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:30:37.886884+0000","last_undegraded":"2024-04-22T13:30:37.886884+0000","last_fullsized":"2024-04-22T13:30:37.886884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888031+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:30:37.888031+0000","last_peered":"2024-04-22T13:30:37.888031+0000","last_clean":"2024-04-22T13:30:37.888031+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:30:37.888031+0000","last_undegraded":"2024-04-22T13:30:37.888031+0000","last_fullsized":"2024-04-22T13:30:37.888031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890023+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:30:37.890023+0000","last_peered":"2024-04-22T13:30:37.890023+0000","last_clean":"2024-04-22T13:30:37.890023+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:30:37.890023+0000","last_undegraded":"2024-04-22T13:30:37.890023+0000","last_fullsized":"2024-04-22T13:30:37.890023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984765+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:30:37.984765+0000","last_peered":"2024-04-22T13:30:37.984765+0000","last_clean":"2024-04-22T13:30:37.984765+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:30:37.984765+0000","last_undegraded":"2024-04-22T13:30:37.984765+0000","last_fullsized":"2024-04-22T13:30:37.984765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":203,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:05.449711+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:05.449711+0000","last_peered":"2024-04-22T13:31:05.449711+0000","last_clean":"2024-04-22T13:31:05.449711+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:05.449711+0000","last_undegraded":"2024-04-22T13:31:05.449711+0000","last_fullsized":"2024-04-22T13:31:05.449711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":157,"num_read_kb":136,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886245+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:30:37.886245+0000","last_peered":"2024-04-22T13:30:37.886245+0000","last_clean":"2024-04-22T13:30:37.886245+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:30:37.886245+0000","last_undegraded":"2024-04-22T13:30:37.886245+0000","last_fullsized":"2024-04-22T13:30:37.886245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.900267+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:30:37.900267+0000","last_peered":"2024-04-22T13:30:37.900267+0000","last_clean":"2024-04-22T13:30:37.900267+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:30:37.900267+0000","last_undegraded":"2024-04-22T13:30:37.900267+0000","last_fullsized":"2024-04-22T13:30:37.900267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.773427+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:06.773427+0000","last_peered":"2024-04-22T13:31:06.773427+0000","last_clean":"2024-04-22T13:31:06.773427+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:06.773427+0000","last_undegraded":"2024-04-22T13:31:06.773427+0000","last_fullsized":"2024-04-22T13:31:06.773427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890400+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:30:37.890400+0000","last_peered":"2024-04-22T13:30:37.890400+0000","last_clean":"2024-04-22T13:30:37.890400+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:30:37.890400+0000","last_undegraded":"2024-04-22T13:30:37.890400+0000","last_fullsized":"2024-04-22T13:30:37.890400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.781746+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:06.781746+0000","last_peered":"2024-04-22T13:31:06.781746+0000","last_clean":"2024-04-22T13:31:06.781746+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:06.781746+0000","last_undegraded":"2024-04-22T13:31:06.781746+0000","last_fullsized":"2024-04-22T13:31:06.781746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:04.431910+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:04.431910+0000","last_peered":"2024-04-22T13:31:04.431910+0000","last_clean":"2024-04-22T13:31:04.431910+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:04.431910+0000","last_undegraded":"2024-04-22T13:31:04.431910+0000","last_fullsized":"2024-04-22T13:31:04.431910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.772851+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:06.772851+0000","last_peered":"2024-04-22T13:31:06.772851+0000","last_clean":"2024-04-22T13:31:06.772851+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:06.772851+0000","last_undegraded":"2024-04-22T13:31:06.772851+0000","last_fullsized":"2024-04-22T13:31:06.772851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.885939+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:30:37.885939+0000","last_peered":"2024-04-22T13:30:37.885939+0000","last_clean":"2024-04-22T13:30:37.885939+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:30:37.885939+0000","last_undegraded":"2024-04-22T13:30:37.885939+0000","last_fullsized":"2024-04-22T13:30:37.885939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889574+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:30:37.889574+0000","last_peered":"2024-04-22T13:30:37.889574+0000","last_clean":"2024-04-22T13:30:37.889574+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:30:37.889574+0000","last_undegraded":"2024-04-22T13:30:37.889574+0000","last_fullsized":"2024-04-22T13:30:37.889574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886461+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:30:37.886461+0000","last_peered":"2024-04-22T13:30:37.886461+0000","last_clean":"2024-04-22T13:30:37.886461+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:30:37.886461+0000","last_undegraded":"2024-04-22T13:30:37.886461+0000","last_fullsized":"2024-04-22T13:30:37.886461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.985586+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:30:37.985586+0000","last_peered":"2024-04-22T13:30:37.985586+0000","last_clean":"2024-04-22T13:30:37.985586+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:30:37.985586+0000","last_undegraded":"2024-04-22T13:30:37.985586+0000","last_fullsized":"2024-04-22T13:30:37.985586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984254+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:30:37.984254+0000","last_peered":"2024-04-22T13:30:37.984254+0000","last_clean":"2024-04-22T13:30:37.984254+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:30:37.984254+0000","last_undegraded":"2024-04-22T13:30:37.984254+0000","last_fullsized":"2024-04-22T13:30:37.984254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891048+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:30:37.891048+0000","last_peered":"2024-04-22T13:30:37.891048+0000","last_clean":"2024-04-22T13:30:37.891048+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:30:37.891048+0000","last_undegraded":"2024-04-22T13:30:37.891048+0000","last_fullsized":"2024-04-22T13:30:37.891048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886760+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:30:37.886760+0000","last_peered":"2024-04-22T13:30:37.886760+0000","last_clean":"2024-04-22T13:30:37.886760+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:30:37.886760+0000","last_undegraded":"2024-04-22T13:30:37.886760+0000","last_fullsized":"2024-04-22T13:30:37.886760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.890958+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:30:37.890958+0000","last_peered":"2024-04-22T13:30:37.890958+0000","last_clean":"2024-04-22T13:30:37.890958+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:30:37.890958+0000","last_undegraded":"2024-04-22T13:30:37.890958+0000","last_fullsized":"2024-04-22T13:30:37.890958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889975+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:30:37.889975+0000","last_peered":"2024-04-22T13:30:37.889975+0000","last_clean":"2024-04-22T13:30:37.889975+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:30:37.889975+0000","last_undegraded":"2024-04-22T13:30:37.889975+0000","last_fullsized":"2024-04-22T13:30:37.889975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:05.432748+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:05.432748+0000","last_peered":"2024-04-22T13:31:05.432748+0000","last_clean":"2024-04-22T13:31:05.432748+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:05.432748+0000","last_undegraded":"2024-04-22T13:31:05.432748+0000","last_fullsized":"2024-04-22T13:31:05.432748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889179+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:30:37.889179+0000","last_peered":"2024-04-22T13:30:37.889179+0000","last_clean":"2024-04-22T13:30:37.889179+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:30:37.889179+0000","last_undegraded":"2024-04-22T13:30:37.889179+0000","last_fullsized":"2024-04-22T13:30:37.889179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984201+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:30:37.984201+0000","last_peered":"2024-04-22T13:30:37.984201+0000","last_clean":"2024-04-22T13:30:37.984201+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:30:37.984201+0000","last_undegraded":"2024-04-22T13:30:37.984201+0000","last_fullsized":"2024-04-22T13:30:37.984201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886564+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:30:37.886564+0000","last_peered":"2024-04-22T13:30:37.886564+0000","last_clean":"2024-04-22T13:30:37.886564+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:30:37.886564+0000","last_undegraded":"2024-04-22T13:30:37.886564+0000","last_fullsized":"2024-04-22T13:30:37.886564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.779393+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:06.779393+0000","last_peered":"2024-04-22T13:31:06.779393+0000","last_clean":"2024-04-22T13:31:06.779393+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:06.779393+0000","last_undegraded":"2024-04-22T13:31:06.779393+0000","last_fullsized":"2024-04-22T13:31:06.779393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886727+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:30:37.886727+0000","last_peered":"2024-04-22T13:30:37.886727+0000","last_clean":"2024-04-22T13:30:37.886727+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:30:37.886727+0000","last_undegraded":"2024-04-22T13:30:37.886727+0000","last_fullsized":"2024-04-22T13:30:37.886727+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886305+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:30:37.886305+0000","last_peered":"2024-04-22T13:30:37.886305+0000","last_clean":"2024-04-22T13:30:37.886305+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:30:37.886305+0000","last_undegraded":"2024-04-22T13:30:37.886305+0000","last_fullsized":"2024-04-22T13:30:37.886305+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.777814+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:06.777814+0000","last_peered":"2024-04-22T13:31:06.777814+0000","last_clean":"2024-04-22T13:31:06.777814+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:06.777814+0000","last_undegraded":"2024-04-22T13:31:06.777814+0000","last_fullsized":"2024-04-22T13:31:06.777814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984799+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:30:37.984799+0000","last_peered":"2024-04-22T13:30:37.984799+0000","last_clean":"2024-04-22T13:30:37.984799+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:30:37.984799+0000","last_undegraded":"2024-04-22T13:30:37.984799+0000","last_fullsized":"2024-04-22T13:30:37.984799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889156+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:30:37.889156+0000","last_peered":"2024-04-22T13:30:37.889156+0000","last_clean":"2024-04-22T13:30:37.889156+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:30:37.889156+0000","last_undegraded":"2024-04-22T13:30:37.889156+0000","last_fullsized":"2024-04-22T13:30:37.889156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984536+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:30:37.984536+0000","last_peered":"2024-04-22T13:30:37.984536+0000","last_clean":"2024-04-22T13:30:37.984536+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:30:37.984536+0000","last_undegraded":"2024-04-22T13:30:37.984536+0000","last_fullsized":"2024-04-22T13:30:37.984536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888114+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:30:37.888114+0000","last_peered":"2024-04-22T13:30:37.888114+0000","last_clean":"2024-04-22T13:30:37.888114+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:30:37.888114+0000","last_undegraded":"2024-04-22T13:30:37.888114+0000","last_fullsized":"2024-04-22T13:30:37.888114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984228+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:30:37.984228+0000","last_peered":"2024-04-22T13:30:37.984228+0000","last_clean":"2024-04-22T13:30:37.984228+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:30:37.984228+0000","last_undegraded":"2024-04-22T13:30:37.984228+0000","last_fullsized":"2024-04-22T13:30:37.984228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.885945+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:30:37.885945+0000","last_peered":"2024-04-22T13:30:37.885945+0000","last_clean":"2024-04-22T13:30:37.885945+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:30:37.885945+0000","last_undegraded":"2024-04-22T13:30:37.885945+0000","last_fullsized":"2024-04-22T13:30:37.885945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888568+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:30:37.888568+0000","last_peered":"2024-04-22T13:30:37.888568+0000","last_clean":"2024-04-22T13:30:37.888568+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:30:37.888568+0000","last_undegraded":"2024-04-22T13:30:37.888568+0000","last_fullsized":"2024-04-22T13:30:37.888568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.887997+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:30:37.887997+0000","last_peered":"2024-04-22T13:30:37.887997+0000","last_clean":"2024-04-22T13:30:37.887997+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:30:37.887997+0000","last_undegraded":"2024-04-22T13:30:37.887997+0000","last_fullsized":"2024-04-22T13:30:37.887997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892071+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:30:37.892071+0000","last_peered":"2024-04-22T13:30:37.892071+0000","last_clean":"2024-04-22T13:30:37.892071+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:30:37.892071+0000","last_undegraded":"2024-04-22T13:30:37.892071+0000","last_fullsized":"2024-04-22T13:30:37.892071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888989+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:30:37.888989+0000","last_peered":"2024-04-22T13:30:37.888989+0000","last_clean":"2024-04-22T13:30:37.888989+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:30:37.888989+0000","last_undegraded":"2024-04-22T13:30:37.888989+0000","last_fullsized":"2024-04-22T13:30:37.888989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.886070+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:30:37.886070+0000","last_peered":"2024-04-22T13:30:37.886070+0000","last_clean":"2024-04-22T13:30:37.886070+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:30:37.886070+0000","last_undegraded":"2024-04-22T13:30:37.886070+0000","last_fullsized":"2024-04-22T13:30:37.886070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.787762+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:06.787762+0000","last_peered":"2024-04-22T13:31:06.787762+0000","last_clean":"2024-04-22T13:31:06.787762+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:06.787762+0000","last_undegraded":"2024-04-22T13:31:06.787762+0000","last_fullsized":"2024-04-22T13:31:06.787762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892621+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:30:37.892621+0000","last_peered":"2024-04-22T13:30:37.892621+0000","last_clean":"2024-04-22T13:30:37.892621+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:30:37.892621+0000","last_undegraded":"2024-04-22T13:30:37.892621+0000","last_fullsized":"2024-04-22T13:30:37.892621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.785590+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:06.785590+0000","last_peered":"2024-04-22T13:31:06.785590+0000","last_clean":"2024-04-22T13:31:06.785590+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:06.785590+0000","last_undegraded":"2024-04-22T13:31:06.785590+0000","last_fullsized":"2024-04-22T13:31:06.785590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889888+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:30:37.889888+0000","last_peered":"2024-04-22T13:30:37.889888+0000","last_clean":"2024-04-22T13:30:37.889888+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:30:37.889888+0000","last_undegraded":"2024-04-22T13:30:37.889888+0000","last_fullsized":"2024-04-22T13:30:37.889888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.780376+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:06.780376+0000","last_peered":"2024-04-22T13:31:06.780376+0000","last_clean":"2024-04-22T13:31:06.780376+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:06.780376+0000","last_undegraded":"2024-04-22T13:31:06.780376+0000","last_fullsized":"2024-04-22T13:31:06.780376+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984277+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:30:37.984277+0000","last_peered":"2024-04-22T13:30:37.984277+0000","last_clean":"2024-04-22T13:30:37.984277+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:30:37.984277+0000","last_undegraded":"2024-04-22T13:30:37.984277+0000","last_fullsized":"2024-04-22T13:30:37.984277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.784118+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:06.784118+0000","last_peered":"2024-04-22T13:31:06.784118+0000","last_clean":"2024-04-22T13:31:06.784118+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:06.784118+0000","last_undegraded":"2024-04-22T13:31:06.784118+0000","last_fullsized":"2024-04-22T13:31:06.784118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888152+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:30:37.888152+0000","last_peered":"2024-04-22T13:30:37.888152+0000","last_clean":"2024-04-22T13:30:37.888152+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:30:37.888152+0000","last_undegraded":"2024-04-22T13:30:37.888152+0000","last_fullsized":"2024-04-22T13:30:37.888152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891051+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:30:37.891051+0000","last_peered":"2024-04-22T13:30:37.891051+0000","last_clean":"2024-04-22T13:30:37.891051+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:30:37.891051+0000","last_undegraded":"2024-04-22T13:30:37.891051+0000","last_fullsized":"2024-04-22T13:30:37.891051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.891826+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:30:37.891826+0000","last_peered":"2024-04-22T13:30:37.891826+0000","last_clean":"2024-04-22T13:30:37.891826+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:30:37.891826+0000","last_undegraded":"2024-04-22T13:30:37.891826+0000","last_fullsized":"2024-04-22T13:30:37.891826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888356+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:30:37.888356+0000","last_peered":"2024-04-22T13:30:37.888356+0000","last_clean":"2024-04-22T13:30:37.888356+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:30:37.888356+0000","last_undegraded":"2024-04-22T13:30:37.888356+0000","last_fullsized":"2024-04-22T13:30:37.888356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.779394+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:06.779394+0000","last_peered":"2024-04-22T13:31:06.779394+0000","last_clean":"2024-04-22T13:31:06.779394+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:06.779394+0000","last_undegraded":"2024-04-22T13:31:06.779394+0000","last_fullsized":"2024-04-22T13:31:06.779394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888852+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:30:37.888852+0000","last_peered":"2024-04-22T13:30:37.888852+0000","last_clean":"2024-04-22T13:30:37.888852+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:30:37.888852+0000","last_undegraded":"2024-04-22T13:30:37.888852+0000","last_fullsized":"2024-04-22T13:30:37.888852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.892395+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:30:37.892395+0000","last_peered":"2024-04-22T13:30:37.892395+0000","last_clean":"2024-04-22T13:30:37.892395+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:30:37.892395+0000","last_undegraded":"2024-04-22T13:30:37.892395+0000","last_fullsized":"2024-04-22T13:30:37.892395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.893674+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:30:37.893674+0000","last_peered":"2024-04-22T13:30:37.893674+0000","last_clean":"2024-04-22T13:30:37.893674+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:30:37.893674+0000","last_undegraded":"2024-04-22T13:30:37.893674+0000","last_fullsized":"2024-04-22T13:30:37.893674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.889806+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:30:37.889806+0000","last_peered":"2024-04-22T13:30:37.889806+0000","last_clean":"2024-04-22T13:30:37.889806+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:30:37.889806+0000","last_undegraded":"2024-04-22T13:30:37.889806+0000","last_fullsized":"2024-04-22T13:30:37.889806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888742+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:30:37.888742+0000","last_peered":"2024-04-22T13:30:37.888742+0000","last_clean":"2024-04-22T13:30:37.888742+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:30:37.888742+0000","last_undegraded":"2024-04-22T13:30:37.888742+0000","last_fullsized":"2024-04-22T13:30:37.888742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.984920+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:30:37.984920+0000","last_peered":"2024-04-22T13:30:37.984920+0000","last_clean":"2024-04-22T13:30:37.984920+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:30:37.984920+0000","last_undegraded":"2024-04-22T13:30:37.984920+0000","last_fullsized":"2024-04-22T13:30:37.984920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:31:06.774277+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:06.774277+0000","last_peered":"2024-04-22T13:31:06.774277+0000","last_clean":"2024-04-22T13:31:06.774277+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:06.774277+0000","last_undegraded":"2024-04-22T13:31:06.774277+0000","last_fullsized":"2024-04-22T13:31:06.774277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T13:30:37.888206+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:30:37.888206+0000","last_peered":"2024-04-22T13:30:37.888206+0000","last_clean":"2024-04-22T13:30:37.888206+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:30:37.888206+0000","last_undegraded":"2024-04-22T13:30:37.888206+0000","last_fullsized":"2024-04-22T13:30:37.888206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281019,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63075,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":305,"num_write":154,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189225,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":456041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106953,"data_compressed":37587,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":456360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1047702,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":514890,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":966656,"data_stored":456903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":516206,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":455826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1047386,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34098,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":59465,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60781,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61043,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:31:10.902 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T13:31:10.902 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T13:31:10.902 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T13:31:10.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T13:31:11.140 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:11 smithi120 ceph-mon[31101]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:11 smithi120 ceph-mon[31101]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:12.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:11 smithi132 ceph-mon[31531]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:12.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:11 smithi132 ceph-mon[31531]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:12.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:11 smithi087 ceph-mon[27999]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:12.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:11 smithi087 ceph-mon[27999]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 -- 172.21.15.87:0/3111761141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89441034b0 msgr2=0x7f894410f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 --2- 172.21.15.87:0/3111761141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89441034b0 0x7f894410f920 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f893c00b0d0 tx=0x7f893c00b3e0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 -- 172.21.15.87:0/3111761141 shutdown_connections 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 --2- 172.21.15.87:0/3111761141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89441034b0 0x7f894410f920 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 --2- 172.21.15.87:0/3111761141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8944107940 0x7f8944107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 --2- 172.21.15.87:0/3111761141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 -- 172.21.15.87:0/3111761141 >> 172.21.15.87:0/3111761141 conn(0x7f8944079ba0 msgr2=0x7f8944079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 -- 172.21.15.87:0/3111761141 shutdown_connections 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.810+0000 7f89495f9700 1 -- 172.21.15.87:0/3111761141 wait complete. 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 Processor -- start 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 -- start start 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89441034b0 0x7f8944199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8944112cd0 con 0x7f8944107940 2024-04-22T13:31:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8944112b50 con 0x7f8944102af0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89495f9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8944112e50 con 0x7f89441034b0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89437fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89437fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54446/0 (socket says 172.21.15.87:54446) 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.811+0000 7f89437fe700 1 -- 172.21.15.87:0/2101973907 learned_addr learned my addr 172.21.15.87:0/2101973907 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f8942ffd700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89427fc700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89441034b0 0x7f8944199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 -- 172.21.15.87:0/2101973907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89441034b0 msgr2=0x7f8944199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89441034b0 0x7f8944199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 -- 172.21.15.87:0/2101973907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 msgr2=0x7f8944199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 -- 172.21.15.87:0/2101973907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894419e1c0 con 0x7f8944107940 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f8942ffd700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89437fe700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f893c007a20 tx=0x7f893c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c01f070 con 0x7f8944107940 2024-04-22T13:31:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894419e450 con 0x7f8944107940 2024-04-22T13:31:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f893c003c00 con 0x7f8944107940 2024-04-22T13:31:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89441a5d30 con 0x7f8944107940 2024-04-22T13:31:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.812+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c01b6a0 con 0x7f8944107940 2024-04-22T13:31:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.813+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8944069730 con 0x7f8944107940 2024-04-22T13:31:12.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.814+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f893c022070 con 0x7f8944107940 2024-04-22T13:31:12.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.815+0000 7f892bfff700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 0x7f892c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:12.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.815+0000 7f8942ffd700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 0x7f892c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:12.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.815+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f893c00f020 con 0x7f8944107940 2024-04-22T13:31:12.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.815+0000 7f8942ffd700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 0x7f892c0a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f893400bcf0 tx=0x7f893400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:12.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:12.818+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f893c029020 con 0x7f8944107940 2024-04-22T13:31:13.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.068+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f894402df80 con 0x7f8944107940 2024-04-22T13:31:13.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.069+0000 7f892bfff700 1 -- 172.21.15.87:0/2101973907 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f893c09e490 con 0x7f8944107940 2024-04-22T13:31:13.065 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:13.065 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T13:31:13.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 msgr2=0x7f892c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 0x7f892c0a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f893400bcf0 tx=0x7f893400b2c0 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 msgr2=0x7f894419da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f893c007a20 tx=0x7f893c007a50 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 shutdown_connections 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f892c0a3c40 0x7f892c0a6100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8944107940 0x7f894419da80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8944102af0 0x7f8944199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 --2- 172.21.15.87:0/2101973907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89441034b0 0x7f8944199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 >> 172.21.15.87:0/2101973907 conn(0x7f8944079ba0 msgr2=0x7f8944100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 shutdown_connections 2024-04-22T13:31:13.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:13.071+0000 7f89495f9700 1 -- 172.21.15.87:0/2101973907 wait complete. 2024-04-22T13:31:13.910 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T13:31:13.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 -- 172.21.15.87:0/1296155956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103b40 msgr2=0x7fdefc10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 --2- 172.21.15.87:0/1296155956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103b40 0x7fdefc10a3d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009ca0 tx=0x7fdef402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 -- 172.21.15.87:0/1296155956 shutdown_connections 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 --2- 172.21.15.87:0/1296155956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103b40 0x7fdefc10a3d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 --2- 172.21.15.87:0/1296155956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdefc103180 0x7fdefc103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 --2- 172.21.15.87:0/1296155956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdefc101f90 0x7fdefc1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 -- 172.21.15.87:0/1296155956 >> 172.21.15.87:0/1296155956 conn(0x7fdefc0fd500 msgr2=0x7fdefc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 -- 172.21.15.87:0/1296155956 shutdown_connections 2024-04-22T13:31:14.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.018+0000 7fdf04738640 1 -- 172.21.15.87:0/1296155956 wait complete. 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 Processor -- start 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 -- start start 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdefc101f90 0x7fdefc198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdefc103b40 0x7fdefc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10dbb0 con 0x7fdefc103180 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10da30 con 0x7fdefc103b40 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf04738640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10dd30 con 0x7fdefc101f90 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54450/0 (socket says 172.21.15.87:54450) 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 -- 172.21.15.87:0/937876482 learned_addr learned my addr 172.21.15.87:0/937876482 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 -- 172.21.15.87:0/937876482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdefc101f90 msgr2=0x7fdefc198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdefc101f90 0x7fdefc198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 -- 172.21.15.87:0/937876482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdefc103b40 msgr2=0x7fdefc19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.019+0000 7fdf01cd3640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdefc103b40 0x7fdefc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdf01cd3640 1 -- 172.21.15.87:0/937876482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc19fe70 con 0x7fdefc103180 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdf01cd3640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00b4a0 tx=0x7fdeec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec014070 con 0x7fdefc103180 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc1a0160 con 0x7fdefc103180 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeec010ac0 con 0x7fdefc103180 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec017c30 con 0x7fdefc103180 2024-04-22T13:31:14.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.020+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc1a0730 con 0x7fdefc103180 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.021+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdefc1092b0 con 0x7fdefc103180 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.022+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdeec017070 con 0x7fdefc103180 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.022+0000 7fdef37fe640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 0x7fdee80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.022+0000 7fdf024d4640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 0x7fdee80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.023+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fdeec0cfb30 con 0x7fdefc103180 2024-04-22T13:31:14.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.023+0000 7fdf024d4640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 0x7fdee80a6070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdefc102fe0 tx=0x7fdef8006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.026+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdeec09c5b0 con 0x7fdefc103180 2024-04-22T13:31:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:13 smithi120 ceph-mon[31101]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:13 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2101973907' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T13:31:14.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.207+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdefc02d870 con 0x7fdefc103180 2024-04-22T13:31:14.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.208+0000 7fdef37fe640 1 -- 172.21.15.87:0/937876482 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7fdeec09beb0 con 0x7fdefc103180 2024-04-22T13:31:14.204 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:14.204 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:14.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 msgr2=0x7fdee80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 0x7fdee80a6070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdefc102fe0 tx=0x7fdef8006c90 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 msgr2=0x7fdefc198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00b4a0 tx=0x7fdeec00b960 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 shutdown_connections 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fdee80a3bb0 0x7fdee80a6070 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdefc103b40 0x7fdefc19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdefc103180 0x7fdefc198790 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 --2- 172.21.15.87:0/937876482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdefc101f90 0x7fdefc198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 >> 172.21.15.87:0/937876482 conn(0x7fdefc0fd500 msgr2=0x7fdefc106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 shutdown_connections 2024-04-22T13:31:14.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.211+0000 7fdf04738640 1 -- 172.21.15.87:0/937876482 wait complete. 2024-04-22T13:31:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:13 smithi132 ceph-mon[31531]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:13 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2101973907' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T13:31:14.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:13 smithi087 ceph-mon[27999]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:14.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:13 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2101973907' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T13:31:14.218 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:14.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.323+0000 7fa65ca5c640 1 -- 172.21.15.87:0/421575834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 msgr2=0x7fa6581093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.323+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/421575834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581093c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa650009ca0 tx=0x7fa65002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 -- 172.21.15.87:0/421575834 shutdown_connections 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/421575834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581093c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/421575834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101160 0x7fa6581015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/421575834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658102430 0x7fa658102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 -- 172.21.15.87:0/421575834 >> 172.21.15.87:0/421575834 conn(0x7fa6580fbb90 msgr2=0x7fa6580fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 -- 172.21.15.87:0/421575834 shutdown_connections 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 -- 172.21.15.87:0/421575834 wait complete. 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.324+0000 7fa65ca5c640 1 Processor -- start 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 -- start start 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658101160 0x7fa6581981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658102430 0x7fa65819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810db00 con 0x7fa658101b20 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810d980 con 0x7fa658102430 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810dc80 con 0x7fa658101160 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54458/0 (socket says 172.21.15.87:54458) 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 -- 172.21.15.87:0/476188791 learned_addr learned my addr 172.21.15.87:0/476188791 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65659c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658101160 0x7fa6581981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa656d9d640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658102430 0x7fa65819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 -- 172.21.15.87:0/476188791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658101160 msgr2=0x7fa6581981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658101160 0x7fa6581981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 -- 172.21.15.87:0/476188791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658102430 msgr2=0x7fa65819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658102430 0x7fa65819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 -- 172.21.15.87:0/476188791 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819fdd0 con 0x7fa658101b20 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64dd9b640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa64000d950 tx=0x7fa64000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640017030 con 0x7fa658101b20 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6581a00c0 con 0x7fa658101b20 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.325+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa640004020 con 0x7fa658101b20 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.326+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6400047a0 con 0x7fa658101b20 2024-04-22T13:31:14.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.326+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a0610 con 0x7fa658101b20 2024-04-22T13:31:14.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.327+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6581082a0 con 0x7fa658101b20 2024-04-22T13:31:14.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.327+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa640009a30 con 0x7fa658101b20 2024-04-22T13:31:14.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.328+0000 7fa64ffff640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 0x7fa6440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.328+0000 7fa65659c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 0x7fa6440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.328+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa6400cfd70 con 0x7fa658101b20 2024-04-22T13:31:14.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.328+0000 7fa65659c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 0x7fa6440a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa65806a2f0 tx=0x7fa64800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.333+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa64009c7f0 con 0x7fa658101b20 2024-04-22T13:31:14.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.513+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa65802d830 con 0x7fa658101b20 2024-04-22T13:31:14.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.514+0000 7fa64ffff640 1 -- 172.21.15.87:0/476188791 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fa64009c0f0 con 0x7fa658101b20 2024-04-22T13:31:14.509 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 msgr2=0x7fa6440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 0x7fa6440a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa65806a2f0 tx=0x7fa64800b390 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 msgr2=0x7fa6581986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa64000d950 tx=0x7fa64000de10 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 shutdown_connections 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa6440a3bb0 0x7fa6440a6070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658102430 0x7fa65819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa658101b20 0x7fa6581986f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 --2- 172.21.15.87:0/476188791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa658101160 0x7fa6581981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 >> 172.21.15.87:0/476188791 conn(0x7fa6580fbb90 msgr2=0x7fa658105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 shutdown_connections 2024-04-22T13:31:14.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.516+0000 7fa65ca5c640 1 -- 172.21.15.87:0/476188791 wait complete. 2024-04-22T13:31:14.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:14.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.629+0000 7f89ceb0d640 1 -- 172.21.15.87:0/845638682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c81053a0 msgr2=0x7f89c8107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.629+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/845638682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c81053a0 0x7f89c8107790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009900 tx=0x7f89b402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- 172.21.15.87:0/845638682 shutdown_connections 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/845638682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8107cd0 0x7f89c810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/845638682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c81053a0 0x7f89c8107790 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/845638682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c8102a70 0x7f89c8104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- 172.21.15.87:0/845638682 >> 172.21.15.87:0/845638682 conn(0x7f89c80fc400 msgr2=0x7f89c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- 172.21.15.87:0/845638682 shutdown_connections 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- 172.21.15.87:0/845638682 wait complete. 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 Processor -- start 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- start start 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 0x7f89c81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c81053a0 0x7f89c81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810daa0 con 0x7f89c8107cd0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d920 con 0x7f89c81053a0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89ceb0d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810dc20 con 0x7f89c8102a70 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89cd0aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89cd0aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54488/0 (socket says 172.21.15.87:54488) 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.630+0000 7f89cd0aa640 1 -- 172.21.15.87:0/2565638947 learned_addr learned my addr 172.21.15.87:0/2565638947 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cc8a9640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 0x7f89c81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89c7fff640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c81053a0 0x7f89c81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 -- 172.21.15.87:0/2565638947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 msgr2=0x7f89c81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 0x7f89c81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 -- 172.21.15.87:0/2565638947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c81053a0 msgr2=0x7f89c81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c81053a0 0x7f89c81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 -- 172.21.15.87:0/2565638947 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c819fdd0 con 0x7f89c8107cd0 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cc8a9640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 0x7f89c81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:14.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89cd0aa640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89c000eb10 tx=0x7f89c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0004b20 con 0x7f89c8107cd0 2024-04-22T13:31:14.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c81a00c0 con 0x7f89c8107cd0 2024-04-22T13:31:14.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c0004cc0 con 0x7f89c8107cd0 2024-04-22T13:31:14.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0010670 con 0x7f89c8107cd0 2024-04-22T13:31:14.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.631+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c81a05e0 con 0x7f89c8107cd0 2024-04-22T13:31:14.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.632+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c80fdef0 con 0x7f89c8107cd0 2024-04-22T13:31:14.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.633+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f89c0005020 con 0x7f89c8107cd0 2024-04-22T13:31:14.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.634+0000 7f89c5ffb640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 0x7f89b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.634+0000 7f89cc8a9640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 0x7f89b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.634+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f89c00d0a40 con 0x7f89c8107cd0 2024-04-22T13:31:14.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.634+0000 7f89cc8a9640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 0x7f89b80a6070 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00abe0 tx=0x7f89bc00a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.639+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89c009d4c0 con 0x7f89c8107cd0 2024-04-22T13:31:14.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.815+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f89c8062b50 con 0x7f89c8107cd0 2024-04-22T13:31:14.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.816+0000 7f89c5ffb640 1 -- 172.21.15.87:0/2565638947 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f89c009cdc0 con 0x7f89c8107cd0 2024-04-22T13:31:14.812 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 msgr2=0x7f89b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 0x7f89b80a6070 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00abe0 tx=0x7f89bc00a5c0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 msgr2=0x7f89c819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89c000eb10 tx=0x7f89c000c2d0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 shutdown_connections 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f89b80a3bb0 0x7f89b80a6070 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89c8107cd0 0x7f89c819f760 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89c81053a0 0x7f89c81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 --2- 172.21.15.87:0/2565638947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89c8102a70 0x7f89c81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 >> 172.21.15.87:0/2565638947 conn(0x7f89c80fc400 msgr2=0x7f89c81034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 shutdown_connections 2024-04-22T13:31:14.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.818+0000 7f89ceb0d640 1 -- 172.21.15.87:0/2565638947 wait complete. 2024-04-22T13:31:14.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 -- 172.21.15.87:0/630163895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 msgr2=0x7fa4641024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 --2- 172.21.15.87:0/630163895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa4641024c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa458009900 tx=0x7fa45802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 -- 172.21.15.87:0/630163895 shutdown_connections 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 --2- 172.21.15.87:0/630163895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464102a00 0x7fa464109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 --2- 172.21.15.87:0/630163895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa4641024c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 --2- 172.21.15.87:0/630163895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464100e60 0x7fa464101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 -- 172.21.15.87:0/630163895 >> 172.21.15.87:0/630163895 conn(0x7fa4640fc420 msgr2=0x7fa4640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 -- 172.21.15.87:0/630163895 shutdown_connections 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.931+0000 7fa469693640 1 -- 172.21.15.87:0/630163895 wait complete. 2024-04-22T13:31:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 Processor -- start 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 -- start start 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464100e60 0x7fa46406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464102a00 0x7fa46419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa46410ca00 con 0x7fa464102040 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa46410c880 con 0x7fa464102a00 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa469693640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa46410cb80 con 0x7fa464100e60 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54512/0 (socket says 172.21.15.87:54512) 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 -- 172.21.15.87:0/202470986 learned_addr learned my addr 172.21.15.87:0/202470986 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa462ffd640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464100e60 0x7fa46406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4637fe640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464102a00 0x7fa46419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 -- 172.21.15.87:0/202470986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464100e60 msgr2=0x7fa46406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464100e60 0x7fa46406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 -- 172.21.15.87:0/202470986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464102a00 msgr2=0x7fa46419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464102a00 0x7fa46419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 -- 172.21.15.87:0/202470986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46419fe60 con 0x7fa464102040 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.932+0000 7fa4627fc640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa458006010 tx=0x7fa4580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.933+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458043020 con 0x7fa464102040 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.933+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa458004aa0 con 0x7fa464102040 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.933+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4641a0090 con 0x7fa464102040 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.933+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458003c70 con 0x7fa464102040 2024-04-22T13:31:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.933+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4641a05e0 con 0x7fa464102040 2024-04-22T13:31:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.934+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa464108220 con 0x7fa464102040 2024-04-22T13:31:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.935+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa458046020 con 0x7fa464102040 2024-04-22T13:31:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.935+0000 7fa44bfff640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 0x7fa44c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.935+0000 7fa462ffd640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 0x7fa44c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.936+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa4580f27e0 con 0x7fa464102040 2024-04-22T13:31:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.936+0000 7fa462ffd640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 0x7fa44c0a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa45c005fd0 tx=0x7fa45c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:14.940+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa45804d020 con 0x7fa464102040 2024-04-22T13:31:15.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.118+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa46402d810 con 0x7fa464102040 2024-04-22T13:31:15.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.118+0000 7fa44bfff640 1 -- 172.21.15.87:0/202470986 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fa4580bf260 con 0x7fa464102040 2024-04-22T13:31:15.114 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 msgr2=0x7fa44c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 0x7fa44c0a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa45c005fd0 tx=0x7fa45c005850 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 msgr2=0x7fa46419cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa458006010 tx=0x7fa4580042f0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 shutdown_connections 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fa44c0a3bb0 0x7fa44c0a6070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa464102a00 0x7fa46419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa464102040 0x7fa46419cf30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 --2- 172.21.15.87:0/202470986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa464100e60 0x7fa46406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 >> 172.21.15.87:0/202470986 conn(0x7fa4640fc420 msgr2=0x7fa4641053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 shutdown_connections 2024-04-22T13:31:15.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.120+0000 7fa469693640 1 -- 172.21.15.87:0/202470986 wait complete. 2024-04-22T13:31:15.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/937876482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:15.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/476188791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.127 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2565638947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:15.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:14 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/937876482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:14 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/476188791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:14 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2565638947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:14 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/937876482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:14 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/476188791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:14 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2565638947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 -- 172.21.15.87:0/1895043169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103190 msgr2=0x7f04e4103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1895043169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103190 0x7f04e4103610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009900 tx=0x7f04d802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 -- 172.21.15.87:0/1895043169 shutdown_connections 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1895043169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4103b50 0x7f04e410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1895043169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103190 0x7f04e4103610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1895043169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4101fa0 0x7f04e41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 -- 172.21.15.87:0/1895043169 >> 172.21.15.87:0/1895043169 conn(0x7f04e40fd530 msgr2=0x7f04e40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 -- 172.21.15.87:0/1895043169 shutdown_connections 2024-04-22T13:31:15.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.231+0000 7f04ea38e640 1 -- 172.21.15.87:0/1895043169 wait complete. 2024-04-22T13:31:15.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 Processor -- start 2024-04-22T13:31:15.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 -- start start 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 0x7f04e4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4103190 0x7f04e41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410dae0 con 0x7f04e4103b50 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410d960 con 0x7f04e4103190 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04ea38e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410dc60 con 0x7f04e4101fa0 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54534/0 (socket says 172.21.15.87:54534) 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 -- 172.21.15.87:0/1566994310 learned_addr learned my addr 172.21.15.87:0/1566994310 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e37fe640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4103190 0x7f04e41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e3fff640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 0x7f04e4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 -- 172.21.15.87:0/1566994310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 msgr2=0x7f04e4198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 0x7f04e4198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 -- 172.21.15.87:0/1566994310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4103190 msgr2=0x7f04e41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4103190 0x7f04e41986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e892b640 1 -- 172.21.15.87:0/1566994310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e419fdb0 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.232+0000 7f04e3fff640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 0x7f04e4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04e892b640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00cad0 tx=0x7f04dc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00ae20 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04dc005020 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e41a00a0 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00f6b0 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.233+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e41a05c0 con 0x7f04e4103b50 2024-04-22T13:31:15.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.234+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e4109340 con 0x7f04e4103b50 2024-04-22T13:31:15.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.235+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f04dc0051d0 con 0x7f04e4103b50 2024-04-22T13:31:15.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.235+0000 7f04e17fa640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 0x7f04cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.236+0000 7f04e3fff640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 0x7f04cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.236+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f04dc0d0370 con 0x7f04e4103b50 2024-04-22T13:31:15.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.236+0000 7f04e3fff640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 0x7f04cc0a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f04d40058c0 tx=0x7f04d4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.240+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04dc014030 con 0x7f04e4103b50 2024-04-22T13:31:15.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.459+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04e402d5c0 con 0x7f04e4103b50 2024-04-22T13:31:15.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.460+0000 7f04e17fa640 1 -- 172.21.15.87:0/1566994310 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7f04dc09cdf0 con 0x7f04e4103b50 2024-04-22T13:31:15.456 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:15.456 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:31:15.456 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 msgr2=0x7f04cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 0x7f04cc0a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f04d40058c0 tx=0x7f04d4005850 comp rx=0 tx=0).stop 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 msgr2=0x7f04e419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00cad0 tx=0x7f04dc00cf90 comp rx=0 tx=0).stop 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 shutdown_connections 2024-04-22T13:31:15.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f04cc0a3bb0 0x7f04cc0a6070 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04e4103b50 0x7f04e419f740 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04e4103190 0x7f04e41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 --2- 172.21.15.87:0/1566994310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04e4101fa0 0x7f04e4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 >> 172.21.15.87:0/1566994310 conn(0x7f04e40fd530 msgr2=0x7f04e4106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 shutdown_connections 2024-04-22T13:31:15.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.462+0000 7f04ea38e640 1 -- 172.21.15.87:0/1566994310 wait complete. 2024-04-22T13:31:15.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.573+0000 7f41435b2640 1 -- 172.21.15.87:0/315995445 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 msgr2=0x7f413c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.573+0000 7f41435b2640 1 --2- 172.21.15.87:0/315995445 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c1092a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4134009ca0 tx=0x7f413402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- 172.21.15.87:0/315995445 shutdown_connections 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- 172.21.15.87:0/315995445 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c1092a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- 172.21.15.87:0/315995445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c101fc0 0x7f413c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- 172.21.15.87:0/315995445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c100e70 0x7f413c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- 172.21.15.87:0/315995445 >> 172.21.15.87:0/315995445 conn(0x7f413c0fc3e0 msgr2=0x7f413c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- 172.21.15.87:0/315995445 shutdown_connections 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- 172.21.15.87:0/315995445 wait complete. 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 Processor -- start 2024-04-22T13:31:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- start start 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 0x7f413c104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c101fc0 0x7f413c103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f413c10ca20 con 0x7f413c102980 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f413c10c8a0 con 0x7f413c101fc0 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f41435b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f413c10cba0 con 0x7f413c100e70 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f4141b4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f4141b4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54558/0 (socket says 172.21.15.87:54558) 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.574+0000 7f4141b4f640 1 -- 172.21.15.87:0/972624114 learned_addr learned my addr 172.21.15.87:0/972624114 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f414134e640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 0x7f413c104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 -- 172.21.15.87:0/972624114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 msgr2=0x7f413c104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 0x7f413c104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 -- 172.21.15.87:0/972624114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c101fc0 msgr2=0x7f413c103430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4140b4d640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c101fc0 0x7f413c103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c101fc0 0x7f413c103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 -- 172.21.15.87:0/972624114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c104200 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f414134e640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 0x7f413c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f4141b4f640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f413402c700 tx=0x7f4134035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134039040 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4134044070 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c1a2270 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134004440 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.575+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c1a27c0 con 0x7f413c102980 2024-04-22T13:31:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.576+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c1a2a20 con 0x7f413c102980 2024-04-22T13:31:15.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.577+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f41340045e0 con 0x7f413c102980 2024-04-22T13:31:15.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.578+0000 7f41327fc640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 0x7f41280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.578+0000 7f414134e640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 0x7f41280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.578+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f41340f35d0 con 0x7f413c102980 2024-04-22T13:31:15.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.578+0000 7f414134e640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 0x7f41280a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f412c006fd0 tx=0x7f412c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.582+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41340bfff0 con 0x7f413c102980 2024-04-22T13:31:15.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.760+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f413c1a2bf0 con 0x7f413c102980 2024-04-22T13:31:15.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.762+0000 7f41327fc640 1 -- 172.21.15.87:0/972624114 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7f41340bf8f0 con 0x7f413c102980 2024-04-22T13:31:15.758 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:15.758 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 msgr2=0x7f41280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 0x7f41280a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f412c006fd0 tx=0x7f412c008040 comp rx=0 tx=0).stop 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 msgr2=0x7f413c103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f413402c700 tx=0x7f4134035d60 comp rx=0 tx=0).stop 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 shutdown_connections 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f41280a3bb0 0x7f41280a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f413c102980 0x7f413c103970 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c101fc0 0x7f413c103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 --2- 172.21.15.87:0/972624114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f413c100e70 0x7f413c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 >> 172.21.15.87:0/972624114 conn(0x7f413c0fc3e0 msgr2=0x7f413c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 shutdown_connections 2024-04-22T13:31:15.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.764+0000 7f41435b2640 1 -- 172.21.15.87:0/972624114 wait complete. 2024-04-22T13:31:15.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:15.773 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 -- 172.21.15.87:0/3235563600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 msgr2=0x7fda8c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 --2- 172.21.15.87:0/3235563600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1024c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fda80009900 tx=0x7fda8002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 -- 172.21.15.87:0/3235563600 shutdown_connections 2024-04-22T13:31:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 --2- 172.21.15.87:0/3235563600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 0x7fda8c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 --2- 172.21.15.87:0/3235563600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1024c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 --2- 172.21.15.87:0/3235563600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda8c100e60 0x7fda8c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 -- 172.21.15.87:0/3235563600 >> 172.21.15.87:0/3235563600 conn(0x7fda8c0fc440 msgr2=0x7fda8c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 -- 172.21.15.87:0/3235563600 shutdown_connections 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.879+0000 7fda91a93640 1 -- 172.21.15.87:0/3235563600 wait complete. 2024-04-22T13:31:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 Processor -- start 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 -- start start 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda8c100e60 0x7fda8c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 0x7fda8c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c10db90 con 0x7fda8c102040 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c10da10 con 0x7fda8c102a00 2024-04-22T13:31:15.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda91a93640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c10dd10 con 0x7fda8c100e60 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54568/0 (socket says 172.21.15.87:54568) 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 -- 172.21.15.87:0/1962804929 learned_addr learned my addr 172.21.15.87:0/1962804929 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8bfff640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 0x7fda8c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 -- 172.21.15.87:0/1962804929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda8c100e60 msgr2=0x7fda8c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda8c100e60 0x7fda8c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 -- 172.21.15.87:0/1962804929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 msgr2=0x7fda8c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 0x7fda8c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.880+0000 7fda8affd640 1 -- 172.21.15.87:0/1962804929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c19fed0 con 0x7fda8c102040 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda8affd640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fda80006010 tx=0x7fda800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda80043020 con 0x7fda8c102040 2024-04-22T13:31:15.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c1a0160 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda80004aa0 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda80003c70 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.881+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1a0730 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.882+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8c1082a0 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.883+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fda80046020 con 0x7fda8c102040 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.883+0000 7fda88ff9640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 0x7fda740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:15.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.883+0000 7fda8b7fe640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 0x7fda740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:15.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.883+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fda800f27e0 con 0x7fda8c102040 2024-04-22T13:31:15.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.884+0000 7fda8b7fe640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 0x7fda740a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda8c101bb0 tx=0x7fda7c006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:15.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:15.888+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda8004d020 con 0x7fda8c102040 2024-04-22T13:31:16.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.065+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda8c02d7e0 con 0x7fda8c102040 2024-04-22T13:31:16.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.065+0000 7fda88ff9640 1 -- 172.21.15.87:0/1962804929 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fda800bf260 con 0x7fda8c102040 2024-04-22T13:31:16.061 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:16.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.067+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 msgr2=0x7fda740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.067+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 0x7fda740a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda8c101bb0 tx=0x7fda7c006c90 comp rx=0 tx=0).stop 2024-04-22T13:31:16.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.067+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 msgr2=0x7fda8c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.067+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fda80006010 tx=0x7fda800042f0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 shutdown_connections 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fda740a3bb0 0x7fda740a6070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda8c102a00 0x7fda8c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda8c102040 0x7fda8c1987f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 --2- 172.21.15.87:0/1962804929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda8c100e60 0x7fda8c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 >> 172.21.15.87:0/1962804929 conn(0x7fda8c0fc440 msgr2=0x7fda8c105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 shutdown_connections 2024-04-22T13:31:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.068+0000 7fda91a93640 1 -- 172.21.15.87:0/1962804929 wait complete. 2024-04-22T13:31:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:15 smithi087 ceph-mon[27999]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:15 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/202470986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:15 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1566994310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:15 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/972624114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:16.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:15 smithi120 ceph-mon[31101]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:15 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/202470986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:15 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1566994310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:15 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/972624114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.185+0000 7f46f0fb1640 1 -- 172.21.15.87:0/3984743324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec1029a0 msgr2=0x7f46ec1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.185+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/3984743324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec1029a0 0x7f46ec1092c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ca0 tx=0x7f46e402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- 172.21.15.87:0/3984743324 shutdown_connections 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/3984743324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec1029a0 0x7f46ec1092c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/3984743324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec101fe0 0x7f46ec102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/3984743324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec100e90 0x7f46ec1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- 172.21.15.87:0/3984743324 >> 172.21.15.87:0/3984743324 conn(0x7f46ec0fc400 msgr2=0x7f46ec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- 172.21.15.87:0/3984743324 shutdown_connections 2024-04-22T13:31:16.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- 172.21.15.87:0/3984743324 wait complete. 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 Processor -- start 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- start start 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 0x7f46ec198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 0x7f46ec19f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec10dbf0 con 0x7f46ec100e90 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec10da70 con 0x7f46ec1029a0 2024-04-22T13:31:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.186+0000 7f46f0fb1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec10dd70 con 0x7f46ec101fe0 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54576/0 (socket says 172.21.15.87:54576) 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 -- 172.21.15.87:0/4093727512 learned_addr learned my addr 172.21.15.87:0/4093727512 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ead9d640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 0x7f46ec19f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46e9d9b640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 0x7f46ec198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 -- 172.21.15.87:0/4093727512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 msgr2=0x7f46ec198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 0x7f46ec198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 -- 172.21.15.87:0/4093727512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 msgr2=0x7f46ec19f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 0x7f46ec19f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 -- 172.21.15.87:0/4093727512 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ec19ff10 con 0x7f46ec100e90 2024-04-22T13:31:16.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ead9d640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 0x7f46ec19f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:16.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46e9d9b640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 0x7f46ec198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46ea59c640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00b4a0 tx=0x7f46dc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc014070 con 0x7f46ec100e90 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ec1a0200 con 0x7f46ec100e90 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46dc010ac0 con 0x7f46ec100e90 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc017c30 con 0x7f46ec100e90 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.187+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ec1a07d0 con 0x7f46ec100e90 2024-04-22T13:31:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.188+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ec108210 con 0x7f46ec100e90 2024-04-22T13:31:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.189+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f46dc017070 con 0x7f46ec100e90 2024-04-22T13:31:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.190+0000 7f46db7fe640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 0x7f46d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.190+0000 7f46e9d9b640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 0x7f46d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.190+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f46dc0cf7d0 con 0x7f46ec100e90 2024-04-22T13:31:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.190+0000 7f46e9d9b640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 0x7f46d40a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.195+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46dc09c250 con 0x7f46ec100e90 2024-04-22T13:31:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:15 smithi132 ceph-mon[31531]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:15 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/202470986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:15 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1566994310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:15 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/972624114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:16.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.372+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f46ec062b50 con 0x7f46ec100e90 2024-04-22T13:31:16.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.372+0000 7f46db7fe640 1 -- 172.21.15.87:0/4093727512 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f46dc09bb50 con 0x7f46ec100e90 2024-04-22T13:31:16.368 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 msgr2=0x7f46d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 0x7f46d40a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0008040 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 msgr2=0x7f46ec1982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00b4a0 tx=0x7f46dc00b960 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 shutdown_connections 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f46d40a3bb0 0x7f46d40a6070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46ec1029a0 0x7f46ec19f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec101fe0 0x7f46ec198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 --2- 172.21.15.87:0/4093727512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46ec100e90 0x7f46ec1982f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 >> 172.21.15.87:0/4093727512 conn(0x7f46ec0fc400 msgr2=0x7f46ec1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 shutdown_connections 2024-04-22T13:31:16.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.374+0000 7f46f0fb1640 1 -- 172.21.15.87:0/4093727512 wait complete. 2024-04-22T13:31:16.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:16.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- 172.21.15.87:0/930043926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20102a30 msgr2=0x7f1c20104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 --2- 172.21.15.87:0/930043926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20102a30 0x7f1c20104e20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009900 tx=0x7f1c1002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- 172.21.15.87:0/930043926 shutdown_connections 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 --2- 172.21.15.87:0/930043926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20107c90 0x7f1c2010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 --2- 172.21.15.87:0/930043926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20105360 0x7f1c20107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 --2- 172.21.15.87:0/930043926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20102a30 0x7f1c20104e20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- 172.21.15.87:0/930043926 >> 172.21.15.87:0/930043926 conn(0x7f1c200fc440 msgr2=0x7f1c200fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- 172.21.15.87:0/930043926 shutdown_connections 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- 172.21.15.87:0/930043926 wait complete. 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 Processor -- start 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.487+0000 7f1c25631640 1 -- start start 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20102a30 0x7f1c201982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 0x7f1c2019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010dba0 con 0x7f1c20105360 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010da20 con 0x7f1c20102a30 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010dd20 con 0x7f1c20107c90 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54584/0 (socket says 172.21.15.87:54584) 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 -- 172.21.15.87:0/1776251824 learned_addr learned my addr 172.21.15.87:0/1776251824 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1f7fe640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 0x7f1c2019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 -- 172.21.15.87:0/1776251824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 msgr2=0x7f1c2019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1effd640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20102a30 0x7f1c201982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 0x7f1c2019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 -- 172.21.15.87:0/1776251824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20102a30 msgr2=0x7f1c201982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20102a30 0x7f1c201982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 -- 172.21.15.87:0/1776251824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2019fe30 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1f7fe640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 0x7f1c2019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c1e7fc640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c970 tx=0x7f1c1400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14014070 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14004440 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c201a0120 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14003e50 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.488+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c201a06f0 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.489+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c200fdf30 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.490+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1c14017070 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.491+0000 7f1c07fff640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 0x7f1c080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.491+0000 7f1c1effd640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 0x7f1c080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.491+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f1c140cff90 con 0x7f1c20105360 2024-04-22T13:31:16.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.491+0000 7f1c1effd640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 0x7f1c080a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1c10005c40 tx=0x7f1c10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.495+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c1401e030 con 0x7f1c20105360 2024-04-22T13:31:16.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.671+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1c2002d830 con 0x7f1c20105360 2024-04-22T13:31:16.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.672+0000 7f1c07fff640 1 -- 172.21.15.87:0/1776251824 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f1c1400de20 con 0x7f1c20105360 2024-04-22T13:31:16.668 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:16.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 msgr2=0x7f1c080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 0x7f1c080a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1c10005c40 tx=0x7f1c10005fd0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 msgr2=0x7f1c201987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c970 tx=0x7f1c1400ce30 comp rx=0 tx=0).stop 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 shutdown_connections 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f1c080a3d50 0x7f1c080a6210 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c20107c90 0x7f1c2019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c20105360 0x7f1c201987f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 --2- 172.21.15.87:0/1776251824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20102a30 0x7f1c201982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 >> 172.21.15.87:0/1776251824 conn(0x7f1c200fc440 msgr2=0x7f1c20103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 shutdown_connections 2024-04-22T13:31:16.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.674+0000 7f1c25631640 1 -- 172.21.15.87:0/1776251824 wait complete. 2024-04-22T13:31:16.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 -- 172.21.15.87:0/4192111013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103180 msgr2=0x7f452c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 --2- 172.21.15.87:0/4192111013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103180 0x7f452c103600 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f451c009900 tx=0x7f451c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 -- 172.21.15.87:0/4192111013 shutdown_connections 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 --2- 172.21.15.87:0/4192111013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c103b40 0x7f452c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 --2- 172.21.15.87:0/4192111013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103180 0x7f452c103600 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 --2- 172.21.15.87:0/4192111013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c101f90 0x7f452c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 -- 172.21.15.87:0/4192111013 >> 172.21.15.87:0/4192111013 conn(0x7f452c0fd500 msgr2=0x7f452c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 -- 172.21.15.87:0/4192111013 shutdown_connections 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.784+0000 7f4533b15640 1 -- 172.21.15.87:0/4192111013 wait complete. 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 Processor -- start 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 -- start start 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c101f90 0x7f452c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c103180 0x7f452c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10dc00 con 0x7f452c103b40 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10da80 con 0x7f452c101f90 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f4533b15640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10dd80 con 0x7f452c103180 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54614/0 (socket says 172.21.15.87:54614) 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 -- 172.21.15.87:0/2933408706 learned_addr learned my addr 172.21.15.87:0/2933408706 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:16.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45318b1640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c101f90 0x7f452c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45310b0640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c103180 0x7f452c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 -- 172.21.15.87:0/2933408706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c103180 msgr2=0x7f452c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c103180 0x7f452c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 -- 172.21.15.87:0/2933408706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c101f90 msgr2=0x7f452c198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c101f90 0x7f452c198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 -- 172.21.15.87:0/2933408706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c19ff50 con 0x7f452c103b40 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.785+0000 7f45320b2640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f452400b970 tx=0x7f452400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.786+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524019070 con 0x7f452c103b40 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.786+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4524014440 con 0x7f452c103b40 2024-04-22T13:31:16.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.786+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c1a0240 con 0x7f452c103b40 2024-04-22T13:31:16.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.786+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45240075d0 con 0x7f452c103b40 2024-04-22T13:31:16.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.786+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c1a0810 con 0x7f452c103b40 2024-04-22T13:31:16.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.787+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c1092d0 con 0x7f452c103b40 2024-04-22T13:31:16.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.788+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f452401c070 con 0x7f452c103b40 2024-04-22T13:31:16.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.788+0000 7f4522ffd640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 0x7f45180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:16.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.788+0000 7f45318b1640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 0x7f45180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:16.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.788+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f45240cfdb0 con 0x7f452c103b40 2024-04-22T13:31:16.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.789+0000 7f45318b1640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 0x7f45180a60f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f452c102fe0 tx=0x7f4528009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:16.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:16.792+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f452409c830 con 0x7f452c103b40 2024-04-22T13:31:16.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:16 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1962804929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:16.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:16 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4093727512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:16.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:16 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1776251824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.010+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f452c02d5c0 con 0x7f452c103b40 2024-04-22T13:31:17.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.011+0000 7f4522ffd640 1 -- 172.21.15.87:0/2933408706 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7f452409c130 con 0x7f452c103b40 2024-04-22T13:31:17.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:17.007 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:31:17.007 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:17.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 msgr2=0x7f45180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 0x7f45180a60f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f452c102fe0 tx=0x7f4528009110 comp rx=0 tx=0).stop 2024-04-22T13:31:17.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 msgr2=0x7f452c19f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f452400b970 tx=0x7f452400bd70 comp rx=0 tx=0).stop 2024-04-22T13:31:17.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 shutdown_connections 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f45180a3c30 0x7f45180a60f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f452c103b40 0x7f452c19f8e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f452c103180 0x7f452c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 --2- 172.21.15.87:0/2933408706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c101f90 0x7f452c198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.013+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 >> 172.21.15.87:0/2933408706 conn(0x7f452c0fd500 msgr2=0x7f452c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.014+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 shutdown_connections 2024-04-22T13:31:17.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.014+0000 7f4533b15640 1 -- 172.21.15.87:0/2933408706 wait complete. 2024-04-22T13:31:17.021 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- 172.21.15.87:0/2148911496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c102aa0 msgr2=0x7f900c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 --2- 172.21.15.87:0/2148911496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c102aa0 0x7f900c104e90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9008009900 tx=0x7f900802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- 172.21.15.87:0/2148911496 shutdown_connections 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 --2- 172.21.15.87:0/2148911496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c107d00 0x7f900c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 --2- 172.21.15.87:0/2148911496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c1053d0 0x7f900c1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 --2- 172.21.15.87:0/2148911496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c102aa0 0x7f900c104e90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- 172.21.15.87:0/2148911496 >> 172.21.15.87:0/2148911496 conn(0x7f900c0fc470 msgr2=0x7f900c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- 172.21.15.87:0/2148911496 shutdown_connections 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- 172.21.15.87:0/2148911496 wait complete. 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 Processor -- start 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.123+0000 7f9013f96640 1 -- start start 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 0x7f900c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c107d00 0x7f900c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f900c10db00 con 0x7f900c1053d0 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f900c10d980 con 0x7f900c107d00 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f900c10dc80 con 0x7f900c102aa0 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54620/0 (socket says 172.21.15.87:54620) 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 -- 172.21.15.87:0/4277856818 learned_addr learned my addr 172.21.15.87:0/4277856818 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011d32640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 0x7f900c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9012533640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c107d00 0x7f900c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 -- 172.21.15.87:0/4277856818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 msgr2=0x7f900c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 0x7f900c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 -- 172.21.15.87:0/4277856818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c107d00 msgr2=0x7f900c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c107d00 0x7f900c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 -- 172.21.15.87:0/4277856818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c19fce0 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011d32640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 0x7f900c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9011531640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c970 tx=0x7f8ffc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc014070 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ffc004440 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc003e50 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f900c19ffd0 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.124+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f900c1a04e0 con 0x7f900c1053d0 2024-04-22T13:31:17.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.125+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f900c0fdf60 con 0x7f900c1053d0 2024-04-22T13:31:17.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.127+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8ffc017070 con 0x7f900c1053d0 2024-04-22T13:31:17.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.127+0000 7f9002ffd640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 0x7f8ff80a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.128+0000 7f9011d32640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 0x7f8ff80a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.128+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f8ffc0d05e0 con 0x7f900c1053d0 2024-04-22T13:31:17.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.128+0000 7f9011d32640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 0x7f8ff80a6290 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9008000c00 tx=0x7f90080095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.131+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ffc01e020 con 0x7f900c1053d0 2024-04-22T13:31:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:16 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1962804929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:16 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4093727512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:16 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1776251824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:16 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1962804929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:16 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4093727512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:16 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1776251824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:17.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.310+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f900c02d810 con 0x7f900c1053d0 2024-04-22T13:31:17.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.312+0000 7f9002ffd640 1 -- 172.21.15.87:0/4277856818 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7f8ffc09d0e0 con 0x7f900c1053d0 2024-04-22T13:31:17.307 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:17.308 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:17.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.313+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 msgr2=0x7f8ff80a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.313+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 0x7f8ff80a6290 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9008000c00 tx=0x7f90080095f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.313+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 msgr2=0x7f900c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.313+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c970 tx=0x7f8ffc00ce30 comp rx=0 tx=0).stop 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 shutdown_connections 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8ff80a3dd0 0x7f8ff80a6290 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f900c107d00 0x7f900c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f900c1053d0 0x7f900c198600 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 --2- 172.21.15.87:0/4277856818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f900c102aa0 0x7f900c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 >> 172.21.15.87:0/4277856818 conn(0x7f900c0fc470 msgr2=0x7f900c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 shutdown_connections 2024-04-22T13:31:17.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.314+0000 7f9013f96640 1 -- 172.21.15.87:0/4277856818 wait complete. 2024-04-22T13:31:17.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 -- 172.21.15.87:0/1337722634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c101ff0 msgr2=0x7fd65c102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 --2- 172.21.15.87:0/1337722634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c101ff0 0x7fd65c102470 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd650009900 tx=0x7fd65002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 -- 172.21.15.87:0/1337722634 shutdown_connections 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 --2- 172.21.15.87:0/1337722634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c1029b0 0x7fd65c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 --2- 172.21.15.87:0/1337722634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c101ff0 0x7fd65c102470 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 --2- 172.21.15.87:0/1337722634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 -- 172.21.15.87:0/1337722634 >> 172.21.15.87:0/1337722634 conn(0x7fd65c0fc410 msgr2=0x7fd65c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 -- 172.21.15.87:0/1337722634 shutdown_connections 2024-04-22T13:31:17.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.423+0000 7fd662025640 1 -- 172.21.15.87:0/1337722634 wait complete. 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 Processor -- start 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 -- start start 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c101ff0 0x7fd65c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c10db40 con 0x7fd65c1029b0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c10d9c0 con 0x7fd65c101ff0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd662025640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c10dcc0 con 0x7fd65c100ea0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54640/0 (socket says 172.21.15.87:54640) 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 -- 172.21.15.87:0/3209543054 learned_addr learned my addr 172.21.15.87:0/3209543054 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65b7fe640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65affd640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c101ff0 0x7fd65c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 -- 172.21.15.87:0/3209543054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 msgr2=0x7fd65c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 -- 172.21.15.87:0/3209543054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c101ff0 msgr2=0x7fd65c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c101ff0 0x7fd65c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65bfff640 1 -- 172.21.15.87:0/3209543054 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c19fe20 con 0x7fd65c1029b0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.424+0000 7fd65b7fe640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd65bfff640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd654009fd0 tx=0x7fd65400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654017070 con 0x7fd65c1029b0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd654003e20 con 0x7fd65c1029b0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654010460 con 0x7fd65c1029b0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c1a0110 con 0x7fd65c1029b0 2024-04-22T13:31:17.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.425+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c1a05e0 con 0x7fd65c1029b0 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.426+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c108220 con 0x7fd65c1029b0 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.427+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd654003970 con 0x7fd65c1029b0 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.427+0000 7fd658ff9640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 0x7fd6440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.427+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd6540d01d0 con 0x7fd65c1029b0 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.427+0000 7fd65b7fe640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 0x7fd6440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.428+0000 7fd65b7fe640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 0x7fd6440a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0058c0 tx=0x7fd64c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.432+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd65409cc50 con 0x7fd65c1029b0 2024-04-22T13:31:17.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.648+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd65c02d5c0 con 0x7fd65c1029b0 2024-04-22T13:31:17.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.649+0000 7fd658ff9640 1 -- 172.21.15.87:0/3209543054 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7fd65409c550 con 0x7fd65c1029b0 2024-04-22T13:31:17.644 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:17.644 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:31:17.645 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:17.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 msgr2=0x7fd6440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 0x7fd6440a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0058c0 tx=0x7fd64c005850 comp rx=0 tx=0).stop 2024-04-22T13:31:17.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 msgr2=0x7fd65c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd654009fd0 tx=0x7fd65400b260 comp rx=0 tx=0).stop 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 shutdown_connections 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd6440a3bb0 0x7fd6440a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd65c1029b0 0x7fd65c19f7b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd65c101ff0 0x7fd65c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 --2- 172.21.15.87:0/3209543054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd65c100ea0 0x7fd65c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.650+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 >> 172.21.15.87:0/3209543054 conn(0x7fd65c0fc410 msgr2=0x7fd65c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.651+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 shutdown_connections 2024-04-22T13:31:17.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.651+0000 7fd662025640 1 -- 172.21.15.87:0/3209543054 wait complete. 2024-04-22T13:31:17.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T13:29:46.666137+0000', 'modified': '2024-04-22T13:29:51.322041+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24482, 'mds_1': 24475, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24407': {'gid': 24407, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1829745193', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1829745193}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1829745193}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6837/3391616456', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 3391616456}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 3391616456}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6835/1119314155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1119314155}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1119314155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.132:6837/453938062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 453938062}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 453938062}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.132:6839/2904643536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2904643536}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2904643536}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6839/2692231328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 2692231328}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 2692231328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24482, 'qdb_cluster': [24460, 24482, 24475]} 2024-04-22T13:31:17.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T13:31:17.659 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 -- 172.21.15.87:0/157160493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04107c90 msgr2=0x7f0f0410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/157160493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04107c90 0x7f0f0410a0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009ca0 tx=0x7f0efc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 -- 172.21.15.87:0/157160493 shutdown_connections 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/157160493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04107c90 0x7f0f0410a0e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/157160493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04105360 0x7f0f04107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/157160493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04102a30 0x7f0f04104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 -- 172.21.15.87:0/157160493 >> 172.21.15.87:0/157160493 conn(0x7f0f040fc440 msgr2=0x7f0f040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 -- 172.21.15.87:0/157160493 shutdown_connections 2024-04-22T13:31:17.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.761+0000 7f0f0abf4640 1 -- 172.21.15.87:0/157160493 wait complete. 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 Processor -- start 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 -- start start 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 0x7f0f041981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04107c90 0x7f0f0419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410daf0 con 0x7f0f04105360 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410d970 con 0x7f0f04102a30 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f0abf4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0410dc70 con 0x7f0f04107c90 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54648/0 (socket says 172.21.15.87:54648) 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 -- 172.21.15.87:0/796975750 learned_addr learned my addr 172.21.15.87:0/796975750 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f08990640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 0x7f0f041981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 -- 172.21.15.87:0/796975750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04107c90 msgr2=0x7f0f0419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f09191640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04107c90 0x7f0f0419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04107c90 0x7f0f0419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 -- 172.21.15.87:0/796975750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 msgr2=0x7f0f041981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 0x7f0f041981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 -- 172.21.15.87:0/796975750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0419fde0 con 0x7f0f04105360 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f08990640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 0x7f0f041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.762+0000 7f0f03fff640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c970 tx=0x7f0ef000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.763+0000 7f0f01ffb640 1 -- 172.21.15.87:0/796975750 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0007bb0 con 0x7f0f04105360 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.763+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f041a00d0 con 0x7f0f04105360 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.763+0000 7f0f01ffb640 1 -- 172.21.15.87:0/796975750 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef0007d50 con 0x7f0f04105360 2024-04-22T13:31:17.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.763+0000 7f0f01ffb640 1 -- 172.21.15.87:0/796975750 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000f450 con 0x7f0f04105360 2024-04-22T13:31:17.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.763+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f041a0620 con 0x7f0f04105360 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f01ffb640 1 -- 172.21.15.87:0/796975750 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0ef0004f20 con 0x7f0f04105360 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 Processor -- start 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 -- start start 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f01ffb640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 0x7f0ef40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7b1fb10 con 0x55fdf7bbf250 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7abd240 con 0x55fdf7bbfc40 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7b1bad0 con 0x55fdf7bc06e0 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f08990640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.765+0000 7f0f09191640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 0x7f0ef40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54656/0 (socket says 172.21.15.87:54656) 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 -- 172.21.15.87:0/1815965403 learned_addr learned my addr 172.21.15.87:0/1815965403 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f01ffb640 1 -- 172.21.15.87:0/796975750 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0ef00d0430 con 0x7f0f04105360 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 -- 172.21.15.87:0/1815965403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 msgr2=0x55fdf7bc72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 -- 172.21.15.87:0/1815965403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbfc40 msgr2=0x55fdf7bc00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f03fff640 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 -- 172.21.15.87:0/1815965403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf78322d0 con 0x55fdf7bbf250 2024-04-22T13:31:17.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f09191640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 0x7f0ef40a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0efc037040 tx=0x7f0efc005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0f08990640 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800bcc0 tx=0x7f0ef800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f69cbf177f24ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0eeeffd640 1 -- 172.21.15.87:0/1815965403 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8015070 con 0x55fdf7bbf250 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0eeeffd640 1 -- 172.21.15.87:0/1815965403 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef801a910 con 0x55fdf7bbf250 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.766+0000 7f0eeeffd640 1 -- 172.21.15.87:0/1815965403 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8011920 con 0x55fdf7bbf250 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/1815965403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 msgr2=0x55fdf7bbf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800bcc0 tx=0x7f0ef800f730 comp rx=0 tx=0).stop 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/1815965403 shutdown_connections 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/1815965403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbf250 0x55fdf7bbf670 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/1815965403 >> 172.21.15.87:0/1815965403 conn(0x55fdf7bb6f40 msgr2=0x55fdf7bb5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/1815965403 shutdown_connections 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/1815965403 wait complete. 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 -- start start 2024-04-22T13:31:17.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.767+0000 7f0f0d7ef740 1 Processor -- start 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bd07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bc78c0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7aa8190 con 0x55fdf7bbf250 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bc7a40 con 0x55fdf7bc06e0 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 10 client.0 _ll_register_callbacks cb 0x55fdf7aa38b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54664/0 (socket says 172.21.15.87:54664) 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 -- 172.21.15.87:0/203513862 learned_addr learned my addr 172.21.15.87:0/203513862 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f09191640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bd07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f08990640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 -- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 msgr2=0x55fdf7bd07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bd07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 -- 172.21.15.87:0/203513862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 msgr2=0x55fdf7bc9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 -- 172.21.15.87:0/203513862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf78fc6b0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f03fff640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00afcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12951916aaf018b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef001d040 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0ef00055d0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef801a910 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0ef00ba050 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 10 client.15075 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.768+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fdf7bd3c20 con 0x55fdf7bbfc40 2024-04-22T13:31:17.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef801a910 con 0x55fdf7bbfc40 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0ef0087070 con 0x55fdf7bbfc40 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 10 client.15075 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 10 client.15075 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 20 client.15075 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7bd7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 4 client.15075 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdf7b2d530 con 0x55fdf7bd62a0 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f08990640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7bd7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0ee3fff640 10 client.15075 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.769+0000 7f0f08990640 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7bd7710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ff8a6643b075210 server_cookie=3479cb337e30d083 in_seq=0 out_seq=0 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0ee3fff640 1 -- 172.21.15.87:0/203513862 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdf7b2d530 con 0x55fdf7bd62a0 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0ee3fff640 10 client.15075 handle_command_reply: tid=0 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 client.15075 shutdown 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 msgr2=0x55fdf7bc97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00afcf0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 shutdown_connections 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7bd7710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bd07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0ee3fff640 0 client.15075 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bc97a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/203513862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.770+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 >> 172.21.15.87:0/203513862 conn(0x55fdf7bb6f40 msgr2=0x55fdf7bb4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.771+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 shutdown_connections 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.771+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/203513862 wait complete. 2024-04-22T13:31:17.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.771+0000 7f0f0d7ef740 20 client.15075 trim_cache size 0 max 16384 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 Processor -- start 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 -- start start 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd5a50 con 0x55fdf7bbfc40 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd6120 con 0x55fdf7bbf250 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.788+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd67f0 con 0x55fdf7bc06e0 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54672/0 (socket says 172.21.15.87:54672) 2024-04-22T13:31:17.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 -- 172.21.15.87:0/4224716599 learned_addr learned my addr 172.21.15.87:0/4224716599 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f09191640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 -- 172.21.15.87:0/4224716599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 msgr2=0x55fdf7bc3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 -- 172.21.15.87:0/4224716599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 msgr2=0x55fdf7bc7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f08990640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 -- 172.21.15.87:0/4224716599 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf7bc6eb0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f09191640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0099180 tx=0x7f0ef00af9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60120d890260625 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0eee7fc640 1 -- 172.21.15.87:0/4224716599 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef001d070 con 0x55fdf7bbfc40 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0eee7fc640 1 -- 172.21.15.87:0/4224716599 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef009ceb0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.789+0000 7f0eee7fc640 1 -- 172.21.15.87:0/4224716599 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef009aab0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4224716599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 msgr2=0x55fdf7bd07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0099180 tx=0x7f0ef00af9d0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4224716599 shutdown_connections 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bc3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bd07a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4224716599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bc7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4224716599 >> 172.21.15.87:0/4224716599 conn(0x55fdf7bb6f40 msgr2=0x55fdf7bb5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4224716599 shutdown_connections 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4224716599 wait complete. 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- start start 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 Processor -- start 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bcba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bcef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd7590 con 0x55fdf7bbfc40 2024-04-22T13:31:17.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd6ec0 con 0x55fdf7bbf250 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.790+0000 7f0f0d7ef740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fdf7bd7c60 con 0x55fdf7bc06e0 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f0d7ef740 10 client.0 _ll_register_callbacks cb 0x7f0ef0098390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54680/0 (socket says 172.21.15.87:54680) 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 -- 172.21.15.87:0/4182608164 learned_addr learned my addr 172.21.15.87:0/4182608164 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f09191640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bcef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f08990640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bcba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 -- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 msgr2=0x55fdf7bcef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bcef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 -- 172.21.15.87:0/4182608164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 msgr2=0x55fdf7bcba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bcba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 -- 172.21.15.87:0/4182608164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf7bcf580 con 0x55fdf7bbfc40 2024-04-22T13:31:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f03fff640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666f49cc8ef0abee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0066070 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0ef00d0040 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef009ceb0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0ef001d070 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f0d7ef740 10 client.15081 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fdf7bd0880 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ef009ceb0 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.791+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0ef00ae050 con 0x55fdf7bbfc40 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 10 client.15081 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 10 client.15081 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 20 client.15081 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7c3f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 4 client.15081 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "status"}) -- 0x55fdf7b273a0 con 0x55fdf7bd62a0 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f08990640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7c3f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f08990640 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7c3f130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fde5606244a41a8b server_cookie=e9f2aade4d98c047 in_seq=0 out_seq=0 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0ee3fff640 10 client.15081 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0ee3fff640 1 -- 172.21.15.87:0/4182608164 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fdf7b273a0 con 0x55fdf7bd62a0 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0ee3fff640 10 client.15081 handle_command_reply: tid=0 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 client.15081 shutdown 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 msgr2=0x55fdf7bcbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef00057e0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 shutdown_connections 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55fdf7bd62a0 0x55fdf7c3f130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.792+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fdf7bc06e0 0x55fdf7bcef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0ee3fff640 0 client.15081 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:17.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fdf7bbfc40 0x55fdf7bcbf70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 1 --2- 172.21.15.87:0/4182608164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fdf7bbf250 0x55fdf7bcba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 >> 172.21.15.87:0/4182608164 conn(0x55fdf7bb6f40 msgr2=0x55fdf7bc3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 shutdown_connections 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 1 -- 172.21.15.87:0/4182608164 wait complete. 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0d7ef740 20 client.15081 trim_cache size 0 max 16384 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 2, 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24460, 2024-04-22T13:31:17.796 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 87.473231811999995, 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 111.17610882699999 2024-04-22T13:31:17.797 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 msgr2=0x7f0ef40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 0x7f0ef40a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0efc037040 tx=0x7f0efc005ef0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 msgr2=0x7f0f04198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c970 tx=0x7f0ef000ce30 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 shutdown_connections 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0ef40a3bb0 0x7f0ef40a6070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f04107c90 0x7f0f0419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f04105360 0x7f0f04198700 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 --2- 172.21.15.87:0/796975750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f04102a30 0x7f0f041981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 >> 172.21.15.87:0/796975750 conn(0x7f0f040fc440 msgr2=0x7f0f04103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 shutdown_connections 2024-04-22T13:31:17.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.793+0000 7f0f0abf4640 1 -- 172.21.15.87:0/796975750 wait complete. 2024-04-22T13:31:17.800 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 -- 172.21.15.87:0/2830763370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1810103580 msgr2=0x7f18101039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 --2- 172.21.15.87:0/2830763370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1810103580 0x7f18101039a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1808009ca0 tx=0x7f180802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 -- 172.21.15.87:0/2830763370 shutdown_connections 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 --2- 172.21.15.87:0/2830763370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18101009b0 0x7f181010a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 --2- 172.21.15.87:0/2830763370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18100ffff0 0x7f1810100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 --2- 172.21.15.87:0/2830763370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1810103580 0x7f18101039a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 -- 172.21.15.87:0/2830763370 >> 172.21.15.87:0/2830763370 conn(0x7f18100fbb90 msgr2=0x7f18100fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 -- 172.21.15.87:0/2830763370 shutdown_connections 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.913+0000 7f1814881640 1 -- 172.21.15.87:0/2830763370 wait complete. 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 Processor -- start 2024-04-22T13:31:17.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- start start 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 0x7f1810198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 0x7f181019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f181010dbb0 con 0x7f18100ffff0 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f181010da30 con 0x7f18101009b0 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f181010dd30 con 0x7f1810103580 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54694/0 (socket says 172.21.15.87:54694) 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 -- 172.21.15.87:0/385439362 learned_addr learned my addr 172.21.15.87:0/385439362 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180dd9b640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 0x7f1810198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180ed9d640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 0x7f181019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 -- 172.21.15.87:0/385439362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 msgr2=0x7f181019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 0x7f181019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 -- 172.21.15.87:0/385439362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 msgr2=0x7f1810198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 0x7f1810198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 -- 172.21.15.87:0/385439362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181019ff00 con 0x7f18100ffff0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180dd9b640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 0x7f1810198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180ed9d640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 0x7f181019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f180e59c640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1808032380 tx=0x7f18080327f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f17ff7fe640 1 -- 172.21.15.87:0/385439362 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808041020 con 0x7f18100ffff0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18101a0190 con 0x7f18100ffff0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f17ff7fe640 1 -- 172.21.15.87:0/385439362 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1808003e20 con 0x7f18100ffff0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f17ff7fe640 1 -- 172.21.15.87:0/385439362 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808039040 con 0x7f18100ffff0 2024-04-22T13:31:17.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.914+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18101a0750 con 0x7f18100ffff0 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.916+0000 7f17ff7fe640 1 -- 172.21.15.87:0/385439362 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1808044020 con 0x7f18100ffff0 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 Processor -- start 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 -- start start 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f17ff7fe640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 0x7f17f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7503e40 0x561be7504260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be7504cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561be7468b10 con 0x561be75052d0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561be7406240 con 0x561be7503e40 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f1817479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561be7464ad0 con 0x561be7504830 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f180ed9d640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 0x7f17f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f17ff7fe640 1 -- 172.21.15.87:0/385439362 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f18080f2570 con 0x7f18100ffff0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f180ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f180ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54708/0 (socket says 172.21.15.87:54708) 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.917+0000 7f180ed9d640 1 -- 172.21.15.87:0/462465058 learned_addr learned my addr 172.21.15.87:0/462465058 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180e59c640 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7503e40 0x561be7504260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180dd9b640 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be7504cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 -- 172.21.15.87:0/462465058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 msgr2=0x561be7504cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be7504cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 -- 172.21.15.87:0/462465058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7503e40 msgr2=0x561be7504260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7503e40 0x561be7504260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 -- 172.21.15.87:0/462465058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561be717b2d0 con 0x561be75052d0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 0x7f17f80a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18000154f0 tx=0x7f1800009860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f180ed9d640 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f18000176e0 tx=0x7f1800018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3732e218fe68e225 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f17fcff9640 1 -- 172.21.15.87:0/462465058 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180000f040 con 0x561be75052d0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f17fcff9640 1 -- 172.21.15.87:0/462465058 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1800018b30 con 0x561be75052d0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.918+0000 7f17fcff9640 1 -- 172.21.15.87:0/462465058 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18000228b0 con 0x561be75052d0 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- 172.21.15.87:0/462465058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 msgr2=0x561be750beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f18000176e0 tx=0x7f1800018000 comp rx=0 tx=0).stop 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- 172.21.15.87:0/462465058 shutdown_connections 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be75052d0 0x561be750beb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be7504cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- 172.21.15.87:0/462465058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7503e40 0x561be7504260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- 172.21.15.87:0/462465058 >> 172.21.15.87:0/462465058 conn(0x561be74fff40 msgr2=0x561be74fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- 172.21.15.87:0/462465058 shutdown_connections 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- 172.21.15.87:0/462465058 wait complete. 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- start start 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 Processor -- start 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be750e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7515320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.919+0000 7f1817479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561be750c3f0 con 0x561be7503e40 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561be73f1190 con 0x561be75052d0 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561be750c570 con 0x561be7504830 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 10 client.0 _ll_register_callbacks cb 0x561be73ec8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54724/0 (socket says 172.21.15.87:54724) 2024-04-22T13:31:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 -- 172.21.15.87:0/759321948 learned_addr learned my addr 172.21.15.87:0/759321948 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180dd9b640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be750e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 -- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 msgr2=0x561be750e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be750e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 -- 172.21.15.87:0/759321948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 msgr2=0x561be7515320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7515320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 -- 172.21.15.87:0/759321948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561be72456b0 con 0x561be7503e40 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180dd9b640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be750e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f180e59c640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f18080c7030 tx=0x7f18080be4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671767cc46e316f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f17f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080beff0 con 0x561be7503e40 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f18080db2b0 con 0x561be7503e40 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1800018b30 con 0x561be7503e40 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f18080098b0 con 0x561be7503e40 2024-04-22T13:31:17.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 10 client.15093 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:17.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561be750c3f0 con 0x561be7503e40 2024-04-22T13:31:17.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.920+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1800018b30 con 0x561be7503e40 2024-04-22T13:31:17.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f180803c030 con 0x561be7503e40 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 10 client.15093 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 20 client.15093 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be751ae40 0x561be751c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 4 client.15093 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561be7476530 con 0x561be751ae40 2024-04-22T13:31:17.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f180dd9b640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be751ae40 0x561be751c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f17f5ffb640 10 client.15093 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.921+0000 7f180dd9b640 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be751ae40 0x561be751c2b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29d6f04f951ad3b2 server_cookie=67309ca83438f2a7 in_seq=0 out_seq=0 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.922+0000 7f17f5ffb640 1 -- 172.21.15.87:0/759321948 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561be7476530 con 0x561be751ae40 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.922+0000 7f17f5ffb640 10 client.15093 handle_command_reply: tid=0 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.922+0000 7f1817479740 1 client.15093 shutdown 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 msgr2=0x561be750dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f18080c7030 tx=0x7f18080be4f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 shutdown_connections 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be751ae40 0x561be751c2b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7515320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be750e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f17f5ffb640 0 client.15093 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 --2- 172.21.15.87:0/759321948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be750dde0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 >> 172.21.15.87:0/759321948 conn(0x561be74fff40 msgr2=0x561be74fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 shutdown_connections 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 1 -- 172.21.15.87:0/759321948 wait complete. 2024-04-22T13:31:17.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.923+0000 7f1817479740 20 client.15093 trim_cache size 0 max 16384 2024-04-22T13:31:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 Processor -- start 2024-04-22T13:31:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 -- start start 2024-04-22T13:31:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be75152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7519d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561be751a5f0 con 0x561be7503e40 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561be751acc0 con 0x561be75052d0 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f1817479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561be751b390 con 0x561be7504830 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.940+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54732/0 (socket says 172.21.15.87:54732) 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 -- 172.21.15.87:0/3938958389 learned_addr learned my addr 172.21.15.87:0/3938958389 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 -- 172.21.15.87:0/3938958389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 msgr2=0x561be75152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be75152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 -- 172.21.15.87:0/3938958389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 msgr2=0x561be7519d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7519d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 -- 172.21.15.87:0/3938958389 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561be750bac0 con 0x561be7503e40 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f180e59c640 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1808003fd0 tx=0x7f18080bce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14189656640a0ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f17f7fff640 1 -- 172.21.15.87:0/3938958389 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080be6c0 con 0x561be7503e40 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f17f7fff640 1 -- 172.21.15.87:0/3938958389 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18080a9320 con 0x561be7503e40 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f17f7fff640 1 -- 172.21.15.87:0/3938958389 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080bedb0 con 0x561be7503e40 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f1817479740 1 -- 172.21.15.87:0/3938958389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 msgr2=0x561be7519830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.941+0000 7f1817479740 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1808003fd0 tx=0x7f18080bce00 comp rx=0 tx=0).stop 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- 172.21.15.87:0/3938958389 shutdown_connections 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be75052d0 0x561be7519d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be7504830 0x561be75152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- 172.21.15.87:0/3938958389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7519830 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- 172.21.15.87:0/3938958389 >> 172.21.15.87:0/3938958389 conn(0x561be74fff40 msgr2=0x561be750dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- 172.21.15.87:0/3938958389 shutdown_connections 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- 172.21.15.87:0/3938958389 wait complete. 2024-04-22T13:31:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- start start 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 Processor -- start 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7504830 0x561be7515250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be75052d0 0x561be7509f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561be751c130 con 0x561be7503e40 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561be751ba60 con 0x561be7504830 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561be751c800 con 0x561be75052d0 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f1817479740 10 client.0 _ll_register_callbacks cb 0x7f18080bab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54742/0 (socket says 172.21.15.87:54742) 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 -- 172.21.15.87:0/1311581523 learned_addr learned my addr 172.21.15.87:0/1311581523 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 -- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be75052d0 msgr2=0x561be7509f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be75052d0 0x561be7509f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 -- 172.21.15.87:0/1311581523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7504830 msgr2=0x561be7515250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7504830 0x561be7515250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.942+0000 7f180e59c640 1 -- 172.21.15.87:0/1311581523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561be750a7a0 con 0x561be7503e40 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f180e59c640 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1808034cc0 tx=0x7f180802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5125b3e03ff33a3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080be6c0 con 0x561be7503e40 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f18080f20d0 con 0x561be7503e40 2024-04-22T13:31:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561be74703a0 con 0x561be7503e40 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f18080be8f0 con 0x561be7503e40 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 10 client.15099 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561be750a7a0 con 0x561be7503e40 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561be74703a0 con 0x561be7503e40 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f18080ba640 con 0x561be7503e40 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 10 client.15099 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 10 client.15099 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 20 client.15099 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be7585690 0x561be75843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 4 client.15099 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "status"}) -- 0x561be73ec960 con 0x561be7585690 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.943+0000 7f180dd9b640 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be7585690 0x561be75843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f180dd9b640 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be7585690 0x561be75843a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39d08842ec026af8 server_cookie=5d5162dc1f1ecf8f in_seq=0 out_seq=0 2024-04-22T13:31:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f17f5ffb640 10 client.15099 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f17f5ffb640 1 -- 172.21.15.87:0/1311581523 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561be73ec960 con 0x561be7585690 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f17f5ffb640 10 client.15099 handle_command_reply: tid=0 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 client.15099 shutdown 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 msgr2=0x561be7509a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1808034cc0 tx=0x7f180802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 shutdown_connections 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561be7585690 0x561be75843a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561be75052d0 0x561be7509f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561be7504830 0x561be7515250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f17f5ffb640 0 client.15099 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 --2- 172.21.15.87:0/1311581523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561be7503e40 0x561be7509a30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 >> 172.21.15.87:0/1311581523 conn(0x561be74fff40 msgr2=0x561be7519830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 shutdown_connections 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 1 -- 172.21.15.87:0/1311581523 wait complete. 2024-04-22T13:31:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.944+0000 7f1817479740 20 client.15099 trim_cache size 0 max 16384 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 1, 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24475, 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 88.637916794999995, 2024-04-22T13:31:17.948 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 105.688719582 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 msgr2=0x7f17f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 0x7f17f80a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18000154f0 tx=0x7f1800009860 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 msgr2=0x7f18101982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1808032380 tx=0x7f18080327f0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 shutdown_connections 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f17f80a3bb0 0x7f17f80a6070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1810103580 0x7f181019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18101009b0 0x7f1810198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 --2- 172.21.15.87:0/385439362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18100ffff0 0x7f18101982e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 >> 172.21.15.87:0/385439362 conn(0x7f18100fbb90 msgr2=0x7f1810106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 shutdown_connections 2024-04-22T13:31:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:17.945+0000 7f1814881640 1 -- 172.21.15.87:0/385439362 wait complete. 2024-04-22T13:31:17.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:17 smithi087 ceph-mon[27999]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:17.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:17 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2933408706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:17.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:17 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4277856818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:17.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:17 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3209543054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:17.952 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T13:31:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/492022452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 msgr2=0x7f7c041023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/492022452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041023c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009900 tx=0x7f7bf402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/492022452 shutdown_connections 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/492022452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/492022452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c04103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/492022452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041023c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/492022452 >> 172.21.15.87:0/492022452 conn(0x7f7c040fd530 msgr2=0x7f7c040ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/492022452 shutdown_connections 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/492022452 wait complete. 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 Processor -- start 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- start start 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c041988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0419f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0410dbf0 con 0x7f7c04101fa0 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0410da70 con 0x7f7c04103b50 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.056+0000 7f7c0b2f1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0410dd70 con 0x7f7c04103190 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54754/0 (socket says 172.21.15.87:54754) 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 -- 172.21.15.87:0/1224500281 learned_addr learned my addr 172.21.15.87:0/1224500281 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c041988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0419f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 -- 172.21.15.87:0/1224500281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 msgr2=0x7f7c041988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c041988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 -- 172.21.15.87:0/1224500281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 msgr2=0x7f7c0419f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0419f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 -- 172.21.15.87:0/1224500281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0419ffd0 con 0x7f7c04101fa0 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:18.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c041988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005ab0 tx=0x7f7bf40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7bfa7fc640 1 -- 172.21.15.87:0/1224500281 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4043020 con 0x7f7c04101fa0 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7bfa7fc640 1 -- 172.21.15.87:0/1224500281 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bf40054c0 con 0x7f7c04101fa0 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c041a0260 con 0x7f7c04101fa0 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7bfa7fc640 1 -- 172.21.15.87:0/1224500281 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4003de0 con 0x7f7c04101fa0 2024-04-22T13:31:18.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.057+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c041a0800 con 0x7f7c04101fa0 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.059+0000 7f7bfa7fc640 1 -- 172.21.15.87:0/1224500281 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7bf4046020 con 0x7f7c04101fa0 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 Processor -- start 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 -- start start 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04791260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f04798eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558f046f5b10 con 0x558f04791830 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f04693240 con 0x558f04790e40 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f046f1ad0 con 0x558f047922d0 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0908d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7bfa7fc640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 0x7f7bf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0908d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54770/0 (socket says 172.21.15.87:54770) 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.060+0000 7f7c0908d640 1 -- 172.21.15.87:0/1326733633 learned_addr learned my addr 172.21.15.87:0/1326733633 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 0x7f7bf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04791260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f04798eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 -- 172.21.15.87:0/1326733633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 msgr2=0x558f04798eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f04798eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 -- 172.21.15.87:0/1326733633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 msgr2=0x558f04791260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04791260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7bfa7fc640 1 -- 172.21.15.87:0/1224500281 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7bf40f2910 con 0x7f7c04101fa0 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 -- 172.21.15.87:0/1326733633 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f044082d0 con 0x558f04791830 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f04798eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4083ec0 tx=0x7f7bf40ba890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9f8879c8f5fb68 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 0x7f7bf00a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x558f0478c200 tx=0x7f7bfc0094d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7bef7fe640 1 -- 172.21.15.87:0/1326733633 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf40be730 con 0x558f04791830 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7bef7fe640 1 -- 172.21.15.87:0/1326733633 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bf4004c50 con 0x558f04791830 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.061+0000 7f7bef7fe640 1 -- 172.21.15.87:0/1326733633 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf40be550 con 0x558f04791830 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1326733633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 msgr2=0x558f04791cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4083ec0 tx=0x7f7bf40ba890 comp rx=0 tx=0).stop 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1326733633 shutdown_connections 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f04798eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04791cb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1326733633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04791260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1326733633 >> 172.21.15.87:0/1326733633 conn(0x558f0478cf40 msgr2=0x558f0478b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1326733633 shutdown_connections 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1326733633 wait complete. 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 -- start start 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 Processor -- start 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.062+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558f04693240 con 0x558f04791830 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f0467e190 con 0x558f04790e40 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f046f5b10 con 0x558f047922d0 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 10 client.0 _ll_register_callbacks cb 0x558f046798b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54778/0 (socket says 172.21.15.87:54778) 2024-04-22T13:31:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 -- 172.21.15.87:0/2096973688 learned_addr learned my addr 172.21.15.87:0/2096973688 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0988e640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 -- 172.21.15.87:0/2096973688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 msgr2=0x558f047a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 -- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 msgr2=0x558f0479a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0908d640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 -- 172.21.15.87:0/2096973688 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f044d26b0 con 0x558f04791830 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0888c640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000e170 tx=0x7f7c0000e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215c06ec3c2569bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7becff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00017020 con 0x558f04791830 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7c00013410 con 0x558f04791830 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf4004c50 con 0x558f04791830 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c0000f040 con 0x558f04791830 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 10 client.15111 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.063+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558f047a5290 con 0x558f04791830 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bf4004c50 con 0x558f04791830 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f7c00012050 con 0x558f04791830 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 10 client.15111 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 20 client.15111 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f047a7a10 0x558f047a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 4 client.15111 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f04703530 con 0x558f047a7a10 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0908d640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f047a7a10 0x558f047a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7c0908d640 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f047a7a10 0x558f047a8e80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8b1e4bc4b8696c server_cookie=dd8fdb8b28f14c95 in_seq=0 out_seq=0 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.064+0000 7f7becff9640 10 client.15111 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7becff9640 1 -- 172.21.15.87:0/2096973688 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f04703530 con 0x558f047a7a10 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7becff9640 10 client.15111 handle_command_reply: tid=0 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 client.15111 shutdown 2024-04-22T13:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 msgr2=0x558f0479af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000e170 tx=0x7f7c0000e630 comp rx=0 tx=0).stop 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 shutdown_connections 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f047a7a10 0x558f047a8e80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7becff9640 0 client.15111 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f0479af10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/2096973688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 >> 172.21.15.87:0/2096973688 conn(0x558f0478cf40 msgr2=0x558f0478aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.065+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 shutdown_connections 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.066+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/2096973688 wait complete. 2024-04-22T13:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.066+0000 7f7c0dfb6740 20 client.15111 trim_cache size 0 max 16384 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 Processor -- start 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 -- start start 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04795850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558f047a71c0 con 0x558f04791830 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f047a7890 con 0x558f04790e40 2024-04-22T13:31:18.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.083+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f047a7f60 con 0x558f047922d0 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54794/0 (socket says 172.21.15.87:54794) 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 -- 172.21.15.87:0/3117397935 learned_addr learned my addr 172.21.15.87:0/3117397935 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0988e640 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 -- 172.21.15.87:0/3117397935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 msgr2=0x558f047a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 -- 172.21.15.87:0/3117397935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 msgr2=0x558f04795850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04795850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 -- 172.21.15.87:0/3117397935 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f04786e70 con 0x558f04791830 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0988e640 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:18.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7c0888c640 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c00001eb0 tx=0x7f7c000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa725fe9cc38a3d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7beeffd640 1 -- 172.21.15.87:0/3117397935 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00017040 con 0x558f04791830 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7beeffd640 1 -- 172.21.15.87:0/3117397935 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0001edb0 con 0x558f04791830 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.084+0000 7f7beeffd640 1 -- 172.21.15.87:0/3117397935 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00027440 con 0x558f04791830 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/3117397935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 msgr2=0x558f04795d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c00001eb0 tx=0x7f7c000046b0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/3117397935 shutdown_connections 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f04795d90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/3117397935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f04795850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/3117397935 >> 172.21.15.87:0/3117397935 conn(0x558f0478cf40 msgr2=0x558f047a6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/3117397935 shutdown_connections 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/3117397935 wait complete. 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 -- start start 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.085+0000 7f7c0dfb6740 1 Processor -- start 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558f047a8d00 con 0x558f04791830 2024-04-22T13:31:18.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f047a8630 con 0x558f04790e40 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f047a93d0 con 0x558f047922d0 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 10 client.0 _ll_register_callbacks cb 0x7f7c00009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54798/0 (socket says 172.21.15.87:54798) 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 -- 172.21.15.87:0/1662469541 learned_addr learned my addr 172.21.15.87:0/1662469541 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0988e640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 -- 172.21.15.87:0/1662469541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 msgr2=0x558f047a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 -- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 msgr2=0x558f0479e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 -- 172.21.15.87:0/1662469541 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f047a13e0 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0888c640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7c0001e940 tx=0x7f7c0000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea777e1eec18149a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7becff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00037020 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7c00013610 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c0001edb0 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c00017040 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 10 client.15117 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.086+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558f0480db80 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0001edb0 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f7c00012070 con 0x558f04791830 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 10 client.15117 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 10 client.15117 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 20 client.15117 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f04812430 0x558f04811140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 4 client.15117 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "status"}) -- 0x558f046fd3a0 con 0x558f04812430 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f04812430 0x558f04811140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7becff9640 10 client.15117 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.087+0000 7f7c0908d640 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f04812430 0x558f04811140 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74958155b46824b1 server_cookie=822b8702e6ad9f49 in_seq=0 out_seq=0 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7becff9640 1 -- 172.21.15.87:0/1662469541 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x558f046fd3a0 con 0x558f04812430 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7becff9640 10 client.15117 handle_command_reply: tid=0 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 client.15117 shutdown 2024-04-22T13:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 msgr2=0x558f047a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7c0001e940 tx=0x7f7c0000ec70 comp rx=0 tx=0).stop 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 shutdown_connections 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558f04812430 0x558f04811140 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f047922d0 0x558f047a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7becff9640 0 client.15117 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558f04791830 0x558f047a0580 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 --2- 172.21.15.87:0/1662469541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f04790e40 0x558f0479e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 >> 172.21.15.87:0/1662469541 conn(0x558f0478cf40 msgr2=0x558f0479a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 shutdown_connections 2024-04-22T13:31:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 1 -- 172.21.15.87:0/1662469541 wait complete. 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0dfb6740 20 client.15117 trim_cache size 0 max 16384 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 0, 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24482, 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:18.095 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 90.761911067, 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 97.211389976999996 2024-04-22T13:31:18.096 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 msgr2=0x7f7bf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 0x7f7bf00a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x558f0478c200 tx=0x7f7bfc0094d0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 msgr2=0x7f7c041983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005ab0 tx=0x7f7bf40042f0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 shutdown_connections 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f7bf00a3bb0 0x7f7bf00a6070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c04103b50 0x7f7c0419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c04103190 0x7f7c041988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 --2- 172.21.15.87:0/1224500281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7c04101fa0 0x7f7c041983b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.088+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 >> 172.21.15.87:0/1224500281 conn(0x7f7c040fd530 msgr2=0x7f7c04106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.089+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 shutdown_connections 2024-04-22T13:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:18.089+0000 7f7c0b2f1640 1 -- 172.21.15.87:0/1224500281 wait complete. 2024-04-22T13:31:18.099 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T13:31:18.099 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T13:31:18.109 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:31:18.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T13:31:18.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:17 smithi132 ceph-mon[31531]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:18.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:17 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2933408706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.114 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:17 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4277856818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.114 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:17 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3209543054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:17 smithi120 ceph-mon[31101]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:17 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2933408706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:17 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4277856818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:17 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3209543054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:20.103 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:19 smithi087 ceph-mon[27999]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:20.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:19 smithi120 ceph-mon[31101]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 -- 172.21.15.87:0/2727567026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854105e60 msgr2=0x7f6854108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 --2- 172.21.15.87:0/2727567026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854105e60 0x7f6854108250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6848009990 tx=0x7f6848009ca0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 -- 172.21.15.87:0/2727567026 shutdown_connections 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 --2- 172.21.15.87:0/2727567026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854108790 0x7f685410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 --2- 172.21.15.87:0/2727567026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854103530 0x7f6854105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 --2- 172.21.15.87:0/2727567026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854105e60 0x7f6854108250 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 -- 172.21.15.87:0/2727567026 >> 172.21.15.87:0/2727567026 conn(0x7f68540fcf40 msgr2=0x7f68540ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.210+0000 7f685910c700 1 -- 172.21.15.87:0/2727567026 shutdown_connections 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.211+0000 7f685910c700 1 -- 172.21.15.87:0/2727567026 wait complete. 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.211+0000 7f685910c700 1 Processor -- start 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.211+0000 7f685910c700 1 -- start start 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 0x7f68541992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 0x7f68541a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f685410e620 con 0x7f6854103530 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f685410e4a0 con 0x7f6854105e60 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f685410e7a0 con 0x7f6854108790 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53868/0 (socket says 172.21.15.87:53868) 2024-04-22T13:31:20.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 -- 172.21.15.87:0/2528919761 learned_addr learned my addr 172.21.15.87:0/2528919761 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685259c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 0x7f68541992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 -- 172.21.15.87:0/2528919761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 msgr2=0x7f68541a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685359e700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 0x7f68541a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 0x7f68541a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 -- 172.21.15.87:0/2528919761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 msgr2=0x7f68541992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 0x7f68541992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 -- 172.21.15.87:0/2528919761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68541a09e0 con 0x7f6854103530 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685259c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 0x7f68541992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685359e700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 0x7f68541a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f6852d9d700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f684400b530 tx=0x7f684400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844015020 con 0x7f6854103530 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68541a0cd0 con 0x7f6854103530 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6844011ce0 con 0x7f6854103530 2024-04-22T13:31:20.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844010d20 con 0x7f6854103530 2024-04-22T13:31:20.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.212+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68541a1230 con 0x7f6854103530 2024-04-22T13:31:20.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.213+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685402df80 con 0x7f6854103530 2024-04-22T13:31:20.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.215+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6844018020 con 0x7f6854103530 2024-04-22T13:31:20.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:19 smithi132 ceph-mon[31531]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:20.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.216+0000 7f683bfff700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 0x7f683c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:20.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.216+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f68440d0110 con 0x7f6854103530 2024-04-22T13:31:20.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.216+0000 7f685259c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 0x7f683c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:20.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.216+0000 7f685259c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 0x7f683c0a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6848005cd0 tx=0x7f6848005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:20.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.220+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f684401f020 con 0x7f6854103530 2024-04-22T13:31:20.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.436+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f6854062e80 con 0x7f683c0a3c40 2024-04-22T13:31:20.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.437+0000 7f683bfff700 1 -- 172.21.15.87:0/2528919761 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6854062e80 con 0x7f683c0a3c40 2024-04-22T13:31:20.433 INFO:teuthology.orchestra.run.smithi087.stdout:Backend: cephadm 2024-04-22T13:31:20.433 INFO:teuthology.orchestra.run.smithi087.stdout:Available: Yes 2024-04-22T13:31:20.433 INFO:teuthology.orchestra.run.smithi087.stdout:Paused: No 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 msgr2=0x7f683c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 0x7f683c0a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6848005cd0 tx=0x7f6848005c20 comp rx=0 tx=0).stop 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 msgr2=0x7f6854198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f684400b530 tx=0x7f684400b8f0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 shutdown_connections 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f683c0a3c40 0x7f683c0a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6854108790 0x7f68541a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6854103530 0x7f6854198d90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 --2- 172.21.15.87:0/2528919761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6854105e60 0x7f68541992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:20.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 >> 172.21.15.87:0/2528919761 conn(0x7f68540fcf40 msgr2=0x7f6854103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:20.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 shutdown_connections 2024-04-22T13:31:20.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:20.439+0000 7f685910c700 1 -- 172.21.15.87:0/2528919761 wait complete. 2024-04-22T13:31:21.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T13:31:21.338 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:22.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:21 smithi120 ceph-mon[31101]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:21 smithi120 ceph-mon[31101]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:22.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:21 smithi132 ceph-mon[31531]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:22.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:21 smithi132 ceph-mon[31531]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:22.222 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:21 smithi087 ceph-mon[27999]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:22.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:21 smithi087 ceph-mon[27999]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 -- 172.21.15.87:0/4248237542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 msgr2=0x7f4f94101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 --2- 172.21.15.87:0/4248237542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f94101dd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4f880099c0 tx=0x7f4f88009cd0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 -- 172.21.15.87:0/4248237542 shutdown_connections 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 --2- 172.21.15.87:0/4248237542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f94103550 0x7f4f94109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 --2- 172.21.15.87:0/4248237542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f94101dd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 --2- 172.21.15.87:0/4248237542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94102bb0 0x7f4f94103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 -- 172.21.15.87:0/4248237542 >> 172.21.15.87:0/4248237542 conn(0x7f4f940fcf70 msgr2=0x7f4f940ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:23.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 -- 172.21.15.87:0/4248237542 shutdown_connections 2024-04-22T13:31:23.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.092+0000 7f4f989ea700 1 -- 172.21.15.87:0/4248237542 wait complete. 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 Processor -- start 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 -- start start 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f94102bb0 0x7f4f94103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 0x7f4f941044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9410d440 con 0x7f4f941019b0 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9410d2c0 con 0x7f4f94103550 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f989ea700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9410d5c0 con 0x7f4f94102bb0 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53890/0 (socket says 172.21.15.87:53890) 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 -- 172.21.15.87:0/3519947803 learned_addr learned my addr 172.21.15.87:0/3519947803 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f92d9d700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 0x7f4f941044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 -- 172.21.15.87:0/3519947803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f94102bb0 msgr2=0x7f4f94103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:23.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f94102bb0 0x7f4f94103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 -- 172.21.15.87:0/3519947803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 msgr2=0x7f4f941044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 0x7f4f941044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f9259c700 1 -- 172.21.15.87:0/3519947803 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f94104da0 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.093+0000 7f4f92d9d700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 0x7f4f941044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f9259c700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800fa80 tx=0x7f4f8800fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88027070 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f941a2d80 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f88017ab0 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88009e60 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.094+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f941a3180 con 0x7f4f941019b0 2024-04-22T13:31:23.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.095+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f941a33b0 con 0x7f4f941019b0 2024-04-22T13:31:23.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.096+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f8802a070 con 0x7f4f941019b0 2024-04-22T13:31:23.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.096+0000 7f4f837fe700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 0x7f4f7c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:23.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.096+0000 7f4f91d9b700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 0x7f4f7c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:23.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.097+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f4f880d1080 con 0x7f4f941019b0 2024-04-22T13:31:23.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.097+0000 7f4f91d9b700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 0x7f4f7c0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4f84005950 tx=0x7f4f840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:23.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.101+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f8805eff0 con 0x7f4f941019b0 2024-04-22T13:31:23.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.308+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f4f9406b8d0 con 0x7f4f7c0a3c40 2024-04-22T13:31:23.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.321+0000 7f4f837fe700 1 -- 172.21.15.87:0/3519947803 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f4f9406b8d0 con 0x7f4f7c0a3c40 2024-04-22T13:31:23.317 INFO:teuthology.orchestra.run.smithi087.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T13:31:23.317 INFO:teuthology.orchestra.run.smithi087.stdout:mds.a smithi087 running (105s) 18s ago 105s 15.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 8ed971f7776c 2024-04-22T13:31:23.317 INFO:teuthology.orchestra.run.smithi087.stdout:mds.b smithi120 running (102s) 36s ago 102s 26.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 3faaa7854721 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.c smithi132 running (117s) 36s ago 116s 17.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 04749dec1a2d 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.d smithi087 running (2m) 18s ago 2m 12.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 4c0c8e550923 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.e smithi120 running (2m) 36s ago 2m 19.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 548800f11996 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.f smithi132 running (119s) 36s ago 119s 20.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 bc3319dc9651 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.g smithi087 running (108s) 18s ago 108s 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 68eeec7baf05 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.h smithi120 running (114s) 36s ago 113s 15.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 858154abee13 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.i smithi132 running (111s) 36s ago 110s 19.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 927e4bf6aceb 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.j smithi087 running (2m) 18s ago 2m 14.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 56184f87efdb 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.k smithi120 running (2m) 36s ago 2m 14.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 ea2124c2d794 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mds.l smithi132 running (2m) 36s ago 2m 15.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 501453153bde 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mgr.x smithi087 *:9283,8765 running (9m) 18s ago 9m 514M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e76228482716 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mgr.y smithi120 *:8443,8765 running (7m) 36s ago 7m 451M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d17c117fa7ca 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mgr.z smithi132 *:8443,8765 running (7m) 36s ago 7m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 08944774e250 2024-04-22T13:31:23.318 INFO:teuthology.orchestra.run.smithi087.stdout:mon.a smithi087 running (9m) 18s ago 9m 50.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 2c60a3591837 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:mon.b smithi120 running (7m) 36s ago 7m 42.6M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 86f7ebe0fc37 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:mon.c smithi132 running (8m) 36s ago 8m 39.2M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 16324ba06b2f 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0 smithi087 running (7m) 18s ago 7m 54.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 8b32e5ee0acb 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1 smithi120 running (6m) 36s ago 6m 46.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 77fc0d67188a 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2 smithi132 running (6m) 36s ago 6m 56.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 95be25886149 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3 smithi087 running (6m) 18s ago 6m 55.3M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 aab30a1aeffe 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4 smithi120 running (5m) 36s ago 5m 52.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 85cd42f6de0b 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5 smithi132 running (5m) 36s ago 5m 44.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 bd41d0c7559f 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6 smithi087 running (4m) 18s ago 4m 41.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 3c5cabe93a64 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7 smithi120 running (4m) 36s ago 4m 56.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 2115bf0b903a 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.8 smithi132 running (3m) 36s ago 3m 56.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 e8e0b54663ee 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.9 smithi087 running (3m) 18s ago 3m 51.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 57316158b18e 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.10 smithi120 running (3m) 36s ago 3m 43.8M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 9a71db4965e4 2024-04-22T13:31:23.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.11 smithi132 running (2m) 36s ago 2m 40.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 e1158dc63aec 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.323+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 msgr2=0x7f4f7c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.323+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 0x7f4f7c0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4f84005950 tx=0x7f4f840058e0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.323+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 msgr2=0x7f4f941058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.323+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800fa80 tx=0x7f4f8800fb60 comp rx=0 tx=0).stop 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 shutdown_connections 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f4f7c0a3c40 0x7f4f7c0a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f94103550 0x7f4f941044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f941019b0 0x7f4f941058f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 --2- 172.21.15.87:0/3519947803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f94102bb0 0x7f4f94103f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:23.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 >> 172.21.15.87:0/3519947803 conn(0x7f4f940fcf70 msgr2=0x7f4f9406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:23.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 shutdown_connections 2024-04-22T13:31:23.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:23.324+0000 7f4f989ea700 1 -- 172.21.15.87:0/3519947803 wait complete. 2024-04-22T13:31:23.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:23 smithi087 ceph-mon[27999]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:24.037 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T13:31:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:23 smithi120 ceph-mon[31101]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:24.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:23 smithi132 ceph-mon[31531]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:31:24.275 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:24.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:24 smithi087 ceph-mon[27999]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:25.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:24 smithi120 ceph-mon[31101]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:25.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:24 smithi132 ceph-mon[31531]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:26.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:25 smithi087 ceph-mon[27999]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:26.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:25 smithi120 ceph-mon[31101]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:26.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:25 smithi132 ceph-mon[31531]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 -- 172.21.15.87:0/939178391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 msgr2=0x7ffa541040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 --2- 172.21.15.87:0/939178391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa541040f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffa500099c0 tx=0x7ffa50009cd0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 -- 172.21.15.87:0/939178391 shutdown_connections 2024-04-22T13:31:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 --2- 172.21.15.87:0/939178391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54104630 0x7ffa5410aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 --2- 172.21.15.87:0/939178391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54102a90 0x7ffa54102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 --2- 172.21.15.87:0/939178391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa541040f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 -- 172.21.15.87:0/939178391 >> 172.21.15.87:0/939178391 conn(0x7ffa540fe030 msgr2=0x7ffa54100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 -- 172.21.15.87:0/939178391 shutdown_connections 2024-04-22T13:31:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.315+0000 7ffa5b516700 1 -- 172.21.15.87:0/939178391 wait complete. 2024-04-22T13:31:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 Processor -- start 2024-04-22T13:31:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 -- start start 2024-04-22T13:31:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54102a90 0x7ffa54198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54104630 0x7ffa541a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5410e5a0 con 0x7ffa54103c90 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5410e420 con 0x7ffa54102a90 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa5b516700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5410e720 con 0x7ffa54104630 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.316+0000 7ffa58ab1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53904/0 (socket says 172.21.15.87:53904) 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 -- 172.21.15.87:0/422057765 learned_addr learned my addr 172.21.15.87:0/422057765 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa59ab3700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54104630 0x7ffa541a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa592b2700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54102a90 0x7ffa54198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 -- 172.21.15.87:0/422057765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54104630 msgr2=0x7ffa541a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54104630 0x7ffa541a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 -- 172.21.15.87:0/422057765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54102a90 msgr2=0x7ffa54198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54102a90 0x7ffa54198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 -- 172.21.15.87:0/422057765 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541a0890 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa58ab1700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffa50003790 tx=0x7ffa50005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50027070 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa50004160 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541a0b20 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50022d90 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.317+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa541a1000 con 0x7ffa54103c90 2024-04-22T13:31:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.318+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa54069730 con 0x7ffa54103c90 2024-04-22T13:31:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.319+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffa5002a070 con 0x7ffa54103c90 2024-04-22T13:31:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.320+0000 7ffa4a7fc700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 0x7ffa400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.320+0000 7ffa592b2700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 0x7ffa400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.320+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ffa500d10a0 con 0x7ffa54103c90 2024-04-22T13:31:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.320+0000 7ffa592b2700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 0x7ffa400a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006fd0 tx=0x7ffa44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.325+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa50013070 con 0x7ffa54103c90 2024-04-22T13:31:26.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.533+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ffa54062e80 con 0x7ffa400a3c40 2024-04-22T13:31:26.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.539+0000 7ffa4a7fc700 1 -- 172.21.15.87:0/422057765 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ffa54062e80 con 0x7ffa400a3c40 2024-04-22T13:31:26.535 INFO:teuthology.orchestra.run.smithi087.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T13:31:26.535 INFO:teuthology.orchestra.run.smithi087.stdout:mds.all 12/12 39s ago 2m smithi087=a;smithi087=d;smithi087=g;smithi087=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-04-22T13:31:26.535 INFO:teuthology.orchestra.run.smithi087.stdout:mgr 3/3 39s ago 7m smithi087=x;smithi120=y;smithi132=z;count:3 2024-04-22T13:31:26.535 INFO:teuthology.orchestra.run.smithi087.stdout:mon 3/3 39s ago 8m smithi087:172.21.15.87=a;smithi120:172.21.15.120=b;smithi132:172.21.15.132=c;count:3 2024-04-22T13:31:26.535 INFO:teuthology.orchestra.run.smithi087.stdout:osd 12 39s ago - 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 msgr2=0x7ffa400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 0x7ffa400a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006fd0 tx=0x7ffa44008040 comp rx=0 tx=0).stop 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 msgr2=0x7ffa54199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffa50003790 tx=0x7ffa50005b00 comp rx=0 tx=0).stop 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 shutdown_connections 2024-04-22T13:31:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa400a3c40 0x7ffa400a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa54104630 0x7ffa541a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa54102a90 0x7ffa54198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 --2- 172.21.15.87:0/422057765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa54103c90 0x7ffa54199180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 >> 172.21.15.87:0/422057765 conn(0x7ffa540fe030 msgr2=0x7ffa54107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 shutdown_connections 2024-04-22T13:31:26.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:26.541+0000 7ffa5b516700 1 -- 172.21.15.87:0/422057765 wait complete. 2024-04-22T13:31:27.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T13:31:27.587 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:27.972 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:27 smithi087 ceph-mon[27999]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:27.972 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:27 smithi087 ceph-mon[27999]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:28.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:27 smithi120 ceph-mon[31101]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:27 smithi120 ceph-mon[31101]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:28.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:27 smithi132 ceph-mon[31531]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:28.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:27 smithi132 ceph-mon[31531]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- 172.21.15.87:0/1229024328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 msgr2=0x7f3838109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 --2- 172.21.15.87:0/1229024328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f3838109df0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f383000b0d0 tx=0x7f383000b3e0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- 172.21.15.87:0/1229024328 shutdown_connections 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 --2- 172.21.15.87:0/1229024328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f3838109df0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 --2- 172.21.15.87:0/1229024328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 0x7f3838101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 --2- 172.21.15.87:0/1229024328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 0x7f3838103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- 172.21.15.87:0/1229024328 >> 172.21.15.87:0/1229024328 conn(0x7f38380fcf60 msgr2=0x7f38380ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- 172.21.15.87:0/1229024328 shutdown_connections 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- 172.21.15.87:0/1229024328 wait complete. 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 Processor -- start 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.459+0000 7f383eef5700 1 -- start start 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 0x7f383806bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 0x7f383819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f383810d4b0 con 0x7f3838103540 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f383810d330 con 0x7f3838102ba0 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383eef5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f383810d630 con 0x7f38381019a0 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53934/0 (socket says 172.21.15.87:53934) 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 -- 172.21.15.87:0/735740837 learned_addr learned my addr 172.21.15.87:0/735740837 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f3837fff700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 0x7f383819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383cc91700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 0x7f383806bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 -- 172.21.15.87:0/735740837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 msgr2=0x7f383806bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 0x7f383806bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 -- 172.21.15.87:0/735740837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 msgr2=0x7f383819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 0x7f383819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 -- 172.21.15.87:0/735740837 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38381a0800 con 0x7f3838103540 2024-04-22T13:31:29.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f383d492700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3830009fd0 tx=0x7f3830004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:29.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.460+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383001f030 con 0x7f3838103540 2024-04-22T13:31:29.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.461+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3830003a00 con 0x7f3838103540 2024-04-22T13:31:29.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.461+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38381a0a90 con 0x7f3838103540 2024-04-22T13:31:29.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.461+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38300047f0 con 0x7f3838103540 2024-04-22T13:31:29.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.461+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38381a0f60 con 0x7f3838103540 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.462+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383802df80 con 0x7f3838103540 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.463+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3830003bb0 con 0x7f3838103540 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.463+0000 7f3835ffb700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 0x7f38280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.464+0000 7f383cc91700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 0x7f38280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.464+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f383000f020 con 0x7f3838103540 2024-04-22T13:31:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.464+0000 7f383cc91700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 0x7f38280a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3838102a00 tx=0x7f382c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:29.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.467+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3830029020 con 0x7f3838103540 2024-04-22T13:31:29.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.671+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f3838062e80 con 0x7f38280a3c40 2024-04-22T13:31:29.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.673+0000 7f3835ffb700 1 -- 172.21.15.87:0/735740837 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f3838062e80 con 0x7f38280a3c40 2024-04-22T13:31:29.668 INFO:teuthology.orchestra.run.smithi087.stdout:HOST ADDR LABELS STATUS 2024-04-22T13:31:29.668 INFO:teuthology.orchestra.run.smithi087.stdout:smithi087 172.21.15.87 2024-04-22T13:31:29.669 INFO:teuthology.orchestra.run.smithi087.stdout:smithi120 172.21.15.120 2024-04-22T13:31:29.669 INFO:teuthology.orchestra.run.smithi087.stdout:smithi132 172.21.15.132 2024-04-22T13:31:29.669 INFO:teuthology.orchestra.run.smithi087.stdout:3 hosts in cluster 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 msgr2=0x7f38280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 0x7f38280a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3838102a00 tx=0x7f382c009190 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 msgr2=0x7f383819ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3830009fd0 tx=0x7f3830004e40 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 shutdown_connections 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f38280a3c40 0x7f38280a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3838103540 0x7f383819ff30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38381019a0 0x7f383806bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 --2- 172.21.15.87:0/735740837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3838102ba0 0x7f383819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 >> 172.21.15.87:0/735740837 conn(0x7f38380fcf60 msgr2=0x7f3838105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 shutdown_connections 2024-04-22T13:31:29.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:29.675+0000 7f383eef5700 1 -- 172.21.15.87:0/735740837 wait complete. 2024-04-22T13:31:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:29 smithi087 ceph-mon[27999]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:30.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:29 smithi120 ceph-mon[31101]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:30.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:29 smithi132 ceph-mon[31531]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:30.548 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T13:31:30.791 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:31.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:30 smithi120 ceph-mon[31101]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:31.191 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:30 smithi087 ceph-mon[27999]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:31.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:30 smithi132 ceph-mon[31531]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:32.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:31 smithi120 ceph-mon[31101]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:32.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:31 smithi132 ceph-mon[31531]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:32.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:31 smithi087 ceph-mon[27999]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1171390381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 msgr2=0x7ff3a810af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1171390381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a810af20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000b0d0 tx=0x7ff3a000b3e0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1171390381 shutdown_connections 2024-04-22T13:31:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1171390381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a810af20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1171390381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a8102ad0 0x7ff3a8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1171390381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8103cd0 0x7ff3a8104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1171390381 >> 172.21.15.87:0/1171390381 conn(0x7ff3a80fe090 msgr2=0x7ff3a81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1171390381 shutdown_connections 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.631+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1171390381 wait complete. 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 Processor -- start 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 -- start start 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8102ad0 0x7ff3a8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a8103cd0 0x7ff3a81990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a810e5e0 con 0x7ff3a8104670 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a810e460 con 0x7ff3a8102ad0 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3aef0b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a810e760 con 0x7ff3a8103cd0 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:32.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46596/0 (socket says 172.21.15.87:46596) 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 -- 172.21.15.87:0/1493820787 learned_addr learned my addr 172.21.15.87:0/1493820787 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3acca7700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8102ad0 0x7ff3a8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 -- 172.21.15.87:0/1493820787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a8103cd0 msgr2=0x7ff3a81990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a8103cd0 0x7ff3a81990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 -- 172.21.15.87:0/1493820787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8102ad0 msgr2=0x7ff3a8198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8102ad0 0x7ff3a8198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.632+0000 7ff3ad4a8700 1 -- 172.21.15.87:0/1493820787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a81a0800 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3ad4a8700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009fd0 tx=0x7ff3a0004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a001f030 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a81a0a90 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3a0007a80 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00047d0 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.633+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a81a0f60 con 0x7ff3a8104670 2024-04-22T13:31:32.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.634+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a802df80 con 0x7ff3a8104670 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.635+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff3a0003a00 con 0x7ff3a8104670 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.635+0000 7ff3a5ffb700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 0x7ff3980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.636+0000 7ff3acca7700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 0x7ff3980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.636+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ff3a00d1550 con 0x7ff3a8104670 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.636+0000 7ff3acca7700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 0x7ff3980a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006fd0 tx=0x7ff39c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:32.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.639+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3a009e060 con 0x7ff3a8104670 2024-04-22T13:31:32.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.848+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff3a8062e80 con 0x7ff3980a3c40 2024-04-22T13:31:32.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.849+0000 7ff3a5ffb700 1 -- 172.21.15.87:0/1493820787 <== mgr.14152 v2:172.21.15.87:6800/1421536657 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff3a8062e80 con 0x7ff3980a3c40 2024-04-22T13:31:32.845 INFO:teuthology.orchestra.run.smithi087.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T13:31:32.845 INFO:teuthology.orchestra.run.smithi087.stdout:smithi087 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NQ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T13:31:32.845 INFO:teuthology.orchestra.run.smithi087.stdout:smithi120 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T13:31:32.845 INFO:teuthology.orchestra.run.smithi087.stdout:smithi132 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232000B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 msgr2=0x7ff3980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 0x7ff3980a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006fd0 tx=0x7ff39c008040 comp rx=0 tx=0).stop 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 msgr2=0x7ff3a81a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009fd0 tx=0x7ff3a0004e00 comp rx=0 tx=0).stop 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 shutdown_connections 2024-04-22T13:31:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ff3980a3c40 0x7ff3980a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3a8104670 0x7ff3a81a0160 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a8102ad0 0x7ff3a8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 --2- 172.21.15.87:0/1493820787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a8103cd0 0x7ff3a81990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.851+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 >> 172.21.15.87:0/1493820787 conn(0x7ff3a80fe090 msgr2=0x7ff3a8106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.852+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 shutdown_connections 2024-04-22T13:31:32.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:32.852+0000 7ff3aef0b700 1 -- 172.21.15.87:0/1493820787 wait complete. 2024-04-22T13:31:33.693 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T13:31:33.702 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:31:33.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T13:31:33.941 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:33.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:33 smithi087 ceph-mon[27999]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:33.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:33 smithi087 ceph-mon[27999]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:34.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:33 smithi120 ceph-mon[31101]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:33 smithi120 ceph-mon[31101]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:34.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:33 smithi132 ceph-mon[31531]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:34.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:33 smithi132 ceph-mon[31531]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 -- 172.21.15.87:0/4277827979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130100db0 msgr2=0x7fd1301011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 --2- 172.21.15.87:0/4277827979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130100db0 0x7fd1301011d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd124009990 tx=0x7fd124009ca0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 -- 172.21.15.87:0/4277827979 shutdown_connections 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 --2- 172.21.15.87:0/4277827979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130104730 0x7fd13010afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 --2- 172.21.15.87:0/4277827979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130100db0 0x7fd1301011d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 --2- 172.21.15.87:0/4277827979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd1301041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 -- 172.21.15.87:0/4277827979 >> 172.21.15.87:0/4277827979 conn(0x7fd1300fc710 msgr2=0x7fd1300feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 -- 172.21.15.87:0/4277827979 shutdown_connections 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.674+0000 7fd134e0d700 1 -- 172.21.15.87:0/4277827979 wait complete. 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 Processor -- start 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 -- start start 2024-04-22T13:31:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130100db0 0x7fd130198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd130199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd13010e5d0 con 0x7fd130104730 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd13010e450 con 0x7fd130103d90 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd134e0d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd13010e750 con 0x7fd130100db0 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd12ed9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd12ed9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46618/0 (socket says 172.21.15.87:46618) 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd12ed9d700 1 -- 172.21.15.87:0/2266807062 learned_addr learned my addr 172.21.15.87:0/2266807062 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd12dd9b700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd130199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.675+0000 7fd12e59c700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130100db0 0x7fd130198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 -- 172.21.15.87:0/2266807062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130100db0 msgr2=0x7fd130198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130100db0 0x7fd130198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 -- 172.21.15.87:0/2266807062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 msgr2=0x7fd130199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:35.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd130199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 -- 172.21.15.87:0/2266807062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1301a08c0 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12dd9b700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd130199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd12ed9d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd12800b970 tx=0x7fd12800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd128019070 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1301a0bb0 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd128014440 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12800ce60 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.676+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1301a1140 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.677+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd13002df80 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.678+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd12801c070 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.679+0000 7fd11f7fe700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 0x7fd1180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.679+0000 7fd12e59c700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 0x7fd1180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.679+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd1280cfc50 con 0x7fd130104730 2024-04-22T13:31:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.679+0000 7fd12e59c700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 0x7fd1180a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd124003790 tx=0x7fd124005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.682+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd12809c6d0 con 0x7fd130104730 2024-04-22T13:31:35.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.926+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd13002e3a0 con 0x7fd130104730 2024-04-22T13:31:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.927+0000 7fd11f7fe700 1 -- 172.21.15.87:0/2266807062 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3050 (secure 0 0 0) 0x7fd12809bfd0 con 0x7fd130104730 2024-04-22T13:31:35.925 INFO:teuthology.orchestra.run.smithi087.stdout:e20 2024-04-22T13:31:35.925 INFO:teuthology.orchestra.run.smithi087.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:legacy client fscid: 1 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:Filesystem 'cephfs' (1) 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:fs_name cephfs 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 20 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-04-22T13:29:46.666137+0000 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:modified 2024-04-22T13:29:51.322041+0000 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:tableserver 0 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:root 0 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:session_timeout 300 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:session_autoclose 300 2024-04-22T13:31:35.926 INFO:teuthology.orchestra.run.smithi087.stdout:max_file_size 1099511627776 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:max_xattr_size 65536 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:required_client_features {} 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:last_failure 0 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:last_failure_osd_epoch 0 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:max_mds 3 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:in 0,1,2 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:up {0=24482,1=24475,2=24460} 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:failed 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:damaged 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:stopped 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:data_pools [3] 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:metadata_pool 2 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:inline_data disabled 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:balancer 2024-04-22T13:31:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:bal_rank_mask -1 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:standby_count_wanted 0 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:qdb_cluster leader: 24482 members: 24460,24482,24475 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.b{0:24482} state up:active seq 3 addr [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.f{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.132:6834/1119314155,v1:172.21.15.132:6835/1119314155] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.i{1:24475} state up:active seq 6 addr [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.h{1:24440} state up:standby-replay seq 1 addr [v2:172.21.15.120:6836/3391616456,v1:172.21.15.120:6837/3391616456] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.c{2:24460} state up:active seq 7 addr [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.e{2:24407} state up:standby-replay seq 1 addr [v2:172.21.15.120:6834/1829745193,v1:172.21.15.120:6835/1829745193] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:Standby daemons: 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.j{-1:14562} state up:standby seq 1 addr [v2:172.21.15.87:6834/915813894,v1:172.21.15.87:6835/915813894] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.87:6836/1472471745,v1:172.21.15.87:6837/1472471745] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.928 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.g{-1:14637} state up:standby seq 1 addr [v2:172.21.15.87:6838/2616185101,v1:172.21.15.87:6839/2616185101] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.87:6840/982852061,v1:172.21.15.87:6841/982852061] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.k{-1:24383} state up:standby seq 1 addr [v2:172.21.15.120:6832/820427937,v1:172.21.15.120:6833/820427937] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stdout:[mds.l{-1:24400} state up:standby seq 1 addr [v2:172.21.15.132:6832/1335303979,v1:172.21.15.132:6833/1335303979] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 msgr2=0x7fd1180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 0x7fd1180a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd124003790 tx=0x7fd124005fb0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 msgr2=0x7fd1301a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd12800b970 tx=0x7fd12800bd30 comp rx=0 tx=0).stop 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 shutdown_connections 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd1180a3c40 0x7fd1180a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd130104730 0x7fd1301a01f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd130100db0 0x7fd130198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 --2- 172.21.15.87:0/2266807062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd130103d90 0x7fd130199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 >> 172.21.15.87:0/2266807062 conn(0x7fd1300fc710 msgr2=0x7fd1301071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 shutdown_connections 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:35.929+0000 7fd134e0d700 1 -- 172.21.15.87:0/2266807062 wait complete. 2024-04-22T13:31:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:35.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:35 smithi087 ceph-mon[27999]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:35 smithi120 ceph-mon[31101]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:36.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:35 smithi132 ceph-mon[31531]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:36.870 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T13:31:37.109 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:37.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:36 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2266807062' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T13:31:37.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:36 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2266807062' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T13:31:37.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:36 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2266807062' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T13:31:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:37 smithi120 ceph-mon[31101]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:38.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:37 smithi132 ceph-mon[31531]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:38.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:37 smithi087 ceph-mon[27999]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 -- 172.21.15.87:0/322159868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 msgr2=0x7fb42010aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 --2- 172.21.15.87:0/322159868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb42010aef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb4100099c0 tx=0x7fb410009cd0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 -- 172.21.15.87:0/322159868 shutdown_connections 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 --2- 172.21.15.87:0/322159868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb42010aef0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 --2- 172.21.15.87:0/322159868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420102aa0 0x7fb420102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 --2- 172.21.15.87:0/322159868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420103ca0 0x7fb420104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 -- 172.21.15.87:0/322159868 >> 172.21.15.87:0/322159868 conn(0x7fb4200fe040 msgr2=0x7fb420100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 -- 172.21.15.87:0/322159868 shutdown_connections 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.759+0000 7fb42773f700 1 -- 172.21.15.87:0/322159868 wait complete. 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 Processor -- start 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 -- start start 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 0x7fb42019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 0x7fb42019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010e5b0 con 0x7fb420104640 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010e430 con 0x7fb420102aa0 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb42773f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010e730 con 0x7fb420103ca0 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:38.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46624/0 (socket says 172.21.15.87:46624) 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 -- 172.21.15.87:0/2893173150 learned_addr learned my addr 172.21.15.87:0/2893173150 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb4254db700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 0x7fb42019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb424cda700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 0x7fb42019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 -- 172.21.15.87:0/2893173150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 msgr2=0x7fb42019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 0x7fb42019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 -- 172.21.15.87:0/2893173150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 msgr2=0x7fb42019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 0x7fb42019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb425cdc700 1 -- 172.21.15.87:0/2893173150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4201a2a30 con 0x7fb420104640 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb424cda700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 0x7fb42019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.760+0000 7fb4254db700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 0x7fb42019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb425cdc700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb4100047b0 tx=0x7fb410009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410027070 con 0x7fb420104640 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb410022470 con 0x7fb420104640 2024-04-22T13:31:38.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4201a2cc0 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41000f6c0 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.761+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4201a3110 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.762+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4201a32e0 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.764+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb41002a070 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.764+0000 7fb41e7fc700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 0x7fb4080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.764+0000 7fb4254db700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 0x7fb4080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.764+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fb4100d1930 con 0x7fb420104640 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.765+0000 7fb4254db700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 0x7fb4080a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb420103b00 tx=0x7fb418009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:38.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.768+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb410031070 con 0x7fb420104640 2024-04-22T13:31:38.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.971+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fb4201a34b0 con 0x7fb420104640 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.972+0000 7fb41e7fc700 1 -- 172.21.15.87:0/2893173150 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5235 (secure 0 0 0) 0x7fb410013070 con 0x7fb420104640 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 87 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:fsid 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-04-22T13:21:43.838276+0000 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:modified 2024-04-22T13:30:37.880012+0000 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:crush_version 26 2024-04-22T13:31:38.968 INFO:teuthology.orchestra.run.smithi087.stdout:full_ratio 0.95 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:backfillfull_ratio 0.9 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:nearfull_ratio 0.85 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:require_min_compat_client luminous 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:min_compat_client luminous 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:require_osd_release squid 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:stretch_mode_enabled false 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:max_osd 12 2024-04-22T13:31:38.969 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.87:6802/4104861709,v1:172.21.15.87:6803/4104861709] [v2:172.21.15.87:6804/4104861709,v1:172.21.15.87:6805/4104861709] exists,up 48d1248e-5bb0-4829-a035-88f1d8b049d4 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/3769829822,v1:172.21.15.120:6801/3769829822] [v2:172.21.15.120:6802/3769829822,v1:172.21.15.120:6803/3769829822] exists,up b96dba19-35ae-4b99-9a8d-43a623679f0e 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/2448340418,v1:172.21.15.132:6801/2448340418] [v2:172.21.15.132:6802/2448340418,v1:172.21.15.132:6803/2448340418] exists,up 02678f49-713f-40a2-8cc6-426c50c594a1 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.87:6810/2474923327,v1:172.21.15.87:6811/2474923327] [v2:172.21.15.87:6812/2474923327,v1:172.21.15.87:6813/2474923327] exists,up cc5b62ee-b979-43d0-8f66-1c4183042bf1 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/172450130,v1:172.21.15.120:6809/172450130] [v2:172.21.15.120:6810/172450130,v1:172.21.15.120:6811/172450130] exists,up dd7d7e47-595c-4f2b-846c-0030654a9954 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/1166176702,v1:172.21.15.132:6809/1166176702] [v2:172.21.15.132:6810/1166176702,v1:172.21.15.132:6811/1166176702] exists,up 1c76c783-6c30-48d4-b20b-c4ef76b03f7f 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.87:6818/3192057229,v1:172.21.15.87:6819/3192057229] [v2:172.21.15.87:6820/3192057229,v1:172.21.15.87:6821/3192057229] exists,up 9d20c5d7-b08d-4a48-8694-1d2faf0a1651 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/1140009169,v1:172.21.15.120:6817/1140009169] [v2:172.21.15.120:6818/1140009169,v1:172.21.15.120:6819/1140009169] exists,up 086e7b21-8587-47e5-a1c3-4cb514d43975 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/1871032509,v1:172.21.15.132:6817/1871032509] [v2:172.21.15.132:6818/1871032509,v1:172.21.15.132:6819/1871032509] exists,up 0d2acda1-2a71-452c-acc3-5fbaf491f53c 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.9 up in weight 1 up_from 67 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.87:6826/2036314844,v1:172.21.15.87:6827/2036314844] [v2:172.21.15.87:6828/2036314844,v1:172.21.15.87:6829/2036314844] exists,up 07f9c4fd-b4f0-48db-b5de-05dd6c55b890 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/2160182398,v1:172.21.15.120:6825/2160182398] [v2:172.21.15.120:6826/2160182398,v1:172.21.15.120:6827/2160182398] exists,up a596e14d-82e2-4b4c-8af0-21f73df9b9d2 2024-04-22T13:31:38.970 INFO:teuthology.orchestra.run.smithi087.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/3735146249,v1:172.21.15.132:6825/3735146249] [v2:172.21.15.132:6826/3735146249,v1:172.21.15.132:6827/3735146249] exists,up c8cbc34d-da2e-48c0-a354-f30062b39ade 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 2.13 [2,11] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 2.1d [8,11] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 3.c [3,9] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 3.e [5,9] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 3.13 [4,9] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 3.27 [4,8] 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/1019204087 expires 2024-04-23T13:22:36.668187+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/3462524867 expires 2024-04-23T13:22:36.668187+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/2538396844 expires 2024-04-23T13:22:36.668187+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6801/35245616 expires 2024-04-23T13:22:36.668187+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6800/4093217616 expires 2024-04-23T13:22:10.831107+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/343208246 expires 2024-04-23T13:22:10.831107+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6801/4093217616 expires 2024-04-23T13:22:10.831107+0000 2024-04-22T13:31:38.971 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/73776732 expires 2024-04-23T13:22:10.831107+0000 2024-04-22T13:31:38.972 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/1882698965 expires 2024-04-23T13:22:10.831107+0000 2024-04-22T13:31:38.972 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6800/35245616 expires 2024-04-23T13:22:36.668187+0000 2024-04-22T13:31:38.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 msgr2=0x7fb4080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 0x7fb4080a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb420103b00 tx=0x7fb418009040 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 msgr2=0x7fb4201a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb4100047b0 tx=0x7fb410009510 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 shutdown_connections 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fb4080a3cc0 0x7fb4080a6180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb420104640 0x7fb4201a2390 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb420102aa0 0x7fb42019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 --2- 172.21.15.87:0/2893173150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb420103ca0 0x7fb42019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.974+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 >> 172.21.15.87:0/2893173150 conn(0x7fb4200fe040 msgr2=0x7fb420074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.975+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 shutdown_connections 2024-04-22T13:31:38.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:38.975+0000 7fb42773f700 1 -- 172.21.15.87:0/2893173150 wait complete. 2024-04-22T13:31:39.862 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T13:31:39.874 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:39.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 -- 172.21.15.87:0/3182698447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680101fc0 msgr2=0x7f8680102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:39.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 --2- 172.21.15.87:0/3182698447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680101fc0 0x7f8680102440 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8674009900 tx=0x7f867402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:39.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 -- 172.21.15.87:0/3182698447 shutdown_connections 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 --2- 172.21.15.87:0/3182698447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680102980 0x7f86801092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 --2- 172.21.15.87:0/3182698447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680101fc0 0x7f8680102440 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 --2- 172.21.15.87:0/3182698447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680100e70 0x7f8680101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 -- 172.21.15.87:0/3182698447 >> 172.21.15.87:0/3182698447 conn(0x7f86800fc3e0 msgr2=0x7f86800fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 -- 172.21.15.87:0/3182698447 shutdown_connections 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.981+0000 7f8684ca8640 1 -- 172.21.15.87:0/3182698447 wait complete. 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 Processor -- start 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 -- start start 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680101fc0 0x7f868019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680102980 0x7f86801a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:39.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f868010c9a0 con 0x7f8680100e70 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f868010c820 con 0x7f8680101fc0 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f8684ca8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f868010cb20 con 0x7f8680102980 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f867e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f867e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59738/0 (socket says 172.21.15.87:59738) 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f867e59c640 1 -- 172.21.15.87:0/2541350067 learned_addr learned my addr 172.21.15.87:0/2541350067 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.982+0000 7f867ed9d640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680102980 0x7f86801a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 -- 172.21.15.87:0/2541350067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680102980 msgr2=0x7f86801a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680102980 0x7f86801a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 -- 172.21.15.87:0/2541350067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680101fc0 msgr2=0x7f868019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867dd9b640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680101fc0 0x7f868019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680101fc0 0x7f868019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 -- 172.21.15.87:0/2541350067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86801a1f80 con 0x7f8680100e70 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f867e59c640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f867000c920 tx=0x7f867000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:39.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670007c70 con 0x7f8680100e70 2024-04-22T13:31:39.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86801a2270 con 0x7f8680100e70 2024-04-22T13:31:39.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8670007e10 con 0x7f8680100e70 2024-04-22T13:31:39.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670018820 con 0x7f8680100e70 2024-04-22T13:31:39.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.983+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86801a2810 con 0x7f8680100e70 2024-04-22T13:31:39.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.984+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86801a2ad0 con 0x7f8680100e70 2024-04-22T13:31:39.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.986+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f867000f450 con 0x7f8680100e70 2024-04-22T13:31:39.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.986+0000 7f866f7fe640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 0x7f86680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:39.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.986+0000 7f867dd9b640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 0x7f86680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:39.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.986+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f86700d0740 con 0x7f8680100e70 2024-04-22T13:31:39.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.987+0000 7f867dd9b640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 0x7f86680a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f867402c840 tx=0x7f8674000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:39.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:39.990+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8670014030 con 0x7f8680100e70 2024-04-22T13:31:40.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:39 smithi120 ceph-mon[31101]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:40.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:39 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2893173150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T13:31:40.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.176+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86801a2ca0 con 0x7f8680100e70 2024-04-22T13:31:40.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.178+0000 7f866f7fe640 1 -- 172.21.15.87:0/2541350067 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7f867009d250 con 0x7f8680100e70 2024-04-22T13:31:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:40.174 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:40.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 msgr2=0x7f86680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 0x7f86680a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f867402c840 tx=0x7f8674000bc0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 msgr2=0x7f868019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f867000c920 tx=0x7f867000cde0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 shutdown_connections 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f86680a3bb0 0x7f86680a6070 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8680102980 0x7f86801a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8680101fc0 0x7f868019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 --2- 172.21.15.87:0/2541350067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8680100e70 0x7f868019a3d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 >> 172.21.15.87:0/2541350067 conn(0x7f86800fc3e0 msgr2=0x7f86800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 shutdown_connections 2024-04-22T13:31:40.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.180+0000 7f8684ca8640 1 -- 172.21.15.87:0/2541350067 wait complete. 2024-04-22T13:31:40.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:39 smithi087 ceph-mon[27999]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:40.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:39 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2893173150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T13:31:40.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:40.188 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:40.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:39 smithi132 ceph-mon[31531]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:31:40.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:39 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2893173150' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- 172.21.15.87:0/2561816429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102040 msgr2=0x7f93081024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 --2- 172.21.15.87:0/2561816429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102040 0x7f93081024c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009900 tx=0x7f92fc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- 172.21.15.87:0/2561816429 shutdown_connections 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 --2- 172.21.15.87:0/2561816429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102a00 0x7f9308109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 --2- 172.21.15.87:0/2561816429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102040 0x7f93081024c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 --2- 172.21.15.87:0/2561816429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 0x7f9308101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- 172.21.15.87:0/2561816429 >> 172.21.15.87:0/2561816429 conn(0x7f93080fc420 msgr2=0x7f93080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- 172.21.15.87:0/2561816429 shutdown_connections 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- 172.21.15.87:0/2561816429 wait complete. 2024-04-22T13:31:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 Processor -- start 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.293+0000 7f930e9e6640 1 -- start start 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 0x7f9308198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102040 0x7f93081986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f930810db50 con 0x7f9308102a00 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f930810d9d0 con 0x7f9308102040 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f930810dcd0 con 0x7f9308100e60 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59766/0 (socket says 172.21.15.87:59766) 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 -- 172.21.15.87:0/1418627609 learned_addr learned my addr 172.21.15.87:0/1418627609 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f93077fe640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102040 0x7f93081986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f9307fff640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 0x7f9308198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 -- 172.21.15.87:0/1418627609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 msgr2=0x7f9308198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 0x7f9308198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 -- 172.21.15.87:0/1418627609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102040 msgr2=0x7f93081986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102040 0x7f93081986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 -- 172.21.15.87:0/1418627609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930819fda0 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930cf83640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f930000eaa0 tx=0x7f930000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930000cdb0 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9300005020 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93081a0090 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.294+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93000116d0 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.295+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93081a0560 con 0x7f9308102a00 2024-04-22T13:31:40.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.295+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9308108200 con 0x7f9308102a00 2024-04-22T13:31:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.296+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f93000051d0 con 0x7f9308102a00 2024-04-22T13:31:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.297+0000 7f93057fa640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 0x7f92f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.297+0000 7f9307fff640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 0x7f92f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.297+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f93000d0560 con 0x7f9308102a00 2024-04-22T13:31:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.298+0000 7f9307fff640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 0x7f92f00a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92f80058c0 tx=0x7f92f8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.302+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9300016030 con 0x7f9308102a00 2024-04-22T13:31:40.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.477+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9308062b50 con 0x7f9308102a00 2024-04-22T13:31:40.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.478+0000 7f93057fa640 1 -- 172.21.15.87:0/1418627609 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f930009cfe0 con 0x7f9308102a00 2024-04-22T13:31:40.474 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:40.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 msgr2=0x7f92f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 0x7f92f00a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92f80058c0 tx=0x7f92f8005850 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 msgr2=0x7f930819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f930000eaa0 tx=0x7f930000ef60 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 shutdown_connections 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f92f00a3bb0 0x7f92f00a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9308102a00 0x7f930819f730 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9308102040 0x7f93081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 --2- 172.21.15.87:0/1418627609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9308100e60 0x7f9308198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 >> 172.21.15.87:0/1418627609 conn(0x7f93080fc420 msgr2=0x7f93081053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 shutdown_connections 2024-04-22T13:31:40.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.480+0000 7f930e9e6640 1 -- 172.21.15.87:0/1418627609 wait complete. 2024-04-22T13:31:40.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 -- 172.21.15.87:0/398911593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48100e60 msgr2=0x7ffa48101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/398911593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48100e60 0x7ffa48101280 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009900 tx=0x7ffa3802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 -- 172.21.15.87:0/398911593 shutdown_connections 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/398911593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48102a00 0x7ffa48109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/398911593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa481024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/398911593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48100e60 0x7ffa48101280 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 -- 172.21.15.87:0/398911593 >> 172.21.15.87:0/398911593 conn(0x7ffa480fc420 msgr2=0x7ffa480fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 -- 172.21.15.87:0/398911593 shutdown_connections 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 -- 172.21.15.87:0/398911593 wait complete. 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.593+0000 7ffa4d72b640 1 Processor -- start 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- start start 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48100e60 0x7ffa48198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa48198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810db30 con 0x7ffa48102a00 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810d9b0 con 0x7ffa48102040 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810dcb0 con 0x7ffa48100e60 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59784/0 (socket says 172.21.15.87:59784) 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 -- 172.21.15.87:0/2004095690 learned_addr learned my addr 172.21.15.87:0/2004095690 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa467fc640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa48198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa46ffd640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48100e60 0x7ffa48198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 -- 172.21.15.87:0/2004095690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48100e60 msgr2=0x7ffa48198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48100e60 0x7ffa48198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 -- 172.21.15.87:0/2004095690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 msgr2=0x7ffa48198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa48198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 -- 172.21.15.87:0/2004095690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819fe60 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa467fc640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa48198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa477fe640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000eaa0 tx=0x7ffa4000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4000cdb0 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa481a0150 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa40005020 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa400116d0 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.594+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a06a0 con 0x7ffa48102a00 2024-04-22T13:31:40.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.595+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa48108270 con 0x7ffa48102a00 2024-04-22T13:31:40.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.596+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffa400051d0 con 0x7ffa48102a00 2024-04-22T13:31:40.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.597+0000 7ffa2ffff640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 0x7ffa300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.597+0000 7ffa46ffd640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 0x7ffa300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.597+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ffa400d05d0 con 0x7ffa48102a00 2024-04-22T13:31:40.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.597+0000 7ffa46ffd640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 0x7ffa300a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036040 tx=0x7ffa38000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.602+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa40016030 con 0x7ffa48102a00 2024-04-22T13:31:40.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.781+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa48062b50 con 0x7ffa48102a00 2024-04-22T13:31:40.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.782+0000 7ffa2ffff640 1 -- 172.21.15.87:0/2004095690 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7ffa4009d050 con 0x7ffa48102a00 2024-04-22T13:31:40.778 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:40.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 msgr2=0x7ffa300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 0x7ffa300a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036040 tx=0x7ffa38000bc0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 msgr2=0x7ffa4819f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000eaa0 tx=0x7ffa4000ef60 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 shutdown_connections 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7ffa300a3bb0 0x7ffa300a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa48102a00 0x7ffa4819f7f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa48102040 0x7ffa48198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 --2- 172.21.15.87:0/2004095690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa48100e60 0x7ffa48198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 >> 172.21.15.87:0/2004095690 conn(0x7ffa480fc420 msgr2=0x7ffa48105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 shutdown_connections 2024-04-22T13:31:40.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.784+0000 7ffa4d72b640 1 -- 172.21.15.87:0/2004095690 wait complete. 2024-04-22T13:31:40.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:40.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3766881294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8101160 msgr2=0x7f8db81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3766881294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8101160 0x7f8db81015e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009900 tx=0x7f8da802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:40.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3766881294 shutdown_connections 2024-04-22T13:31:40.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3766881294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101b20 0x7f8db81093c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3766881294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8101160 0x7f8db81015e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3766881294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8102430 0x7f8db8102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3766881294 >> 172.21.15.87:0/3766881294 conn(0x7f8db80fbb90 msgr2=0x7f8db80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3766881294 shutdown_connections 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.896+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3766881294 wait complete. 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 Processor -- start 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 -- start start 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 0x7f8db81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 0x7f8db81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810db60 con 0x7f8db8102430 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810d9e0 con 0x7f8db8101160 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbf7b6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810dce0 con 0x7f8db8101b20 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59800/0 (socket says 172.21.15.87:59800) 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 -- 172.21.15.87:0/3198105354 learned_addr learned my addr 172.21.15.87:0/3198105354 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbcd51640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 0x7f8db81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbd552640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 0x7f8db81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 -- 172.21.15.87:0/3198105354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 msgr2=0x7f8db81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 0x7f8db81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 -- 172.21.15.87:0/3198105354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 msgr2=0x7f8db81982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 0x7f8db81982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 -- 172.21.15.87:0/3198105354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db819fed0 con 0x7f8db8102430 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbcd51640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 0x7f8db81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbd552640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 0x7f8db81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.897+0000 7f8dbdd53640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8db000b970 tx=0x7f8db000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.898+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0019070 con 0x7f8db8102430 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.898+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8db0014440 con 0x7f8db8102430 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.898+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db81a01c0 con 0x7f8db8102430 2024-04-22T13:31:40.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.898+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db00075d0 con 0x7f8db8102430 2024-04-22T13:31:40.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.898+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db81a0790 con 0x7f8db8102430 2024-04-22T13:31:40.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.899+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db8108370 con 0x7f8db8102430 2024-04-22T13:31:40.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.900+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8db001c070 con 0x7f8db8102430 2024-04-22T13:31:40.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.900+0000 7f8dae7fc640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 0x7f8da40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:40.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.901+0000 7f8dbd552640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 0x7f8da40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:40.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.901+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f8db00cf940 con 0x7f8db8102430 2024-04-22T13:31:40.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.901+0000 7f8dbd552640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 0x7f8da40a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8db806a2f0 tx=0x7f8db4005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:40.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:40.905+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8db009c3c0 con 0x7f8db8102430 2024-04-22T13:31:41.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.085+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8db8062b50 con 0x7f8db8102430 2024-04-22T13:31:41.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.086+0000 7f8dae7fc640 1 -- 172.21.15.87:0/3198105354 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f8db009bcc0 con 0x7f8db8102430 2024-04-22T13:31:41.081 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 msgr2=0x7f8da40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 0x7f8da40a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8db806a2f0 tx=0x7f8db4005e70 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 msgr2=0x7f8db819f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8db000b970 tx=0x7f8db000be30 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 shutdown_connections 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f8da40a3c30 0x7f8da40a60f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8db8102430 0x7f8db819f860 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8db8101b20 0x7f8db81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 --2- 172.21.15.87:0/3198105354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8101160 0x7f8db81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 >> 172.21.15.87:0/3198105354 conn(0x7f8db80fbb90 msgr2=0x7f8db8105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 shutdown_connections 2024-04-22T13:31:41.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.088+0000 7f8dbf7b6640 1 -- 172.21.15.87:0/3198105354 wait complete. 2024-04-22T13:31:41.093 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2541350067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:41.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1418627609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2004095690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:41.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2541350067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1418627609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2004095690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 -- 172.21.15.87:0/1793279500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6444103cb0 msgr2=0x7f644410a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 --2- 172.21.15.87:0/1793279500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6444103cb0 0x7f644410a540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6440009cd0 tx=0x7f644002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 -- 172.21.15.87:0/1793279500 shutdown_connections 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 --2- 172.21.15.87:0/1793279500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6444103cb0 0x7f644410a540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 --2- 172.21.15.87:0/1793279500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64441032f0 0x7f6444103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 --2- 172.21.15.87:0/1793279500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444100280 0x7f64441006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 -- 172.21.15.87:0/1793279500 >> 172.21.15.87:0/1793279500 conn(0x7f64440fbb90 msgr2=0x7f64440fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 -- 172.21.15.87:0/1793279500 shutdown_connections 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 -- 172.21.15.87:0/1793279500 wait complete. 2024-04-22T13:31:41.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.197+0000 7f644c375640 1 Processor -- start 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- start start 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6444100280 0x7f64441980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444103cb0 0x7f644419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f644410db30 con 0x7f64441032f0 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f644410d9b0 con 0x7f6444103cb0 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f644410dcb0 con 0x7f6444100280 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59824/0 (socket says 172.21.15.87:59824) 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 -- 172.21.15.87:0/2397519727 learned_addr learned my addr 172.21.15.87:0/2397519727 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644a111640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6444100280 0x7f64441980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644a912640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444103cb0 0x7f644419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 -- 172.21.15.87:0/2397519727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6444100280 msgr2=0x7f64441980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6444100280 0x7f64441980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 -- 172.21.15.87:0/2397519727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444103cb0 msgr2=0x7f644419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444103cb0 0x7f644419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 -- 172.21.15.87:0/2397519727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644419fd10 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f6449910640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f643400b4a0 tx=0x7f643400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434014070 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644419ffa0 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6434010ac0 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434017c30 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.198+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64441a04c0 con 0x7f64441032f0 2024-04-22T13:31:41.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.199+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6444109420 con 0x7f64441032f0 2024-04-22T13:31:41.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.200+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6434017070 con 0x7f64441032f0 2024-04-22T13:31:41.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.201+0000 7f643b7fe640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 0x7f64300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.201+0000 7f644a111640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 0x7f64300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.201+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f64340cfa90 con 0x7f64441032f0 2024-04-22T13:31:41.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.201+0000 7f644a111640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 0x7f64300a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f644406a380 tx=0x7f643c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.205+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f643409c510 con 0x7f64441032f0 2024-04-22T13:31:41.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2541350067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:41.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1418627609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2004095690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.421+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f644402d5c0 con 0x7f64441032f0 2024-04-22T13:31:41.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.421+0000 7f643b7fe640 1 -- 172.21.15.87:0/2397519727 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7f643409be10 con 0x7f64441032f0 2024-04-22T13:31:41.417 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:41.417 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:31:41.417 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 msgr2=0x7f64300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 0x7f64300a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f644406a380 tx=0x7f643c005f90 comp rx=0 tx=0).stop 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 msgr2=0x7f6444198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f643400b4a0 tx=0x7f643400b960 comp rx=0 tx=0).stop 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 shutdown_connections 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64300a3bb0 0x7f64300a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6444103cb0 0x7f644419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64441032f0 0x7f6444198630 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 --2- 172.21.15.87:0/2397519727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6444100280 0x7f64441980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 >> 172.21.15.87:0/2397519727 conn(0x7f64440fbb90 msgr2=0x7f64441065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 shutdown_connections 2024-04-22T13:31:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.423+0000 7f644c375640 1 -- 172.21.15.87:0/2397519727 wait complete. 2024-04-22T13:31:41.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 -- 172.21.15.87:0/823536995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920107c70 msgr2=0x7f392010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 --2- 172.21.15.87:0/823536995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920107c70 0x7f392010a0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3918009c90 tx=0x7f391802c0a0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 -- 172.21.15.87:0/823536995 shutdown_connections 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 --2- 172.21.15.87:0/823536995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920107c70 0x7f392010a0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 --2- 172.21.15.87:0/823536995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920105340 0x7f3920107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 --2- 172.21.15.87:0/823536995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920102a10 0x7f3920104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 -- 172.21.15.87:0/823536995 >> 172.21.15.87:0/823536995 conn(0x7f39200fc420 msgr2=0x7f39200fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 -- 172.21.15.87:0/823536995 shutdown_connections 2024-04-22T13:31:41.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.533+0000 7f3925a3a640 1 -- 172.21.15.87:0/823536995 wait complete. 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 Processor -- start 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 -- start start 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920102a10 0x7f39201981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920107c70 0x7f392019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f392010dae0 con 0x7f3920105340 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f392010d960 con 0x7f3920102a10 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f3925a3a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f392010dc60 con 0x7f3920107c70 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59844/0 (socket says 172.21.15.87:59844) 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 -- 172.21.15.87:0/4107862490 learned_addr learned my addr 172.21.15.87:0/4107862490 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391f7fe640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920107c70 0x7f392019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391effd640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920102a10 0x7f39201981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 -- 172.21.15.87:0/4107862490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920107c70 msgr2=0x7f392019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920107c70 0x7f392019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 -- 172.21.15.87:0/4107862490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920102a10 msgr2=0x7f39201981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920102a10 0x7f39201981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 -- 172.21.15.87:0/4107862490 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392019fe10 con 0x7f3920105340 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.534+0000 7f391e7fc640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f391400c970 tx=0x7f391400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.535+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914007bb0 con 0x7f3920105340 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.535+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3914007d50 con 0x7f3920105340 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.535+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39201a0100 con 0x7f3920105340 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.535+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391400f450 con 0x7f3920105340 2024-04-22T13:31:41.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.535+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39201a0650 con 0x7f3920105340 2024-04-22T13:31:41.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.536+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39200fdf10 con 0x7f3920105340 2024-04-22T13:31:41.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.537+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3914004f20 con 0x7f3920105340 2024-04-22T13:31:41.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.537+0000 7f3924a38640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 0x7f39080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.538+0000 7f391effd640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 0x7f39080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.538+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f39140d0380 con 0x7f3920105340 2024-04-22T13:31:41.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.538+0000 7f391effd640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 0x7f39080a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3910005fd0 tx=0x7f3910005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.542+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3914014030 con 0x7f3920105340 2024-04-22T13:31:41.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.723+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f392002d810 con 0x7f3920105340 2024-04-22T13:31:41.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.724+0000 7f3924a38640 1 -- 172.21.15.87:0/4107862490 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7f391409ce00 con 0x7f3920105340 2024-04-22T13:31:41.720 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:41.720 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 msgr2=0x7f39080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 0x7f39080a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3910005fd0 tx=0x7f3910005850 comp rx=0 tx=0).stop 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 msgr2=0x7f3920198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f391400c970 tx=0x7f391400ce30 comp rx=0 tx=0).stop 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 shutdown_connections 2024-04-22T13:31:41.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f39080a3bb0 0x7f39080a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3920107c70 0x7f392019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3920105340 0x7f3920198730 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 --2- 172.21.15.87:0/4107862490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920102a10 0x7f39201981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.726+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 >> 172.21.15.87:0/4107862490 conn(0x7f39200fc420 msgr2=0x7f3920103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.727+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 shutdown_connections 2024-04-22T13:31:41.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.727+0000 7f3925a3a640 1 -- 172.21.15.87:0/4107862490 wait complete. 2024-04-22T13:31:41.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:41.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 -- 172.21.15.87:0/2356105577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 msgr2=0x7fe0681023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 --2- 172.21.15.87:0/2356105577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe0681023c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe058009900 tx=0x7fe05802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 -- 172.21.15.87:0/2356105577 shutdown_connections 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 --2- 172.21.15.87:0/2356105577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103b50 0x7fe06810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 --2- 172.21.15.87:0/2356105577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103190 0x7fe068103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 --2- 172.21.15.87:0/2356105577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe0681023c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 -- 172.21.15.87:0/2356105577 >> 172.21.15.87:0/2356105577 conn(0x7fe0680fd530 msgr2=0x7fe0680ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 -- 172.21.15.87:0/2356105577 shutdown_connections 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.841+0000 7fe06d09a640 1 -- 172.21.15.87:0/2356105577 wait complete. 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 Processor -- start 2024-04-22T13:31:41.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 -- start start 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103190 0x7fe068198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103b50 0x7fe06819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810dae0 con 0x7fe068101fa0 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810d960 con 0x7fe068103b50 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06d09a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810dc60 con 0x7fe068103190 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59860/0 (socket says 172.21.15.87:59860) 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 -- 172.21.15.87:0/1827409003 learned_addr learned my addr 172.21.15.87:0/1827409003 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 -- 172.21.15.87:0/1827409003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103190 msgr2=0x7fe068198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe05ffff640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103190 0x7fe068198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103190 0x7fe068198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe06759e640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103b50 0x7fe06819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 -- 172.21.15.87:0/1827409003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103b50 msgr2=0x7fe06819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103b50 0x7fe06819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.842+0000 7fe066d9d640 1 -- 172.21.15.87:0/1827409003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06819fe30 con 0x7fe068101fa0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe066d9d640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe058000c00 tx=0x7fe058004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058043020 con 0x7fe068101fa0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe058004440 con 0x7fe068101fa0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0681a00c0 con 0x7fe068101fa0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058003c90 con 0x7fe068101fa0 2024-04-22T13:31:41.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.843+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0681a0610 con 0x7fe068101fa0 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.844+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe068109380 con 0x7fe068101fa0 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.845+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe058046020 con 0x7fe068101fa0 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.845+0000 7fe064d99640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 0x7fe0540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.846+0000 7fe05ffff640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 0x7fe0540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.846+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fe0580f2ad0 con 0x7fe068101fa0 2024-04-22T13:31:41.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.846+0000 7fe05ffff640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 0x7fe0540a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe050005fd0 tx=0x7fe050005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:41.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:41.849+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe05804d020 con 0x7fe068101fa0 2024-04-22T13:31:41.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:41 smithi087 ceph-mon[27999]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:41.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3198105354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:41.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2397519727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:41.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4107862490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.029+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe068062b50 con 0x7fe068101fa0 2024-04-22T13:31:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.030+0000 7fe064d99640 1 -- 172.21.15.87:0/1827409003 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe0580bf550 con 0x7fe068101fa0 2024-04-22T13:31:42.026 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 msgr2=0x7fe0540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 0x7fe0540a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe050005fd0 tx=0x7fe050005850 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 msgr2=0x7fe068198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe058000c00 tx=0x7fe058004040 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 shutdown_connections 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fe0540a3bb0 0x7fe0540a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe068103b50 0x7fe06819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe068103190 0x7fe068198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 --2- 172.21.15.87:0/1827409003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe068101fa0 0x7fe068198210 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 >> 172.21.15.87:0/1827409003 conn(0x7fe0680fd530 msgr2=0x7fe068106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 shutdown_connections 2024-04-22T13:31:42.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.032+0000 7fe06d09a640 1 -- 172.21.15.87:0/1827409003 wait complete. 2024-04-22T13:31:42.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:42.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- 172.21.15.87:0/3955368676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23341029a0 msgr2=0x7f23341092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 --2- 172.21.15.87:0/3955368676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23341029a0 0x7f23341092c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f232c009ca0 tx=0x7f232c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- 172.21.15.87:0/3955368676 shutdown_connections 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 --2- 172.21.15.87:0/3955368676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23341029a0 0x7f23341092c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 --2- 172.21.15.87:0/3955368676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2334101fe0 0x7f2334102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 --2- 172.21.15.87:0/3955368676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- 172.21.15.87:0/3955368676 >> 172.21.15.87:0/3955368676 conn(0x7f23340fc400 msgr2=0x7f23340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- 172.21.15.87:0/3955368676 shutdown_connections 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- 172.21.15.87:0/3955368676 wait complete. 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 Processor -- start 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 -- start start 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.145+0000 7f233a855640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23341029a0 0x7f233419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f233410cae0 con 0x7f2334101fe0 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f233410c960 con 0x7f23341029a0 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f233410cc60 con 0x7f2334100e90 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59882/0 (socket says 172.21.15.87:59882) 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 -- 172.21.15.87:0/1207542992 learned_addr learned my addr 172.21.15.87:0/1207542992 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f2333fff640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f2338df2640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23341029a0 0x7f233419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 -- 172.21.15.87:0/1207542992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 msgr2=0x7f23341982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 -- 172.21.15.87:0/1207542992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23341029a0 msgr2=0x7f233419f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23341029a0 0x7f233419f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 -- 172.21.15.87:0/1207542992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233419fee0 con 0x7f2334101fe0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f2333fff640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23337fe640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f232800b450 tx=0x7f232800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328015020 con 0x7f2334101fe0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2328011d40 con 0x7f2334101fe0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800f340 con 0x7f2334101fe0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23341a01d0 con 0x7f2334101fe0 2024-04-22T13:31:42.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.146+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23341a07a0 con 0x7f2334101fe0 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.147+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23341a0a60 con 0x7f2334101fe0 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.148+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2328018020 con 0x7f2334101fe0 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.149+0000 7f23317fa640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 0x7f231c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.149+0000 7f2333fff640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 0x7f231c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.149+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f23280cfea0 con 0x7f2334101fe0 2024-04-22T13:31:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.149+0000 7f2333fff640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 0x7f231c0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2324009930 tx=0x7f2324008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.153+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f232801f020 con 0x7f2334101fe0 2024-04-22T13:31:42.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:41 smithi120 ceph-mon[31101]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:42.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3198105354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:42.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2397519727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:42.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4107862490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:42.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:41 smithi132 ceph-mon[31531]: pgmap v349: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:31:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3198105354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2397519727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:42.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4107862490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:42.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.331+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f23341a0c30 con 0x7f2334101fe0 2024-04-22T13:31:42.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.332+0000 7f23317fa640 1 -- 172.21.15.87:0/1207542992 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f232800d040 con 0x7f2334101fe0 2024-04-22T13:31:42.327 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:42.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 msgr2=0x7f231c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 0x7f231c0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2324009930 tx=0x7f2324008040 comp rx=0 tx=0).stop 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 msgr2=0x7f2334198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f232800b450 tx=0x7f232800b910 comp rx=0 tx=0).stop 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 shutdown_connections 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f231c0a3bb0 0x7f231c0a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23341029a0 0x7f233419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2334101fe0 0x7f2334198800 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 --2- 172.21.15.87:0/1207542992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2334100e90 0x7f23341982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 >> 172.21.15.87:0/1207542992 conn(0x7f23340fc400 msgr2=0x7f23340fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 shutdown_connections 2024-04-22T13:31:42.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.334+0000 7f233a855640 1 -- 172.21.15.87:0/1207542992 wait complete. 2024-04-22T13:31:42.342 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:42.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.446+0000 7f2477d06640 1 -- 172.21.15.87:0/664442989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470101fe0 msgr2=0x7f2470102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.446+0000 7f2477d06640 1 --2- 172.21.15.87:0/664442989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470101fe0 0x7f2470102460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2464009900 tx=0x7f246402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- 172.21.15.87:0/664442989 shutdown_connections 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- 172.21.15.87:0/664442989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f24701092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- 172.21.15.87:0/664442989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470101fe0 0x7f2470102460 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- 172.21.15.87:0/664442989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2470100e90 0x7f24701012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- 172.21.15.87:0/664442989 >> 172.21.15.87:0/664442989 conn(0x7f24700fc400 msgr2=0x7f24700fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- 172.21.15.87:0/664442989 shutdown_connections 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- 172.21.15.87:0/664442989 wait complete. 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 Processor -- start 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- start start 2024-04-22T13:31:42.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2470101fe0 0x7f2470196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f247019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f247010b920 con 0x7f2470100e90 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f247010b7a0 con 0x7f24701029a0 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2477d06640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f247010baa0 con 0x7f2470101fe0 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2475aa2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2475aa2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59898/0 (socket says 172.21.15.87:59898) 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.447+0000 7f2475aa2640 1 -- 172.21.15.87:0/1873699265 learned_addr learned my addr 172.21.15.87:0/1873699265 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f24762a3640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f247019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 -- 172.21.15.87:0/1873699265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2470101fe0 msgr2=0x7f2470196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2470101fe0 0x7f2470196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 -- 172.21.15.87:0/1873699265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 msgr2=0x7f247019d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f247019d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 -- 172.21.15.87:0/1873699265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247019dbe0 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f24762a3640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f247019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2475aa2640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f246800c890 tx=0x7f246800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468004220 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24680043c0 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247019ded0 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468004a00 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.448+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247019e390 con 0x7f2470100e90 2024-04-22T13:31:42.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.449+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2470108270 con 0x7f2470100e90 2024-04-22T13:31:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.450+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f24680079b0 con 0x7f2470100e90 2024-04-22T13:31:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.451+0000 7f246effd640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 0x7f24580a8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.451+0000 7f24752a1640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 0x7f24580a8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.451+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f24680d0500 con 0x7f2470100e90 2024-04-22T13:31:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.451+0000 7f24752a1640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 0x7f24580a8280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2464036040 tx=0x7f2464000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.454+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2468014030 con 0x7f2470100e90 2024-04-22T13:31:42.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.632+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f247002d870 con 0x7f2470100e90 2024-04-22T13:31:42.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.632+0000 7f246effd640 1 -- 172.21.15.87:0/1873699265 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f246809cf80 con 0x7f2470100e90 2024-04-22T13:31:42.628 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 msgr2=0x7f24580a8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 0x7f24580a8280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2464036040 tx=0x7f2464000bc0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 msgr2=0x7f2470195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f246800c890 tx=0x7f246800cd50 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 shutdown_connections 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f24580a5dc0 0x7f24580a8280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24701029a0 0x7f247019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2470101fe0 0x7f2470196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 --2- 172.21.15.87:0/1873699265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2470100e90 0x7f2470195fc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 >> 172.21.15.87:0/1873699265 conn(0x7f24700fc400 msgr2=0x7f2470105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 shutdown_connections 2024-04-22T13:31:42.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.634+0000 7f2477d06640 1 -- 172.21.15.87:0/1873699265 wait complete. 2024-04-22T13:31:42.641 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:42.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- 172.21.15.87:0/297653673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 msgr2=0x7f0d3010a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 --2- 172.21.15.87:0/297653673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3010a0b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009ca0 tx=0x7f0d2802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- 172.21.15.87:0/297653673 shutdown_connections 2024-04-22T13:31:42.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 --2- 172.21.15.87:0/297653673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3010a0b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 --2- 172.21.15.87:0/297653673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30105330 0x7f0d30107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 --2- 172.21.15.87:0/297653673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30102a00 0x7f0d30104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- 172.21.15.87:0/297653673 >> 172.21.15.87:0/297653673 conn(0x7f0d300fc410 msgr2=0x7f0d300fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- 172.21.15.87:0/297653673 shutdown_connections 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- 172.21.15.87:0/297653673 wait complete. 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 Processor -- start 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.744+0000 7f0d3638f640 1 -- start start 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30102a00 0x7f0d30198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30105330 0x7f0d30198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3010dba0 con 0x7f0d30107c60 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3010da20 con 0x7f0d30105330 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3010dd20 con 0x7f0d30102a00 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59910/0 (socket says 172.21.15.87:59910) 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 -- 172.21.15.87:0/3579492052 learned_addr learned my addr 172.21.15.87:0/3579492052 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d2f7fe640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30105330 0x7f0d30198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d2ffff640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30102a00 0x7f0d30198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 -- 172.21.15.87:0/3579492052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30102a00 msgr2=0x7f0d30198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30102a00 0x7f0d30198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 -- 172.21.15.87:0/3579492052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30105330 msgr2=0x7f0d30198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30105330 0x7f0d30198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 -- 172.21.15.87:0/3579492052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3019fe40 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3492c640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c700 tx=0x7f0d28035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28039040 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d28044070 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d301a00d0 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28004440 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.745+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d301a0640 con 0x7f0d30107c60 2024-04-22T13:31:42.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.746+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d300fdf00 con 0x7f0d30107c60 2024-04-22T13:31:42.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.748+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0d280045e0 con 0x7f0d30107c60 2024-04-22T13:31:42.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.749+0000 7f0d2d7fa640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 0x7f0d180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:42.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.749+0000 7f0d2ffff640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 0x7f0d180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:42.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.749+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0d280f3570 con 0x7f0d30107c60 2024-04-22T13:31:42.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.749+0000 7f0d2ffff640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 0x7f0d180a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009890 tx=0x7f0d24008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:42.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.752+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d280bfff0 con 0x7f0d30107c60 2024-04-22T13:31:42.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.968+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d3002d5c0 con 0x7f0d30107c60 2024-04-22T13:31:42.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.968+0000 7f0d2d7fa640 1 -- 172.21.15.87:0/3579492052 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11872 (secure 0 0 0) 0x7f0d280bf8f0 con 0x7f0d30107c60 2024-04-22T13:31:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:42.964 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:29:51.322041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24482,24475]},"id":1}]} 2024-04-22T13:31:42.965 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 20 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 msgr2=0x7f0d180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 0x7f0d180a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009890 tx=0x7f0d24008040 comp rx=0 tx=0).stop 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 msgr2=0x7f0d3019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c700 tx=0x7f0d28035d60 comp rx=0 tx=0).stop 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 shutdown_connections 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f0d180a3bb0 0x7f0d180a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d30107c60 0x7f0d3019f7d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d30105330 0x7f0d30198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 --2- 172.21.15.87:0/3579492052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d30102a00 0x7f0d30198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:42.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 >> 172.21.15.87:0/3579492052 conn(0x7f0d300fc410 msgr2=0x7f0d30103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:42.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 shutdown_connections 2024-04-22T13:31:42.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:42.970+0000 7f0d3638f640 1 -- 172.21.15.87:0/3579492052 wait complete. 2024-04-22T13:31:42.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 -- 172.21.15.87:0/1712085856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30105320 msgr2=0x7f2a30107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/1712085856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30105320 0x7f2a30107710 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c009900 tx=0x7f2a1c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 -- 172.21.15.87:0/1712085856 shutdown_connections 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/1712085856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a30107c50 0x7f2a3010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/1712085856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30105320 0x7f2a30107710 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/1712085856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a301029f0 0x7f2a30104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 -- 172.21.15.87:0/1712085856 >> 172.21.15.87:0/1712085856 conn(0x7f2a300fc3e0 msgr2=0x7f2a300fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 -- 172.21.15.87:0/1712085856 shutdown_connections 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.079+0000 7f2a36cfd640 1 -- 172.21.15.87:0/1712085856 wait complete. 2024-04-22T13:31:43.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 Processor -- start 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 -- start start 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 0x7f2a30198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a30105320 0x7f2a301987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010daf0 con 0x7f2a30107c50 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010d970 con 0x7f2a301029f0 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a36cfd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010dc70 con 0x7f2a30105320 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a3529a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a3529a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59932/0 (socket says 172.21.15.87:59932) 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a3529a640 1 -- 172.21.15.87:0/2976190275 learned_addr learned my addr 172.21.15.87:0/2976190275 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a2ffff640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a30105320 0x7f2a301987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.080+0000 7f2a34a99640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 0x7f2a30198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 -- 172.21.15.87:0/2976190275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a30105320 msgr2=0x7f2a301987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a30105320 0x7f2a301987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 -- 172.21.15.87:0/2976190275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 msgr2=0x7f2a30198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 0x7f2a30198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 -- 172.21.15.87:0/2976190275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3019fe80 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a34a99640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 0x7f2a30198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a3529a640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800cad0 tx=0x7f2a2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2800ae20 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a301a0170 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a28005020 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2800f6b0 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.081+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a301a06c0 con 0x7f2a30107c50 2024-04-22T13:31:43.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.082+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a300fded0 con 0x7f2a30107c50 2024-04-22T13:31:43.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.083+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2a280051d0 con 0x7f2a30107c50 2024-04-22T13:31:43.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.083+0000 7f2a2dffb640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 0x7f2a200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.084+0000 7f2a34a99640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 0x7f2a200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.084+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f2a280d04d0 con 0x7f2a30107c50 2024-04-22T13:31:43.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.084+0000 7f2a34a99640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 0x7f2a200a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a24006fd0 tx=0x7f2a24008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.088+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a28014030 con 0x7f2a30107c50 2024-04-22T13:31:43.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:42 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1827409003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:42 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1207542992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:42 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1873699265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1827409003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1207542992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:42 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1873699265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.267+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a3002d7e0 con 0x7f2a30107c50 2024-04-22T13:31:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.269+0000 7f2a2dffb640 1 -- 172.21.15.87:0/2976190275 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22578 (secure 0 0 0) 0x7f2a2809cf50 con 0x7f2a30107c50 2024-04-22T13:31:43.265 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:43.265 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":87,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:30:37.880012+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 msgr2=0x7f2a200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 0x7f2a200a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a24006fd0 tx=0x7f2a24008040 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 msgr2=0x7f2a3019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800cad0 tx=0x7f2a2800cf90 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 shutdown_connections 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f2a200a3bb0 0x7f2a200a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a30107c50 0x7f2a3019f810 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a30105320 0x7f2a301987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 --2- 172.21.15.87:0/2976190275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a301029f0 0x7f2a30198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 >> 172.21.15.87:0/2976190275 conn(0x7f2a300fc3e0 msgr2=0x7f2a30103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 shutdown_connections 2024-04-22T13:31:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.271+0000 7f2a36cfd640 1 -- 172.21.15.87:0/2976190275 wait complete. 2024-04-22T13:31:43.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T13:29:46.666137+0000', 'modified': '2024-04-22T13:29:51.322041+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24482, 'mds_1': 24475, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24407': {'gid': 24407, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1829745193', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1829745193}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1829745193}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6837/3391616456', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 3391616456}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 3391616456}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6835/1119314155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1119314155}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1119314155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.132:6837/453938062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 453938062}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 453938062}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.132:6839/2904643536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2904643536}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2904643536}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6839/2692231328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 2692231328}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 2692231328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24482, 'qdb_cluster': [24460, 24482, 24475]} 2024-04-22T13:31:43.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T13:31:43.279 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T13:31:43.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 -- 172.21.15.87:0/3196810115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 msgr2=0x7f93cc102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/3196810115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc102460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009900 tx=0x7f93c002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 -- 172.21.15.87:0/3196810115 shutdown_connections 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/3196810115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc1029a0 0x7f93cc1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/3196810115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc102460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/3196810115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc100e90 0x7f93cc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 -- 172.21.15.87:0/3196810115 >> 172.21.15.87:0/3196810115 conn(0x7f93cc0fc400 msgr2=0x7f93cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.382+0000 7f93d0dc7640 1 -- 172.21.15.87:0/3196810115 shutdown_connections 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 -- 172.21.15.87:0/3196810115 wait complete. 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 Processor -- start 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 -- start start 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 0x7f93cc069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 0x7f93cc19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc10c9f0 con 0x7f93cc101fe0 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc10c870 con 0x7f93cc100e90 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93d0dc7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc10cb70 con 0x7f93cc1029a0 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59946/0 (socket says 172.21.15.87:59946) 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.383+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1876154164 learned_addr learned my addr 172.21.15.87:0/1876154164 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93ca59c640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 0x7f93cc069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93cad9d640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 0x7f93cc19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1876154164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 msgr2=0x7f93cc19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 0x7f93cc19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1876154164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 msgr2=0x7f93cc069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 0x7f93cc069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1876154164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc19fe50 con 0x7f93cc101fe0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93ca59c640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 0x7f93cc069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93cad9d640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 0x7f93cc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f93c0000c00 tx=0x7f93c0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93bb7fe640 1 -- 172.21.15.87:0/1876154164 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0043020 con 0x7f93cc101fe0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93bb7fe640 1 -- 172.21.15.87:0/1876154164 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c0004350 con 0x7f93cc101fe0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93cc1a00e0 con 0x7f93cc101fe0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93bb7fe640 1 -- 172.21.15.87:0/1876154164 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0003b80 con 0x7f93cc101fe0 2024-04-22T13:31:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.384+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93cc1a0630 con 0x7f93cc101fe0 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93bb7fe640 1 -- 172.21.15.87:0/1876154164 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f93c0046020 con 0x7f93cc101fe0 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 Processor -- start 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 -- start start 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6be40 0x562f5ea6c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6d2d0 0x562f5ea73eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562f5e9d0b10 con 0x562f5ea6c830 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f5e96e240 con 0x562f5ea6d2d0 2024-04-22T13:31:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93d39b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f5e9ccad0 con 0x562f5ea6be40 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59960/0 (socket says 172.21.15.87:59960) 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.387+0000 7f93ca59c640 1 -- 172.21.15.87:0/4043907256 learned_addr learned my addr 172.21.15.87:0/4043907256 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93bb7fe640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 0x7f93b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93cad9d640 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6be40 0x562f5ea6c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 -- 172.21.15.87:0/4043907256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6be40 msgr2=0x562f5ea6c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6be40 0x562f5ea6c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93cad9d640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 0x7f93b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 -- 172.21.15.87:0/4043907256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6d2d0 msgr2=0x562f5ea73eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6d2d0 0x562f5ea73eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 -- 172.21.15.87:0/4043907256 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5e6e32d0 con 0x562f5ea6c830 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93bb7fe640 1 -- 172.21.15.87:0/1876154164 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f93c00f25c0 con 0x7f93cc101fe0 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93ca59c640 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009d40 tx=0x7f93c40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c56dbab038c41e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93b8ff9640 1 -- 172.21.15.87:0/4043907256 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4012650 con 0x562f5ea6c830 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93b8ff9640 1 -- 172.21.15.87:0/4043907256 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c4009e30 con 0x562f5ea6c830 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.388+0000 7f93b8ff9640 1 -- 172.21.15.87:0/4043907256 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c400fbd0 con 0x562f5ea6c830 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- 172.21.15.87:0/4043907256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 msgr2=0x562f5ea6ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93cad9d640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 0x7f93b40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005c40 tx=0x7f93bc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009d40 tx=0x7f93c40117b0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- 172.21.15.87:0/4043907256 shutdown_connections 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6d2d0 0x562f5ea73eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea6ccb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- 172.21.15.87:0/4043907256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6be40 0x562f5ea6c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- 172.21.15.87:0/4043907256 >> 172.21.15.87:0/4043907256 conn(0x562f5ea67f40 msgr2=0x562f5ea66490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- 172.21.15.87:0/4043907256 shutdown_connections 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- 172.21.15.87:0/4043907256 wait complete. 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- start start 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 Processor -- start 2024-04-22T13:31:43.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea75ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea7d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea744b0 con 0x562f5ea6c830 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f5e959190 con 0x562f5ea6be40 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.389+0000 7f93d39b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea74630 con 0x562f5ea6d2d0 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93d39b9740 10 client.0 _ll_register_callbacks cb 0x562f5e9548b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59962/0 (socket says 172.21.15.87:59962) 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 -- 172.21.15.87:0/2307023279 learned_addr learned my addr 172.21.15.87:0/2307023279 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93cad9d640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea7d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 -- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 msgr2=0x562f5ea7d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea7d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 -- 172.21.15.87:0/2307023279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 msgr2=0x562f5ea75ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea75ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 -- 172.21.15.87:0/2307023279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5e7ad6b0 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f93c00bada0 tx=0x7f93c004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b970e720643521 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c00be2a0 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f93c00bafb0 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93c4009e30 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f93c00bf040 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93d39b9740 10 client.15237 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f5ea807b0 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.390+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c4009e30 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f93c00c7120 con 0x562f5ea6c830 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 10 client.15237 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 10 client.15237 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 20 client.15237 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5ea82e30 0x562f5ea842a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 4 client.15237 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f5e9de530 con 0x562f5ea82e30 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93ca59c640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5ea82e30 0x562f5ea842a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93ca59c640 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5ea82e30 0x562f5ea842a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38d41bf0f79d2441 server_cookie=c2e9e9d80a4a78dc in_seq=0 out_seq=0 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.391+0000 7f93b1ffb640 10 client.15237 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93b1ffb640 1 -- 172.21.15.87:0/2307023279 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f5e9de530 con 0x562f5ea82e30 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93b1ffb640 10 client.15237 handle_command_reply: tid=0 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 client.15237 shutdown 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 msgr2=0x562f5ea76400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f93c00bada0 tx=0x7f93c004f000 comp rx=0 tx=0).stop 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 shutdown_connections 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5ea82e30 0x562f5ea842a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea7d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93b1ffb640 0 client.15237 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea76400 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 --2- 172.21.15.87:0/2307023279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea75ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 >> 172.21.15.87:0/2307023279 conn(0x562f5ea67f40 msgr2=0x562f5ea65a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 shutdown_connections 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 1 -- 172.21.15.87:0/2307023279 wait complete. 2024-04-22T13:31:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.392+0000 7f93d39b9740 20 client.15237 trim_cache size 0 max 16384 2024-04-22T13:31:43.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 Processor -- start 2024-04-22T13:31:43.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 -- start start 2024-04-22T13:31:43.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea70840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea75ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea825e0 con 0x562f5ea6c830 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea82cb0 con 0x562f5ea6be40 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93d39b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea83380 con 0x562f5ea6d2d0 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.410+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59964/0 (socket says 172.21.15.87:59964) 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1039776818 learned_addr learned my addr 172.21.15.87:0/1039776818 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1039776818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 msgr2=0x562f5ea75ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93cad9d640 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea75ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea75ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1039776818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 msgr2=0x562f5ea70840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea70840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 -- 172.21.15.87:0/1039776818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5ea66be0 con 0x562f5ea6c830 2024-04-22T13:31:43.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f93c0005ab0 tx=0x7f93c002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a84f5c9b469d5f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93b3fff640 1 -- 172.21.15.87:0/1039776818 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c00be2a0 con 0x562f5ea6c830 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93b3fff640 1 -- 172.21.15.87:0/1039776818 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c00be940 con 0x562f5ea6c830 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.411+0000 7f93b3fff640 1 -- 172.21.15.87:0/1039776818 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c00be710 con 0x562f5ea6c830 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- 172.21.15.87:0/1039776818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 msgr2=0x562f5ea70d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f93c0005ab0 tx=0x7f93c002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- 172.21.15.87:0/1039776818 shutdown_connections 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea75ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea70d80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- 172.21.15.87:0/1039776818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea70840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- 172.21.15.87:0/1039776818 >> 172.21.15.87:0/1039776818 conn(0x562f5ea67f40 msgr2=0x562f5ea81b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- 172.21.15.87:0/1039776818 shutdown_connections 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- 172.21.15.87:0/1039776818 wait complete. 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- start start 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 Processor -- start 2024-04-22T13:31:43.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea78c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea77870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea84120 con 0x562f5ea6c830 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea83a50 con 0x562f5ea6be40 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f5ea847f0 con 0x562f5ea6d2d0 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93d39b9740 10 client.0 _ll_register_callbacks cb 0x7f93c00057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59970/0 (socket says 172.21.15.87:59970) 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 -- 172.21.15.87:0/3908873578 learned_addr learned my addr 172.21.15.87:0/3908873578 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93cad9d640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea77870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93ca59c640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea78c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 -- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 msgr2=0x562f5ea77870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea77870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 -- 172.21.15.87:0/3908873578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 msgr2=0x562f5ea78c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea78c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.412+0000 7f93c9d9b640 1 -- 172.21.15.87:0/3908873578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5ea77f60 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93c9d9b640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f93c0006010 tx=0x7f93c0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4df49b3eb81a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0088060 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f93c00f21a0 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93c00be940 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f93c00be2a0 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 10 client.15243 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f5eaedd20 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c00be940 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f93c00892e0 con 0x562f5ea6c830 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 10 client.15243 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 10 client.15243 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 20 client.15243 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5eaf23a0 0x562f5eaeeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 4 client.15243 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "status"}) -- 0x562f5e9d83a0 con 0x562f5eaf23a0 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.413+0000 7f93ca59c640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5eaf23a0 0x562f5eaeeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93b1ffb640 10 client.15243 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93ca59c640 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5eaf23a0 0x562f5eaeeeb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5380e9129a893319 server_cookie=81a9d958e5536da0 in_seq=0 out_seq=0 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93b1ffb640 1 -- 172.21.15.87:0/3908873578 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562f5e9d83a0 con 0x562f5eaf23a0 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93b1ffb640 10 client.15243 handle_command_reply: tid=0 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 client.15243 shutdown 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 msgr2=0x562f5ea77330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f93c0006010 tx=0x7f93c0003680 comp rx=0 tx=0).stop 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 shutdown_connections 2024-04-22T13:31:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x562f5eaf23a0 0x562f5eaeeeb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f5ea6d2d0 0x562f5ea77870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93b1ffb640 0 client.15243 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562f5ea6c830 0x562f5ea77330 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 --2- 172.21.15.87:0/3908873578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f5ea6be40 0x562f5ea78c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 >> 172.21.15.87:0/3908873578 conn(0x562f5ea67f40 msgr2=0x562f5ea81b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 shutdown_connections 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 1 -- 172.21.15.87:0/3908873578 wait complete. 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.414+0000 7f93d39b9740 20 client.15243 trim_cache size 0 max 16384 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:43.417 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 2, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24460, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 113.09477891100001, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 136.79765755400001 2024-04-22T13:31:43.418 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 msgr2=0x7f93b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 0x7f93b40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005c40 tx=0x7f93bc009500 comp rx=0 tx=0).stop 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 msgr2=0x7f93cc19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f93c0000c00 tx=0x7f93c0005dc0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 shutdown_connections 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f93b40a3bb0 0x7f93b40a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93cc1029a0 0x7f93cc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93cc101fe0 0x7f93cc19cf50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 --2- 172.21.15.87:0/1876154164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93cc100e90 0x7f93cc069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 >> 172.21.15.87:0/1876154164 conn(0x7f93cc0fc400 msgr2=0x7f93cc105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 shutdown_connections 2024-04-22T13:31:43.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.415+0000 7f93d0dc7640 1 -- 172.21.15.87:0/1876154164 wait complete. 2024-04-22T13:31:43.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:42 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1827409003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:42 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1207542992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:42 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1873699265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:43.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T13:31:43.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- 172.21.15.87:0/773867940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103190 msgr2=0x7fd608103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 --2- 172.21.15.87:0/773867940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103190 0x7fd608103610 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009ca0 tx=0x7fd5fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- 172.21.15.87:0/773867940 shutdown_connections 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 --2- 172.21.15.87:0/773867940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103b50 0x7fd60810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 --2- 172.21.15.87:0/773867940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103190 0x7fd608103610 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 --2- 172.21.15.87:0/773867940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 0x7fd6081023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- 172.21.15.87:0/773867940 >> 172.21.15.87:0/773867940 conn(0x7fd6080fd530 msgr2=0x7fd6080ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- 172.21.15.87:0/773867940 shutdown_connections 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- 172.21.15.87:0/773867940 wait complete. 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 Processor -- start 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.524+0000 7fd60c998640 1 -- start start 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 0x7fd608198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103190 0x7fd608198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd60810dae0 con 0x7fd608103b50 2024-04-22T13:31:43.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd60810d960 con 0x7fd608103190 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd60810dc60 con 0x7fd608101fa0 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59976/0 (socket says 172.21.15.87:59976) 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 -- 172.21.15.87:0/3401728827 learned_addr learned my addr 172.21.15.87:0/3401728827 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60659c640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 0x7fd608198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd605d9b640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103190 0x7fd608198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 -- 172.21.15.87:0/3401728827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 msgr2=0x7fd608198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 0x7fd608198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 -- 172.21.15.87:0/3401728827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103190 msgr2=0x7fd608198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103190 0x7fd608198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 -- 172.21.15.87:0/3401728827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60819fe50 con 0x7fd608103b50 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd606d9d640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c970 tx=0x7fd5f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd5f77fe640 1 -- 172.21.15.87:0/3401728827 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8007bb0 con 0x7fd608103b50 2024-04-22T13:31:43.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd5f77fe640 1 -- 172.21.15.87:0/3401728827 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5f8007d50 con 0x7fd608103b50 2024-04-22T13:31:43.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6081a0140 con 0x7fd608103b50 2024-04-22T13:31:43.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd5f77fe640 1 -- 172.21.15.87:0/3401728827 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8004e60 con 0x7fd608103b50 2024-04-22T13:31:43.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.525+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6081a0610 con 0x7fd608103b50 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.527+0000 7fd5f77fe640 1 -- 172.21.15.87:0/3401728827 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd5f8005020 con 0x7fd608103b50 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 Processor -- start 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 -- start start 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389250 0x561c07389670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 0x561c073912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561c072e9b10 con 0x561c07389c40 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c07287240 con 0x561c0738a6e0 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60f581740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561c072e5ad0 con 0x561c07389250 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd5f77fe640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 0x7fd5f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59982/0 (socket says 172.21.15.87:59982) 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60659c640 1 -- 172.21.15.87:0/362403103 learned_addr learned my addr 172.21.15.87:0/362403103 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd605d9b640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 0x561c073912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd606d9d640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 0x7fd5f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd605d9b640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389250 0x561c07389670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60659c640 1 -- 172.21.15.87:0/362403103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389250 msgr2=0x561c07389670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.528+0000 7fd60659c640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389250 0x561c07389670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd60659c640 1 -- 172.21.15.87:0/362403103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 msgr2=0x561c073912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd60659c640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 0x561c073912c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd60659c640 1 -- 172.21.15.87:0/362403103 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c06ffc2d0 con 0x561c07389c40 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd605d9b640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 0x561c073912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd5f77fe640 1 -- 172.21.15.87:0/3401728827 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd5f80d0030 con 0x7fd608103b50 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd60659c640 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd600009ca0 tx=0x7fd600011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f23a5593e25f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd606d9d640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 0x7fd5f00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c940 tx=0x7fd5f806de40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd5f4ff9640 1 -- 172.21.15.87:0/362403103 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600004220 con 0x561c07389c40 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd5f4ff9640 1 -- 172.21.15.87:0/362403103 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000043c0 con 0x561c07389c40 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.529+0000 7fd5f4ff9640 1 -- 172.21.15.87:0/362403103 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600019830 con 0x561c07389c40 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- 172.21.15.87:0/362403103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 msgr2=0x561c0738a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd600009ca0 tx=0x7fd600011810 comp rx=0 tx=0).stop 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- 172.21.15.87:0/362403103 shutdown_connections 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c0738a6e0 0x561c073912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389c40 0x561c0738a0c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- 172.21.15.87:0/362403103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389250 0x561c07389670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- 172.21.15.87:0/362403103 >> 172.21.15.87:0/362403103 conn(0x561c07380f40 msgr2=0x561c0737f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- 172.21.15.87:0/362403103 shutdown_connections 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- 172.21.15.87:0/362403103 wait complete. 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- start start 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 Processor -- start 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c073932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c073937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561c073918c0 con 0x561c0738a6e0 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c07272190 con 0x561c07389250 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561c07391a40 con 0x561c07389c40 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd60f581740 10 client.0 _ll_register_callbacks cb 0x561c0726d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59992/0 (socket says 172.21.15.87:59992) 2024-04-22T13:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.530+0000 7fd605d9b640 1 -- 172.21.15.87:0/1875103016 learned_addr learned my addr 172.21.15.87:0/1875103016 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60659c640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c073937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c073932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 -- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 msgr2=0x561c073937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c073937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 -- 172.21.15.87:0/1875103016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 msgr2=0x561c073932a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c073932a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 -- 172.21.15.87:0/1875103016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c070c66b0 con 0x561c0738a6e0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60659c640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c073937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd605d9b640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc032520 tx=0x7fd5fc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7988a88e0ff943d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc048020 con 0x561c0738a6e0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd5fc003b70 con 0x561c0738a6e0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6000043c0 con 0x561c0738a6e0 2024-04-22T13:31:43.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd5fc039040 con 0x561c0738a6e0 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60f581740 10 client.15255 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561c0739dc00 con 0x561c0738a6e0 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000043c0 con 0x561c0738a6e0 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fd5fc03f030 con 0x561c0738a6e0 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.531+0000 7fd60f581740 10 client.15255 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60f581740 10 client.15255 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:31:43.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60f581740 20 client.15255 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c073a0280 0x561c073a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60f581740 4 client.15255 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c072f7530 con 0x561c073a0280 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60659c640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c073a0280 0x561c073a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd5edffb640 10 client.15255 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.532+0000 7fd60659c640 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c073a0280 0x561c073a16f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c73c56fd7840ce server_cookie=abfabf1b86490e0c in_seq=0 out_seq=0 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd5edffb640 1 -- 172.21.15.87:0/1875103016 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c072f7530 con 0x561c073a0280 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd5edffb640 10 client.15255 handle_command_reply: tid=0 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 client.15255 shutdown 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 msgr2=0x561c0739a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc032520 tx=0x7fd5fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 shutdown_connections 2024-04-22T13:31:43.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c073a0280 0x561c073a16f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a820 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c073937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd5edffb640 0 client.15255 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 --2- 172.21.15.87:0/1875103016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c073932a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 >> 172.21.15.87:0/1875103016 conn(0x561c07380f40 msgr2=0x561c0737ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 shutdown_connections 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 1 -- 172.21.15.87:0/1875103016 wait complete. 2024-04-22T13:31:43.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.533+0000 7fd60f581740 20 client.15255 trim_cache size 0 max 16384 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 Processor -- start 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 -- start start 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c0738dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c0738e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561c0739fa30 con 0x561c0738a6e0 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c073a0100 con 0x561c07389250 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd60f581740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561c073a07d0 con 0x561c07389c40 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60000/0 (socket says 172.21.15.87:60000) 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.551+0000 7fd605d9b640 1 -- 172.21.15.87:0/2666959953 learned_addr learned my addr 172.21.15.87:0/2666959953 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 -- 172.21.15.87:0/2666959953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 msgr2=0x561c0738e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c0738e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 -- 172.21.15.87:0/2666959953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 msgr2=0x561c0738dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c0738dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 -- 172.21.15.87:0/2666959953 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0737ae70 con 0x561c0738a6e0 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd605d9b640 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc000f80 tx=0x7fd5fc0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9abbb57cc9ec0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd5effff640 1 -- 172.21.15.87:0/2666959953 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc048030 con 0x561c0738a6e0 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd5effff640 1 -- 172.21.15.87:0/2666959953 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc0359f0 con 0x561c0738a6e0 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd5effff640 1 -- 172.21.15.87:0/2666959953 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc004380 con 0x561c0738a6e0 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 -- 172.21.15.87:0/2666959953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 msgr2=0x561c0739a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc000f80 tx=0x7fd5fc0095a0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 -- 172.21.15.87:0/2666959953 shutdown_connections 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c0738a6e0 0x561c0739a740 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c07389c40 0x561c0738e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.552+0000 7fd60f581740 1 --2- 172.21.15.87:0/2666959953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389250 0x561c0738dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- 172.21.15.87:0/2666959953 >> 172.21.15.87:0/2666959953 conn(0x561c07380f40 msgr2=0x561c073912a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- 172.21.15.87:0/2666959953 shutdown_connections 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- 172.21.15.87:0/2666959953 wait complete. 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- start start 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 Processor -- start 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 0x561c07398c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c0738a6e0 0x561c073991b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561c073a1570 con 0x561c07389250 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c073a0ea0 con 0x561c07389c40 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561c073a1c40 con 0x561c0738a6e0 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60f581740 10 client.0 _ll_register_callbacks cb 0x7fd5fc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60004/0 (socket says 172.21.15.87:60004) 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 -- 172.21.15.87:0/550093247 learned_addr learned my addr 172.21.15.87:0/550093247 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60659c640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 0x561c07398c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 -- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c0738a6e0 msgr2=0x561c073991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c0738a6e0 0x561c073991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 -- 172.21.15.87:0/550093247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 msgr2=0x561c07398c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 0x561c07398c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd605d9b640 1 -- 172.21.15.87:0/550093247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c073998a0 con 0x561c07389250 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.553+0000 7fd60659c640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 0x561c07398c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd605d9b640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc035770 tx=0x7fd5fc035680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fc6f4f35ef4590 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc048070 con 0x561c07389250 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd5fc004520 con 0x561c07389250 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5fc0359f0 con 0x561c07389250 2024-04-22T13:31:43.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd5fc004720 con 0x561c07389250 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 10 client.15261 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561c07406230 con 0x561c07389250 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc0359f0 con 0x561c07389250 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fd5fc060030 con 0x561c07389250 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 10 client.15261 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 10 client.15261 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 20 client.15261 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c0740a980 0x561c07409690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 4 client.15261 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.554+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "status"}) -- 0x561c072f13a0 con 0x561c0740a980 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60659c640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c0740a980 0x561c07409690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd5edffb640 10 client.15261 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60659c640 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c0740a980 0x561c07409690 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=469b761d76628c0d server_cookie=316c83c450cc5a4c in_seq=0 out_seq=0 2024-04-22T13:31:43.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd5edffb640 1 -- 172.21.15.87:0/550093247 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561c072f13a0 con 0x561c0740a980 2024-04-22T13:31:43.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd5edffb640 10 client.15261 handle_command_reply: tid=0 2024-04-22T13:31:43.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 client.15261 shutdown 2024-04-22T13:31:43.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 msgr2=0x561c07396720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc035770 tx=0x7fd5fc035680 comp rx=0 tx=0).stop 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 shutdown_connections 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x561c0740a980 0x561c07409690 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561c0738a6e0 0x561c073991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c07389c40 0x561c07398c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 --2- 172.21.15.87:0/550093247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561c07389250 0x561c07396720 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd5edffb640 0 client.15261 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 >> 172.21.15.87:0/550093247 conn(0x561c07380f40 msgr2=0x561c07392eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 shutdown_connections 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 1 -- 172.21.15.87:0/550093247 wait complete. 2024-04-22T13:31:43.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.555+0000 7fd60f581740 20 client.15261 trim_cache size 0 max 16384 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 1, 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24475, 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 114.24918619499999, 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:43.560 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:43.561 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 131.29998742699999 2024-04-22T13:31:43.561 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:43.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 msgr2=0x7fd5f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 0x7fd5f00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c940 tx=0x7fd5f806de40 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 msgr2=0x7fd60819f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c970 tx=0x7fd5f800ce30 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 shutdown_connections 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fd5f00a3bb0 0x7fd5f00a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd608103b50 0x7fd60819f7e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd608103190 0x7fd608198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 --2- 172.21.15.87:0/3401728827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd608101fa0 0x7fd608198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 >> 172.21.15.87:0/3401728827 conn(0x7fd6080fd530 msgr2=0x7fd6081065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 shutdown_connections 2024-04-22T13:31:43.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.556+0000 7fd60c998640 1 -- 172.21.15.87:0/3401728827 wait complete. 2024-04-22T13:31:43.563 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- 172.21.15.87:0/2849554062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 msgr2=0x7f15681039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 --2- 172.21.15.87:0/2849554062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f15681039a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1558009ca0 tx=0x7f155802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- 172.21.15.87:0/2849554062 shutdown_connections 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 --2- 172.21.15.87:0/2849554062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 0x7f156810a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 --2- 172.21.15.87:0/2849554062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f1568100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 --2- 172.21.15.87:0/2849554062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f15681039a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- 172.21.15.87:0/2849554062 >> 172.21.15.87:0/2849554062 conn(0x7f15680fbb90 msgr2=0x7f15680fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- 172.21.15.87:0/2849554062 shutdown_connections 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- 172.21.15.87:0/2849554062 wait complete. 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 Processor -- start 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.667+0000 7f156e23a640 1 -- start start 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f15681981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 0x7f15681986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f156810db10 con 0x7f1568103580 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f156810d990 con 0x7f15680ffff0 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f156810dc90 con 0x7f15681009b0 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60020/0 (socket says 172.21.15.87:60020) 2024-04-22T13:31:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 -- 172.21.15.87:0/3427477881 learned_addr learned my addr 172.21.15.87:0/3427477881 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1566ffd640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 0x7f15681986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f15677fe640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f15681981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 -- 172.21.15.87:0/3427477881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 msgr2=0x7f15681986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 0x7f15681986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 -- 172.21.15.87:0/3427477881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 msgr2=0x7f15681981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f15681981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 -- 172.21.15.87:0/3427477881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156819fdc0 con 0x7f1568103580 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f15677fe640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f15681981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1567fff640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f156000d750 tx=0x7f156000db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1564ff9640 1 -- 172.21.15.87:0/3427477881 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560017070 con 0x7f1568103580 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1564ff9640 1 -- 172.21.15.87:0/3427477881 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15600097c0 con 0x7f1568103580 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15681a00b0 con 0x7f1568103580 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.668+0000 7f1564ff9640 1 -- 172.21.15.87:0/3427477881 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560004ba0 con 0x7f1568103580 2024-04-22T13:31:43.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.669+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15681a0600 con 0x7f1568103580 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1564ff9640 1 -- 172.21.15.87:0/3427477881 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1560009c90 con 0x7f1568103580 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 Processor -- start 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 -- start start 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 0x55d725189260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725190eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d7250edb10 con 0x55d725189830 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d72508b240 con 0x55d72518a2d0 2024-04-22T13:31:43.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.671+0000 7f1570e20740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7250e9ad0 con 0x55d725188e40 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 0x55d725189260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60028/0 (socket says 172.21.15.87:60028) 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 -- 172.21.15.87:0/2101667064 learned_addr learned my addr 172.21.15.87:0/2101667064 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1566ffd640 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725190eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1564ff9640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 0x7f15540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1567fff640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 0x7f15540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 -- 172.21.15.87:0/2101667064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 msgr2=0x55d725189260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 0x55d725189260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 -- 172.21.15.87:0/2101667064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 msgr2=0x55d725190eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725190eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 -- 172.21.15.87:0/2101667064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d724e002d0 con 0x55d725189830 2024-04-22T13:31:43.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1566ffd640 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 0x55d725189260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1564ff9640 1 -- 172.21.15.87:0/3427477881 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f15600cfd30 con 0x7f1568103580 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f15677fe640 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1558032380 tx=0x7f155802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d447d3041f1cae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f1567fff640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 0x7f15540a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f156001d000 tx=0x7f156006ebd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f155dffb640 1 -- 172.21.15.87:0/2101667064 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558043020 con 0x55d725189830 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f155dffb640 1 -- 172.21.15.87:0/2101667064 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1558032d80 con 0x55d725189830 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.672+0000 7f155dffb640 1 -- 172.21.15.87:0/2101667064 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558039040 con 0x55d725189830 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- 172.21.15.87:0/2101667064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 msgr2=0x55d725189cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1558032380 tx=0x7f155802cd20 comp rx=0 tx=0).stop 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- 172.21.15.87:0/2101667064 shutdown_connections 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725190eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725189830 0x55d725189cb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 --2- 172.21.15.87:0/2101667064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725188e40 0x55d725189260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- 172.21.15.87:0/2101667064 >> 172.21.15.87:0/2101667064 conn(0x55d725184f40 msgr2=0x55d725183490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- 172.21.15.87:0/2101667064 shutdown_connections 2024-04-22T13:31:43.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- 172.21.15.87:0/2101667064 wait complete. 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.673+0000 7f1570e20740 1 -- start start 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 Processor -- start 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d725189830 0x55d7251977b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d72518a2d0 0x55d72519e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d725195860 con 0x55d725188e40 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d725076190 con 0x55d725189830 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7251959e0 con 0x55d72518a2d0 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 10 client.0 _ll_register_callbacks cb 0x55d7250718b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60036/0 (socket says 172.21.15.87:60036) 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 -- 172.21.15.87:0/1516831454 learned_addr learned my addr 172.21.15.87:0/1516831454 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f15677fe640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d725189830 0x55d7251977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 -- 172.21.15.87:0/1516831454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d72518a2d0 msgr2=0x55d72519e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d72518a2d0 0x55d72519e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 -- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d725189830 msgr2=0x55d7251977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d725189830 0x55d7251977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 -- 172.21.15.87:0/1516831454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d724eca6b0 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f155000ac10 tx=0x7f155000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ae38623739cb89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1546ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550005220 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1558032d80 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.674+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f15500053c0 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f15500055c0 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 10 client.15273 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d7251a1c20 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1558032d80 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f155000e8c0 con 0x55d725188e40 2024-04-22T13:31:43.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 10 client.15273 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 20 client.15273 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7251a4280 0x55d7251a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 4 client.15273 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7250fb530 con 0x55d7251a4280 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.675+0000 7f15677fe640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7251a4280 0x55d7251a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.676+0000 7f15677fe640 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7251a4280 0x55d7251a56f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b13ca42b438c7a11 server_cookie=c6dcf2550be2e80b in_seq=0 out_seq=0 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.676+0000 7f1546ffd640 10 client.15273 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.676+0000 7f1546ffd640 1 -- 172.21.15.87:0/1516831454 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7250fb530 con 0x55d7251a4280 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.676+0000 7f1546ffd640 10 client.15273 handle_command_reply: tid=0 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.676+0000 7f1570e20740 1 client.15273 shutdown 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 msgr2=0x55d725197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f155000ac10 tx=0x7f155000f7f0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 shutdown_connections 2024-04-22T13:31:43.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7251a4280 0x55d7251a56f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d72518a2d0 0x55d72519e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d725189830 0x55d7251977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1546ffd640 0 client.15273 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 --2- 172.21.15.87:0/1516831454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d725197270 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 >> 172.21.15.87:0/1516831454 conn(0x55d725184f40 msgr2=0x55d725182a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 shutdown_connections 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 1 -- 172.21.15.87:0/1516831454 wait complete. 2024-04-22T13:31:43.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.677+0000 7f1570e20740 20 client.15273 trim_cache size 0 max 16384 2024-04-22T13:31:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.694+0000 7f1570e20740 1 Processor -- start 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.694+0000 7f1570e20740 1 -- start start 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725183400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a3a30 con 0x55d725188e40 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a4100 con 0x55d72518a2d0 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1570e20740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a47d0 con 0x55d725189830 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60046/0 (socket says 172.21.15.87:60046) 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 -- 172.21.15.87:0/2107967349 learned_addr learned my addr 172.21.15.87:0/2107967349 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 -- 172.21.15.87:0/2107967349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 msgr2=0x55d72519e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 -- 172.21.15.87:0/2107967349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 msgr2=0x55d725183400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725183400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 -- 172.21.15.87:0/2107967349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d725184d40 con 0x55d725188e40 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f1566ffd640 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f155000f000 tx=0x7f155000af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d1c80ed003dd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f155d7fa640 1 -- 172.21.15.87:0/2107967349 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550011230 con 0x55d725188e40 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f155d7fa640 1 -- 172.21.15.87:0/2107967349 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155000ea70 con 0x55d725188e40 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.695+0000 7f155d7fa640 1 -- 172.21.15.87:0/2107967349 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550010ad0 con 0x55d725188e40 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- 172.21.15.87:0/2107967349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 msgr2=0x55d72519f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f155000f000 tx=0x7f155000af00 comp rx=0 tx=0).stop 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- 172.21.15.87:0/2107967349 shutdown_connections 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d725183400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 --2- 172.21.15.87:0/2107967349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519f240 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- 172.21.15.87:0/2107967349 >> 172.21.15.87:0/2107967349 conn(0x55d725184f40 msgr2=0x55d72518d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- 172.21.15.87:0/2107967349 shutdown_connections 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- 172.21.15.87:0/2107967349 wait complete. 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 -- start start 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 Processor -- start 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.696+0000 7f1570e20740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d72519d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a5570 con 0x55d725188e40 2024-04-22T13:31:43.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a4ea0 con 0x55d72518a2d0 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7251a5c40 con 0x55d725189830 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 10 client.0 _ll_register_callbacks cb 0x7f155000acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60056/0 (socket says 172.21.15.87:60056) 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 -- 172.21.15.87:0/1397881058 learned_addr learned my addr 172.21.15.87:0/1397881058 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f15677fe640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 -- 172.21.15.87:0/1397881058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 msgr2=0x55d72519cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 -- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 msgr2=0x55d72519d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d72519d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 -- 172.21.15.87:0/1397881058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d72519d6c0 con 0x55d725188e40 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f15677fe640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1566ffd640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f155000ec60 tx=0x7f15500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3845c4d4f7e4a197 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1546ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155003a020 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f1550032640 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155000ea70 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f155000e890 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 10 client.15279 fetch_fsmap learned FSMap version 20 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d72519e9c0 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.697+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155000ea70 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1550041430 con 0x55d725188e40 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 10 client.15279 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 20 client.15279 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d72520e6e0 0x55d72520b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 4 client.15279 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:31:43.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "status"}) -- 0x55d7250f53a0 con 0x55d72520e6e0 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f15677fe640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d72520e6e0 0x55d72520b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f15677fe640 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d72520e6e0 0x55d72520b1f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae97656b70ac13da server_cookie=58b6f878460addee in_seq=0 out_seq=0 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1546ffd640 10 client.15279 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1546ffd640 1 -- 172.21.15.87:0/1397881058 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7250f53a0 con 0x55d72520e6e0 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1546ffd640 10 client.15279 handle_command_reply: tid=0 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.698+0000 7f1570e20740 1 client.15279 shutdown 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 msgr2=0x55d72519a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f155000ec60 tx=0x7f15500057f0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 shutdown_connections 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d72520e6e0 0x55d72520b1f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d72518a2d0 0x55d72519d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1546ffd640 0 client.15279 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d725189830 0x55d72519cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 --2- 172.21.15.87:0/1397881058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d725188e40 0x55d72519a6f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 >> 172.21.15.87:0/1397881058 conn(0x55d725184f40 msgr2=0x55d725183400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 shutdown_connections 2024-04-22T13:31:43.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 1 -- 172.21.15.87:0/1397881058 wait complete. 2024-04-22T13:31:43.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f1570e20740 20 client.15279 trim_cache size 0 max 16384 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 0, 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24482, 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 116.37280077299999, 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 20, 2024-04-22T13:31:43.707 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 87, 2024-04-22T13:31:43.708 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T13:31:43.708 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 122.82227961700001 2024-04-22T13:31:43.708 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 msgr2=0x7f15540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 0x7f15540a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f156001d000 tx=0x7f156006ebd0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 msgr2=0x7f156819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f156000d750 tx=0x7f156000db50 comp rx=0 tx=0).stop 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 shutdown_connections 2024-04-22T13:31:43.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f15540a3bb0 0x7f15540a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1568103580 0x7f156819f750 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15681009b0 0x7f15681986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 --2- 172.21.15.87:0/3427477881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15680ffff0 0x7f15681981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 >> 172.21.15.87:0/3427477881 conn(0x7f15680fbb90 msgr2=0x7f15681066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 shutdown_connections 2024-04-22T13:31:43.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.699+0000 7f156e23a640 1 -- 172.21.15.87:0/3427477881 wait complete. 2024-04-22T13:31:43.710 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T13:31:43.720 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T13:31:43.721 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T13:31:43.721 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T13:31:43.721 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T13:31:43.722 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T13:31:43.722 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:31:43.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-22T13:31:43.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- 172.21.15.87:0/3152238420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102a00 msgr2=0x7fc618109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3152238420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102a00 0x7fc618109320 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc608009900 tx=0x7fc60802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- 172.21.15.87:0/3152238420 shutdown_connections 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3152238420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102a00 0x7fc618109320 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3152238420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102040 0x7fc6181024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3152238420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618100e60 0x7fc618101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- 172.21.15.87:0/3152238420 >> 172.21.15.87:0/3152238420 conn(0x7fc6180fc420 msgr2=0x7fc6180fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- 172.21.15.87:0/3152238420 shutdown_connections 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- 172.21.15.87:0/3152238420 wait complete. 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 Processor -- start 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- start start 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102040 0x7fc618198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 0x7fc61819f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810db30 con 0x7fc618100e60 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810d9b0 con 0x7fc618102a00 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.824+0000 7fc61ed32640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810dcb0 con 0x7fc618102040 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60078/0 (socket says 172.21.15.87:60078) 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 -- 172.21.15.87:0/3153476289 learned_addr learned my addr 172.21.15.87:0/3153476289 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc617fff640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102040 0x7fc618198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61d2cf640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 0x7fc61819f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 -- 172.21.15.87:0/3153476289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102040 msgr2=0x7fc618198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102040 0x7fc618198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 -- 172.21.15.87:0/3153476289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 msgr2=0x7fc61819f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 0x7fc61819f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 -- 172.21.15.87:0/3153476289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61819fd00 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61d2cf640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 0x7fc61819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61cace640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc61000ebc0 tx=0x7fc61000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610003b70 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc610003d10 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61819fff0 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610019800 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.825+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a04b0 con 0x7fc618100e60 2024-04-22T13:31:43.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.826+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc618108200 con 0x7fc618100e60 2024-04-22T13:31:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.827+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc6100199a0 con 0x7fc618100e60 2024-04-22T13:31:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.828+0000 7fc615ffb640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 0x7fc6040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.828+0000 7fc617fff640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 0x7fc6040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.828+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fc6100d0ce0 con 0x7fc618100e60 2024-04-22T13:31:43.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.828+0000 7fc617fff640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 0x7fc6040a6190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc6181995f0 tx=0x7fc60000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:43.833+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc61009d760 con 0x7fc618100e60 2024-04-22T13:31:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.061+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc61802d5c0 con 0x7fc618100e60 2024-04-22T13:31:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:43 smithi132 ceph-mon[31531]: pgmap v350: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:43 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3579492052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:43 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2976190275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:43 smithi087 ceph-mon[27999]: pgmap v350: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:43 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3579492052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:43 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2976190275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:43 smithi120 ceph-mon[31101]: pgmap v350: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-22T13:31:44.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:43 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3579492052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:43 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2976190275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:44.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.938+0000 7fc615ffb640 1 -- 172.21.15.87:0/3153476289 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fc61802d5c0 con 0x7fc618100e60 2024-04-22T13:31:44.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.940+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 msgr2=0x7fc6040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:44.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.940+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 0x7fc6040a6190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc6181995f0 tx=0x7fc60000a4c0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.940+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 msgr2=0x7fc6181980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:44.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.940+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc61000ebc0 tx=0x7fc61000c6a0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 shutdown_connections 2024-04-22T13:31:44.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc6040a3cd0 0x7fc6040a6190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102a00 0x7fc61819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102040 0x7fc618198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 --2- 172.21.15.87:0/3153476289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618100e60 0x7fc6181980e0 secure :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc61000ebc0 tx=0x7fc61000c6a0 comp rx=0 tx=0).stop 2024-04-22T13:31:44.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.941+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 >> 172.21.15.87:0/3153476289 conn(0x7fc6180fc420 msgr2=0x7fc6181053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:44.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.943+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 shutdown_connections 2024-04-22T13:31:44.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:44.943+0000 7fc61ed32640 1 -- 172.21.15.87:0/3153476289 wait complete. 2024-04-22T13:31:44.957 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-22T13:31:45.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 -- 172.21.15.87:0/3290820961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c102a30 msgr2=0x7f755c104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 --2- 172.21.15.87:0/3290820961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c102a30 0x7f755c104e20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f754c009900 tx=0x7f754c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 -- 172.21.15.87:0/3290820961 shutdown_connections 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 --2- 172.21.15.87:0/3290820961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c107c90 0x7f755c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 --2- 172.21.15.87:0/3290820961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c105360 0x7f755c107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 --2- 172.21.15.87:0/3290820961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c102a30 0x7f755c104e20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 -- 172.21.15.87:0/3290820961 >> 172.21.15.87:0/3290820961 conn(0x7f755c0fc420 msgr2=0x7f755c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 -- 172.21.15.87:0/3290820961 shutdown_connections 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.066+0000 7f7564081640 1 -- 172.21.15.87:0/3290820961 wait complete. 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 Processor -- start 2024-04-22T13:31:45.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 -- start start 2024-04-22T13:31:45.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c102a30 0x7f755c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c107c90 0x7f755c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f755c10daf0 con 0x7f755c105360 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f755c10d970 con 0x7f755c102a30 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7564081640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f755c10dc70 con 0x7f755c107c90 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f756161c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f756161c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60116/0 (socket says 172.21.15.87:60116) 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f756161c640 1 -- 172.21.15.87:0/2648570223 learned_addr learned my addr 172.21.15.87:0/2648570223 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.067+0000 7f7561e1d640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c102a30 0x7f755c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756261e640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c107c90 0x7f755c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 -- 172.21.15.87:0/2648570223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c107c90 msgr2=0x7f755c19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c107c90 0x7f755c19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 -- 172.21.15.87:0/2648570223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c102a30 msgr2=0x7f755c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c102a30 0x7f755c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 -- 172.21.15.87:0/2648570223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755c19fdc0 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f756161c640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f755800d740 tx=0x7f755800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558017070 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f755800de50 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558004ce0 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755c1a00b0 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.068+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755c1a0600 con 0x7f755c105360 2024-04-22T13:31:45.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.069+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f755c0fdf10 con 0x7f755c105360 2024-04-22T13:31:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.071+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f7558004e80 con 0x7f755c105360 2024-04-22T13:31:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.071+0000 7f7552ffd640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 0x7f75480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.071+0000 7f7561e1d640 1 -- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 msgr2=0x7f75480a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.072+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f75580cfc30 con 0x7f755c105360 2024-04-22T13:31:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.072+0000 7f7561e1d640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 0x7f75480a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:31:45.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.076+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f755809c060 con 0x7f755c105360 2024-04-22T13:31:45.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:44 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T13:31:45.213 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:44 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: ignoring --setuser ceph since I am not root 2024-04-22T13:31:45.213 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:44 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: ignoring --setgroup ceph since I am not root 2024-04-22T13:31:45.213 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:44 smithi132 ceph-mgr[33386]: -- 172.21.15.132:0/897206005 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d55c305a0 con 0x555d55c04800 2024-04-22T13:31:45.213 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:45.117+0000 7f9820e7a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:31:45.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:44 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T13:31:45.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setuser ceph since I am not root 2024-04-22T13:31:45.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: ignoring --setgroup ceph since I am not root 2024-04-22T13:31:45.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:45.127+0000 7f07b55cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:31:45.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.262+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f755c02d5c0 con 0x7f755c105360 2024-04-22T13:31:45.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.266+0000 7f7552ffd640 1 -- 172.21.15.87:0/2648570223 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f75580a0360 con 0x7f755c105360 2024-04-22T13:31:45.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 msgr2=0x7f75480a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:31:45.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 0x7f75480a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 msgr2=0x7f755c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f755800d740 tx=0x7f755800dc00 comp rx=0 tx=0).stop 2024-04-22T13:31:45.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 shutdown_connections 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f75480a3d20 0x7f75480a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f755c107c90 0x7f755c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f755c105360 0x7f755c1986e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 --2- 172.21.15.87:0/2648570223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f755c102a30 0x7f755c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.268+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 >> 172.21.15.87:0/2648570223 conn(0x7f755c0fc420 msgr2=0x7f755c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.269+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 shutdown_connections 2024-04-22T13:31:45.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.269+0000 7f7564081640 1 -- 172.21.15.87:0/2648570223 wait complete. 2024-04-22T13:31:45.276 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-22T13:31:45.300 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:44 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T13:31:45.301 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:44 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: ignoring --setuser ceph since I am not root 2024-04-22T13:31:45.301 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:44 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: ignoring --setgroup ceph since I am not root 2024-04-22T13:31:45.301 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:44 smithi120 ceph-mgr[32492]: -- 172.21.15.120:0/2344978620 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed174a25a0 con 0x55ed17476c00 2024-04-22T13:31:45.301 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:45.122+0000 7f6cecf9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T13:31:45.301 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:45.304+0000 7f6cecf9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:45.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- 172.21.15.87:0/1259144465 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904102af0 msgr2=0x7fc904104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 --2- 172.21.15.87:0/1259144465 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904102af0 0x7fc904104ee0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009ca0 tx=0x7fc8fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- 172.21.15.87:0/1259144465 shutdown_connections 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 --2- 172.21.15.87:0/1259144465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904107d50 0x7fc90410a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 --2- 172.21.15.87:0/1259144465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 0x7fc904107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 --2- 172.21.15.87:0/1259144465 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904102af0 0x7fc904104ee0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- 172.21.15.87:0/1259144465 >> 172.21.15.87:0/1259144465 conn(0x7fc9040fc420 msgr2=0x7fc9040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- 172.21.15.87:0/1259144465 shutdown_connections 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- 172.21.15.87:0/1259144465 wait complete. 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 Processor -- start 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.382+0000 7fc90997e640 1 -- start start 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904102af0 0x7fc9041981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 0x7fc904198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc90410db00 con 0x7fc904107d50 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc90410d980 con 0x7fc904102af0 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc90410dc80 con 0x7fc904105420 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60142/0 (socket says 172.21.15.87:60142) 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 -- 172.21.15.87:0/1835581407 learned_addr learned my addr 172.21.15.87:0/1835581407 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9027fc640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 0x7fc904198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc902ffd640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904102af0 0x7fc9041981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 -- 172.21.15.87:0/1835581407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 msgr2=0x7fc904198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 0x7fc904198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 -- 172.21.15.87:0/1835581407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904102af0 msgr2=0x7fc9041981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904102af0 0x7fc9041981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 -- 172.21.15.87:0/1835581407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90419fe00 con 0x7fc904107d50 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc9037fe640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400d950 tx=0x7fc8f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4017030 con 0x7fc904107d50 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9041a00f0 con 0x7fc904107d50 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8f4004020 con 0x7fc904107d50 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003b50 con 0x7fc904107d50 2024-04-22T13:31:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.383+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9041a0640 con 0x7fc904107d50 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.385+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9040fdf10 con 0x7fc904107d50 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.385+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fc8f401a070 con 0x7fc904107d50 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.386+0000 7fc90897c640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 0x7fc8ec0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.386+0000 7fc902ffd640 1 -- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 msgr2=0x7fc8ec0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.386+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fc8f40cfae0 con 0x7fc904107d50 2024-04-22T13:31:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.386+0000 7fc902ffd640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 0x7fc8ec0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:31:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.391+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8f4021070 con 0x7fc904107d50 2024-04-22T13:31:45.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.576+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fc904062b50 con 0x7fc904107d50 2024-04-22T13:31:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.582+0000 7fc90897c640 1 -- 172.21.15.87:0/1835581407 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7fc8f409c560 con 0x7fc904107d50 2024-04-22T13:31:45.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 msgr2=0x7fc8ec0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:31:45.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 0x7fc8ec0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 msgr2=0x7fc90419f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400d950 tx=0x7fc8f400de10 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 shutdown_connections 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fc8ec0a3ca0 0x7fc8ec0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc904107d50 0x7fc90419f790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc904105420 0x7fc904198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 --2- 172.21.15.87:0/1835581407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc904102af0 0x7fc9041981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 >> 172.21.15.87:0/1835581407 conn(0x7fc9040fc420 msgr2=0x7fc904103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 shutdown_connections 2024-04-22T13:31:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:45.586+0000 7fc90997e640 1 -- 172.21.15.87:0/1835581407 wait complete. 2024-04-22T13:31:45.596 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:45.318+0000 7f07b55cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:45.597 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:45.427+0000 7f07b55cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:31:45.598 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T13:31:45.623 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T13:31:45.623 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T13:31:45.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:45.623 DEBUG:teuthology.orchestra.run.smithi087:> ip netns list 2024-04-22T13:31:45.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:45.639 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link delete ceph-brx 2024-04-22T13:31:45.669 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:45.408+0000 7f6cecf9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:31:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:Cannot find device "ceph-brx" 2024-04-22T13:31:45.704 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:45.299+0000 7f9820e7a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:45.704 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:45.402+0000 7f9820e7a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T13:31:45.705 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:45.709+0000 7f9820e7a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:31:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:31:45.706 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-22T13:31:45.706 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T13:31:45.706 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T13:31:45.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T13:31:45.938 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:45.751+0000 7f07b55cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:31:45.938 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:45.840+0000 7f07b55cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:31:45.962 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:45.793+0000 7f9820e7a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:31:45.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T13:31:45.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-mon[31531]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T13:31:45.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2648570223' entity='client.admin' 2024-04-22T13:31:45.966 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1835581407' entity='client.admin' 2024-04-22T13:31:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:31:46.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T13:31:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-mon[31101]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T13:31:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2648570223' entity='client.admin' 2024-04-22T13:31:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1835581407' entity='client.admin' 2024-04-22T13:31:46.170 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:45.716+0000 7f6cecf9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T13:31:46.170 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:45.800+0000 7f6cecf9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T13:31:46.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3153476289' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T13:31:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-mon[27999]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T13:31:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2648570223' entity='client.admin' 2024-04-22T13:31:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1835581407' entity='client.admin' 2024-04-22T13:31:47.212 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:46.961+0000 7f9820e7a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:31:47.213 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:47.129+0000 7f9820e7a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:47.309 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:46 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:46.975+0000 7f6cecf9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:31:47.309 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:47.144+0000 7f6cecf9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:47.309 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:47.313+0000 7f6cecf9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:31:47.311 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.044+0000 7f07b55cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T13:31:47.311 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.216+0000 7f07b55cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:47.583 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:47.297+0000 7f9820e7a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:31:47.612 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.395+0000 7f07b55cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.862+0000 7fcf3a394700 1 -- 172.21.15.87:0/2320327320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 msgr2=0x7fcf34108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.862+0000 7fcf3a394700 1 --2- 172.21.15.87:0/2320327320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf34108eb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009990 tx=0x7fcf24009ca0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 -- 172.21.15.87:0/2320327320 shutdown_connections 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 --2- 172.21.15.87:0/2320327320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 0x7fcf3410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 --2- 172.21.15.87:0/2320327320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf34108eb0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 --2- 172.21.15.87:0/2320327320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf34101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 -- 172.21.15.87:0/2320327320 >> 172.21.15.87:0/2320327320 conn(0x7fcf34077fb0 msgr2=0x7fcf340ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:47.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 -- 172.21.15.87:0/2320327320 shutdown_connections 2024-04-22T13:31:47.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 -- 172.21.15.87:0/2320327320 wait complete. 2024-04-22T13:31:47.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 Processor -- start 2024-04-22T13:31:47.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.863+0000 7fcf3a394700 1 -- start start 2024-04-22T13:31:47.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf341990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:47.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 0x7fcf34199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:47.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf34112bc0 con 0x7fcf34108ad0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf34112a40 con 0x7fcf34101030 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf3a394700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf34112d40 con 0x7fcf341019d0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60170/0 (socket says 172.21.15.87:60170) 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 -- 172.21.15.87:0/1391705677 learned_addr learned my addr 172.21.15.87:0/1391705677 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf33fff700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf341990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf337fe700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 0x7fcf34199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 msgr2=0x7fcf34199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 0x7fcf34199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 msgr2=0x7fcf341990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf341990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 -- 172.21.15.87:0/1391705677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3419e0e0 con 0x7fcf34108ad0 2024-04-22T13:31:47.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf33fff700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf341990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.864+0000 7fcf38931700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c007ee0 tx=0x7fcf2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.865+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c004bb0 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.865+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3419e3d0 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.865+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf2c004d50 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.865+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c005920 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.865+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf341a5c50 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.866+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf34104350 con 0x7fcf34108ad0 2024-04-22T13:31:47.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.867+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fcf2c01a020 con 0x7fcf34108ad0 2024-04-22T13:31:47.865 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.691+0000 7f07b55cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:31:47.865 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.785+0000 7f07b55cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:31:47.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.868+0000 7fcf317fa700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 0x7fcf1c0a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:47.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.868+0000 7fcf33fff700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 msgr2=0x7fcf1c0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:47.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.868+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fcf2c0d0080 con 0x7fcf34108ad0 2024-04-22T13:31:47.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.868+0000 7fcf33fff700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 0x7fcf1c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:31:47.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:47.874+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf2c010070 con 0x7fcf34108ad0 2024-04-22T13:31:47.876 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:47.587+0000 7f9820e7a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:31:47.876 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:47.679+0000 7f9820e7a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:31:47.877 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:47.881+0000 7f9820e7a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:31:47.891 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:47.603+0000 7f6cecf9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T13:31:47.891 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:47.694+0000 7f6cecf9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T13:31:47.891 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:47.895+0000 7f6cecf9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:31:48.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.069+0000 7fcf33fff700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 msgr2=0x7fcf1c0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:48.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.069+0000 7fcf33fff700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 0x7fcf1c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:31:48.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.127+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcf3419a430 con 0x7fcf34108ad0 2024-04-22T13:31:48.124 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:47.996+0000 7f07b55cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T13:31:48.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.128+0000 7fcf317fa700 1 -- 172.21.15.87:0/1391705677 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7fcf2c09cb00 con 0x7fcf34108ad0 2024-04-22T13:31:48.125 INFO:teuthology.orchestra.run.smithi087.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 msgr2=0x7fcf1c0a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 0x7fcf1c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 msgr2=0x7fcf3419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c007ee0 tx=0x7fcf2c00d3b0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 shutdown_connections 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7fcf1c0a3c90 0x7fcf1c0a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf34108ad0 0x7fcf3419d9a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf34101030 0x7fcf341990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 --2- 172.21.15.87:0/1391705677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf341019d0 0x7fcf34199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 >> 172.21.15.87:0/1391705677 conn(0x7fcf34077fb0 msgr2=0x7fcf340ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 shutdown_connections 2024-04-22T13:31:48.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48.130+0000 7fcf3a394700 1 -- 172.21.15.87:0/1391705677 wait complete. 2024-04-22T13:31:48.169 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:48 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:48.087+0000 7f6cecf9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:48.176 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:48.074+0000 7f9820e7a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:48.452 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:48 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:48.203+0000 7f07b55cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T13:31:48.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:48 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1391705677' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T13:31:48.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:48 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1391705677' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T13:31:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:48 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1391705677' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T13:31:48.814 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T13:31:48.814 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi087.front.sepia.ceph.com 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T13:31:48.815 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T13:31:48.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:48.815 DEBUG:teuthology.orchestra.run.smithi087:> ip addr 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: valid_lft forever preferred_lft forever 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: inet6 ::1/128 scope host 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: valid_lft forever preferred_lft forever 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: link/ether 0c:c4:7a:d8:da:fc brd ff:ff:ff:ff:ff:ff 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout: altname enp7s0f0 2024-04-22T13:31:48.834 INFO:teuthology.orchestra.run.smithi087.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: link/ether 0c:c4:7a:d8:da:fd brd ff:ff:ff:ff:ff:ff 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: altname enp7s0f1 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: link/ether 0c:c4:7a:8f:d5:f4 brd ff:ff:ff:ff:ff:ff 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: link/ether 0c:c4:7a:8f:d5:f5 brd ff:ff:ff:ff:ff:ff 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: inet 172.21.15.87/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: valid_lft 41942sec preferred_lft 41942sec 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: inet6 fe80::6bca:fe34:2cb0:648d/64 scope link noprefixroute 2024-04-22T13:31:48.835 INFO:teuthology.orchestra.run.smithi087.stdout: valid_lft forever preferred_lft forever 2024-04-22T13:31:48.836 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> set -e 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link add name ceph-brx type bridge 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip addr flush dev ceph-brx 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link set ceph-brx up 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T13:31:48.836 DEBUG:teuthology.orchestra.run.smithi087:> ') 2024-04-22T13:31:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:48.963 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:48.671+0000 7f9820e7a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:31:49.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:49.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:49.020 DEBUG:teuthology.orchestra.run.smithi087:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T13:31:49.039 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:48 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:48.808+0000 7f07b55cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:31:49.045 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-04-22T13:31:49.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:49.051 DEBUG:teuthology.orchestra.run.smithi087:> route 2024-04-22T13:31:49.096 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:48 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:48.683+0000 7f6cecf9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T13:31:49.119 INFO:teuthology.orchestra.run.smithi087.stdout:Kernel IP routing table 2024-04-22T13:31:49.120 INFO:teuthology.orchestra.run.smithi087.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T13:31:49.120 INFO:teuthology.orchestra.run.smithi087.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T13:31:49.120 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T13:31:49.120 INFO:teuthology.orchestra.run.smithi087.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T13:31:49.120 DEBUG:teuthology.orchestra.run.smithi087:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T13:31:49.120 DEBUG:teuthology.orchestra.run.smithi087:> set -e 2024-04-22T13:31:49.120 DEBUG:teuthology.orchestra.run.smithi087:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T13:31:49.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T13:31:49.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T13:31:49.121 DEBUG:teuthology.orchestra.run.smithi087:> ') 2024-04-22T13:31:49.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:49.318 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:49.267+0000 7f07b55cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:31:49.390 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:49.096+0000 7f9820e7a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:31:49.390 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:49.192+0000 7f9820e7a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:31:49.390 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:49.394+0000 7f9820e7a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:31:49.393 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:49.101+0000 7f6cecf9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T13:31:49.393 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:49.198+0000 7f6cecf9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:31:49.393 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:49.397+0000 7f6cecf9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:31:49.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:49.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:49.403 DEBUG:teuthology.orchestra.run.smithi087:> ip netns list 2024-04-22T13:31:49.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:49.419 DEBUG:teuthology.orchestra.run.smithi087:> ip netns list-id 2024-04-22T13:31:49.473 DEBUG:teuthology.orchestra.run.smithi087:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T13:31:49.474 DEBUG:teuthology.orchestra.run.smithi087:> set -e 2024-04-22T13:31:49.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T13:31:49.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T13:31:49.474 DEBUG:teuthology.orchestra.run.smithi087:> ') 2024-04-22T13:31:49.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:49.569 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:49.369+0000 7f07b55cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T13:31:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:49.589 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> set -e 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T13:31:49.590 DEBUG:teuthology.orchestra.run.smithi087:> ') 2024-04-22T13:31:49.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:49.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:49.719 DEBUG:teuthology.orchestra.run.smithi087:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T13:31:49.719 DEBUG:teuthology.orchestra.run.smithi087:> set -e 2024-04-22T13:31:49.719 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link set brx.0 up 2024-04-22T13:31:49.719 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T13:31:49.719 DEBUG:teuthology.orchestra.run.smithi087:> ') 2024-04-22T13:31:49.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:49.798 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T13:31:49.798 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T13:31:49.813 INFO:teuthology.orchestra.run.smithi087.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T13:31:49.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T13:31:49.813 DEBUG:teuthology.orchestra.run.smithi087:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T13:31:49.863 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T13:31:49.863 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T13:31:49.863 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/8f72c7cd-96ba-4153-be89-fe989bd2e42e 2024-04-22T13:31:49.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:49.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/8f72c7cd-96ba-4153-be89-fe989bd2e42e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T13:31:49.923 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:49.586+0000 7f07b55cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T13:31:49.957 INFO:teuthology.orchestra.run.smithi087.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T13:31:49.957 INFO:teuthology.orchestra.run.smithi087.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T13:31:49.957 INFO:teuthology.orchestra.run.smithi087.stdout:invalid new device string format 2024-04-22T13:31:50.070 INFO:teuthology.orchestra.run.smithi087.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T13:31:50.070 INFO:teuthology.orchestra.run.smithi087.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T13:31:50.070 INFO:teuthology.orchestra.run.smithi087.stdout:invalid new device string format 2024-04-22T13:31:50.071 INFO:teuthology.orchestra.run.smithi087.stdout:mount.ceph: resolved to: "172.21.15.87:3300,172.21.15.120:3300,172.21.15.132:3300" 2024-04-22T13:31:50.071 INFO:teuthology.orchestra.run.smithi087.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.87:3300,172.21.15.120:3300,172.21.15.132:3300:/volumes/qa/sv_1/8f72c7cd-96ba-4153-be89-fe989bd2e42e 2024-04-22T13:31:50.071 INFO:teuthology.orchestra.run.smithi087.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=176003e8-00ab-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T13:31:50.072 INFO:teuthology.orchestra.run.smithi087.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T13:31:50.072 INFO:teuthology.orchestra.run.smithi087.stdout: You just mounted a file system that supports labels which does not 2024-04-22T13:31:50.072 INFO:teuthology.orchestra.run.smithi087.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T13:31:50.072 INFO:teuthology.orchestra.run.smithi087.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T13:31:50.072 INFO:teuthology.orchestra.run.smithi087.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T13:31:50.074 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T13:31:50.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:50.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T13:31:50.142 DEBUG:teuthology.orchestra.run.smithi087:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T13:31:50.169 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:49.913+0000 7f6cecf9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:31:50.170 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.005+0000 7f6cecf9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:31:50.212 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:49.922+0000 7f9820e7a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:31:50.212 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.015+0000 7f9820e7a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:31:50.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T13:31:50.232 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.138+0000 7f07b55cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T13:31:50.232 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.236+0000 7f07b55cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T13:31:50.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:50.240 INFO:teuthology.orchestra.run.smithi087.stdout:client24842 2024-04-22T13:31:50.240 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:31:50.240 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T13:31:50.258 DEBUG:teuthology.orchestra.run.smithi087:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T13:31:50.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T13:31:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:50.355 INFO:teuthology.orchestra.run.smithi087.stdout:client24842 2024-04-22T13:31:50.356 DEBUG:teuthology.orchestra.run.smithi087:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T13:31:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T13:31:50.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:50.412 INFO:teuthology.orchestra.run.smithi087.stdout:client24842 2024-04-22T13:31:50.412 DEBUG:teuthology.orchestra.run.smithi087:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/176003e8-00ab-11ef-bc93-c7b262605968.client24842/status') 2024-04-22T13:31:50.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T13:31:50.470 INFO:teuthology.orchestra.run.smithi087.stdout:instance: client.24842 (3)172.21.15.87:0/1757116619 2024-04-22T13:31:50.470 INFO:teuthology.orchestra.run.smithi087.stdout:blocklisted: false 2024-04-22T13:31:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:0+1 records in 2024-04-22T13:31:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:0+1 records out 2024-04-22T13:31:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:71 bytes copied, 2.2769e-05 s, 3.1 MB/s 2024-04-22T13:31:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T13:31:50.479 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T13:31:50.488 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T13:31:50.488 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:31:50.489 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T13:31:50.592 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.248+0000 7f6cecf9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:31:50.592 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.380+0000 7f6cecf9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:31:50.592 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.474+0000 7f6cecf9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:31:50.592 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.591+0000 7f6cecf9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 -- 172.21.15.87:0/3281179900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 msgr2=0x7f64d410a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 --2- 172.21.15.87:0/3281179900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d410a480 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009cd0 tx=0x7f64cc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 -- 172.21.15.87:0/3281179900 shutdown_connections 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 --2- 172.21.15.87:0/3281179900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d410a480 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 --2- 172.21.15.87:0/3281179900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d41036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 --2- 172.21.15.87:0/3281179900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 -- 172.21.15.87:0/3281179900 >> 172.21.15.87:0/3281179900 conn(0x7f64d40fbbb0 msgr2=0x7f64d40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 -- 172.21.15.87:0/3281179900 shutdown_connections 2024-04-22T13:31:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.595+0000 7f64da6d9640 1 -- 172.21.15.87:0/3281179900 wait complete. 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 Processor -- start 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 -- start start 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f64d410db00 con 0x7f64d4103bf0 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f64d410d980 con 0x7f64d4103230 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.596+0000 7f64da6d9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f64d410dc80 con 0x7f64d4100260 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57748/0 (socket says 172.21.15.87:57748) 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 -- 172.21.15.87:0/820619674 learned_addr learned my addr 172.21.15.87:0/820619674 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 msgr2=0x7f64d41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 msgr2=0x7f64d4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 -- 172.21.15.87:0/820619674 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d419fd00 con 0x7f64d4103bf0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d3fff640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d37fe640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d8c76640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ca0 tx=0x7f64cc0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc041020 con 0x7f64d4103bf0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d419ff90 con 0x7f64d4103bf0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc0327c0 con 0x7f64d4103bf0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d41a0450 con 0x7f64d4103bf0 2024-04-22T13:31:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.597+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc039040 con 0x7f64d4103bf0 2024-04-22T13:31:50.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.598+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d4109380 con 0x7f64d4103bf0 2024-04-22T13:31:50.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.600+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f64cc044020 con 0x7f64d4103bf0 2024-04-22T13:31:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.601+0000 7f64d17fa640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 0x7f64c00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.601+0000 7f64d3fff640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 msgr2=0x7f64c00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.601+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f64cc0f25a0 con 0x7f64d4103bf0 2024-04-22T13:31:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.601+0000 7f64d3fff640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 0x7f64c00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:31:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.604+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64cc04b020 con 0x7f64d4103bf0 2024-04-22T13:31:50.601 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.258+0000 7f9820e7a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:31:50.601 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.391+0000 7f9820e7a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:31:50.601 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.487+0000 7f9820e7a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:31:50.601 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.605+0000 7f9820e7a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:50.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.802+0000 7f64d3fff640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 msgr2=0x7f64c00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:50.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.802+0000 7f64d3fff640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 0x7f64c00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:31:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.831+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f64d402d770 con 0x7f64d4103bf0 2024-04-22T13:31:50.864 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.503+0000 7f07b55cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T13:31:50.864 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.647+0000 7f07b55cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T13:31:50.864 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.748+0000 7f07b55cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T13:31:50.864 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:50.868+0000 7f07b55cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T13:31:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.883+0000 7f64d17fa640 1 -- 172.21.15.87:0/820619674 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7f64cc0bf020 con 0x7f64d4103bf0 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 msgr2=0x7f64c00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 0x7f64c00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 msgr2=0x7f64d419f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ca0 tx=0x7f64cc0346d0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 shutdown_connections 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f64c00a3ca0 0x7f64c00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64d4103bf0 0x7f64d419f690 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64d4103230 0x7f64d4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 --2- 172.21.15.87:0/820619674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64d4100260 0x7f64d41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 >> 172.21.15.87:0/820619674 conn(0x7f64d40fbbb0 msgr2=0x7f64d4106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 shutdown_connections 2024-04-22T13:31:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:50.886+0000 7f64da6d9640 1 -- 172.21.15.87:0/820619674 wait complete. 2024-04-22T13:31:50.894 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 -- 172.21.15.87:0/2902276982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0105460 msgr2=0x7f56a0107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 --2- 172.21.15.87:0/2902276982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0105460 0x7f56a0107850 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5690009900 tx=0x7f569002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 -- 172.21.15.87:0/2902276982 shutdown_connections 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 --2- 172.21.15.87:0/2902276982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a0107d90 0x7f56a010a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 --2- 172.21.15.87:0/2902276982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0105460 0x7f56a0107850 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 --2- 172.21.15.87:0/2902276982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56a0102b30 0x7f56a0104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:50.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 -- 172.21.15.87:0/2902276982 >> 172.21.15.87:0/2902276982 conn(0x7f56a00fc460 msgr2=0x7f56a00fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:50.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 -- 172.21.15.87:0/2902276982 shutdown_connections 2024-04-22T13:31:50.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.001+0000 7f56a69ef640 1 -- 172.21.15.87:0/2902276982 wait complete. 2024-04-22T13:31:50.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 Processor -- start 2024-04-22T13:31:50.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 -- start start 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 0x7f56a01981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56a0105460 0x7f56a0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a0107d90 0x7f56a019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f56a010db40 con 0x7f56a0102b30 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f56a010d9c0 con 0x7f56a0107d90 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a69ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f56a010dcc0 con 0x7f56a0105460 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f569ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 0x7f56a01981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f569ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 0x7f56a01981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57774/0 (socket says 172.21.15.87:57774) 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f569ffff640 1 -- 172.21.15.87:0/3327969844 learned_addr learned my addr 172.21.15.87:0/3327969844 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56a0105460 0x7f56a0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:51.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.002+0000 7f56a4f8c640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a0107d90 0x7f56a019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56a0105460 msgr2=0x7f56a0198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56a0105460 0x7f56a0198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a0107d90 msgr2=0x7f56a019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56a0107d90 0x7f56a019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a019fde0 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569ffff640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 0x7f56a01981c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f569400e9f0 tx=0x7f569400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569400cd80 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a01a00d0 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56940054d0 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694010690 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.003+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a01a05f0 con 0x7f56a0102b30 2024-04-22T13:31:51.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.004+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a00fdf50 con 0x7f56a0102b30 2024-04-22T13:31:51.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.005+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f5694004e90 con 0x7f56a0102b30 2024-04-22T13:31:51.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.006+0000 7f569d7fa640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 0x7f56880a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:51.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.006+0000 7f569f7fe640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 msgr2=0x7f56880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:51.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.006+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f56940d07b0 con 0x7f56a0102b30 2024-04-22T13:31:51.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.006+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 0x7f56880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T13:31:51.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.010+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f569409d230 con 0x7f56a0102b30 2024-04-22T13:31:51.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T13:31:51.170 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:50.891+0000 7f6cecf9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:31:51.175 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:50 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T13:31:51.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.207+0000 7f569f7fe640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 msgr2=0x7f56880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:51.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.207+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 0x7f56880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T13:31:51.271 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T13:31:51.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.217+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f56a0062430 con 0x7f56880a3ca0 2024-04-22T13:31:51.271 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:50.910+0000 7f9820e7a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:31:51.489 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:51.180+0000 7f07b55cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T13:31:51.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.608+0000 7f569f7fe640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 msgr2=0x7f56880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1421536657 2024-04-22T13:31:51.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.608+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 0x7f56880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T13:31:51.887 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:51.515+0000 7f9820e7a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:31:51.887 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z[33382]: 2024-04-22T13:31:51.731+0000 7f9820e7a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:31:51.889 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:51.492+0000 7f6cecf9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:31:51.889 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y[32488]: 2024-04-22T13:31:51.707+0000 7f6cecf9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:31:51.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:51.905+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f56940991a0 con 0x7f56a0102b30 2024-04-22T13:31:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:52.025+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f56940a36d0 con 0x7f56a0102b30 2024-04-22T13:31:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:52.026+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 msgr2=0x7f56880a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:31:52.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:52.026+0000 7f569d7fa640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/1421536657,v1:172.21.15.87:6801/1421536657] conn(0x7f56880a3ca0 0x7f56880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:52.146 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: Standby manager daemon y restarted 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: Standby manager daemon y started 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: Standby manager daemon z restarted 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: Standby manager daemon z started 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:31:52.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:51 smithi120 ceph-mon[31101]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: Standby manager daemon y restarted 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: Standby manager daemon y started 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: Standby manager daemon z restarted 2024-04-22T13:31:52.148 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: Standby manager daemon z started 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:51 smithi132 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:51.786+0000 7f07b55cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.014+0000 7f07b55cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T13:31:52.149 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/820619674' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: Standby manager daemon y restarted 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: Standby manager daemon y started 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.120:0/369222144' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: Standby manager daemon z restarted 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: Standby manager daemon z started 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T13:31:52.150 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:51 smithi087 ceph-mon[27999]: from='mgr.? 172.21.15.132:0/1991340917' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0717f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0717f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0717f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0717f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0717f45700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0714f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0714f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.487 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0714f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0714f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:52.488 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:52.257+0000 7f0714f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.032+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mon.0 v2:172.21.15.87:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f569405f330 con 0x7f56a0102b30 2024-04-22T13:31:53.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.033+0000 7f569d7fa640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5688035660 0x7f5688035a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.033+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f5694020ec0 con 0x7f5688035660 2024-04-22T13:31:53.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.037+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5688035660 0x7f5688035a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.038+0000 7f569f7fe640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5688035660 0x7f5688035a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5690005ab0 tx=0x7f5690005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.057+0000 7f569d7fa640 1 -- 172.21.15.87:0/3327969844 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f5694020ec0 con 0x7f5688035660 2024-04-22T13:31:53.053 INFO:teuthology.orchestra.run.smithi087.stdout:{} 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5688035660 msgr2=0x7f5688035a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5688035660 0x7f5688035a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5690005ab0 tx=0x7f5690005fd0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 msgr2=0x7f56a01981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 --2- 172.21.15.87:0/3327969844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56a0102b30 0x7f56a01981c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f569400e9f0 tx=0x7f569400edf0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a4f8c640 1 -- 172.21.15.87:0/3327969844 reap_dead start 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 shutdown_connections 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 >> 172.21.15.87:0/3327969844 conn(0x7f56a00fc460 msgr2=0x7f56a0103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 shutdown_connections 2024-04-22T13:31:53.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.059+0000 7f56a69ef640 1 -- 172.21.15.87:0/3327969844 wait complete. 2024-04-22T13:31:53.066 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T13:31:53.078 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T13:31:53.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: Active manager daemon x restarted 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: Activating manager daemon x 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: mgrmap e21: x(active, starting, since 0.0119163s), standbys: y, z 2024-04-22T13:31:53.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:31:53.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:31:53.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: Manager daemon x is now available 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:31:53.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:52 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:31:53.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 -- 172.21.15.87:0/2228262555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 msgr2=0x7fa84810a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 --2- 172.21.15.87:0/2228262555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84810a3d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa840009ca0 tx=0x7fa84002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 -- 172.21.15.87:0/2228262555 shutdown_connections 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 --2- 172.21.15.87:0/2228262555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84810a3d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 --2- 172.21.15.87:0/2228262555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848103180 0x7fa848103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 --2- 172.21.15.87:0/2228262555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848101f90 0x7fa8481023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 -- 172.21.15.87:0/2228262555 >> 172.21.15.87:0/2228262555 conn(0x7fa8480fd500 msgr2=0x7fa8480ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 -- 172.21.15.87:0/2228262555 shutdown_connections 2024-04-22T13:31:53.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.194+0000 7fa84da43640 1 -- 172.21.15.87:0/2228262555 wait complete. 2024-04-22T13:31:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 Processor -- start 2024-04-22T13:31:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 -- start start 2024-04-22T13:31:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848101f90 0x7fa848198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848103180 0x7fa848198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa84810da90 con 0x7fa848103b40 2024-04-22T13:31:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa84810d910 con 0x7fa848103180 2024-04-22T13:31:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa84da43640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa84810dc10 con 0x7fa848101f90 2024-04-22T13:31:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57936/0 (socket says 172.21.15.87:57936) 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 -- 172.21.15.87:0/1157680485 learned_addr learned my addr 172.21.15.87:0/1157680485 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa846ffd640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848101f90 0x7fa848198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa83ffff640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848103180 0x7fa848198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 -- 172.21.15.87:0/1157680485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848101f90 msgr2=0x7fa848198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848101f90 0x7fa848198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 -- 172.21.15.87:0/1157680485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848103180 msgr2=0x7fa848198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848103180 0x7fa848198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.195+0000 7fa8477fe640 1 -- 172.21.15.87:0/1157680485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84819fd50 con 0x7fa848103b40 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa8477fe640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa84002c700 tx=0x7fa8400093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840041020 con 0x7fa848103b40 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa840034d60 con 0x7fa848103b40 2024-04-22T13:31:53.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840039040 con 0x7fa848103b40 2024-04-22T13:31:53.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa84819ffe0 con 0x7fa848103b40 2024-04-22T13:31:53.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.196+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8481a0440 con 0x7fa848103b40 2024-04-22T13:31:53.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.197+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8481092b0 con 0x7fa848103b40 2024-04-22T13:31:53.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.198+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fa840044020 con 0x7fa848103b40 2024-04-22T13:31:53.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.198+0000 7fa84ca41640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 0x7fa8340a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.198+0000 7fa846ffd640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 0x7fa8340a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.198+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa8400f2010 con 0x7fa848103b40 2024-04-22T13:31:53.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.199+0000 7fa846ffd640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 0x7fa8340a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa838005fd0 tx=0x7fa83800b410 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.203+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa84004b020 con 0x7fa848103b40 2024-04-22T13:31:53.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T13:31:53.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: Active manager daemon x restarted 2024-04-22T13:31:53.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: Activating manager daemon x 2024-04-22T13:31:53.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T13:31:53.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: mgrmap e21: x(active, starting, since 0.0119163s), standbys: y, z 2024-04-22T13:31:53.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:31:53.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:31:53.216 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:31:53.216 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:31:53.216 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:31:53.216 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:31:53.216 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:31:53.217 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:31:53.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:31:53.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:31:53.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:31:53.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:31:53.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:31:53.219 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:31:53.219 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:31:53.219 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:31:53.221 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:31:53.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: Manager daemon x is now available 2024-04-22T13:31:53.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:53.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:31:53.222 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:52 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:31:53.236 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:53.044+0000 7f0711ef9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:53.044+0000 7f0711ef9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:53.044+0000 7f0711ef9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:53.044+0000 7f0711ef9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.237 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:31:53.044+0000 7f0711ef9700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:31:53.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: Active manager daemon x restarted 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: Activating manager daemon x 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: mgrmap e21: x(active, starting, since 0.0119163s), standbys: y, z 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T13:31:53.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T13:31:53.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T13:31:53.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: Manager daemon x is now available 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T13:31:53.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:52 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T13:31:53.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.390+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa848062bf0 con 0x7fa848103b40 2024-04-22T13:31:53.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.392+0000 7fa84ca41640 1 -- 172.21.15.87:0/1157680485 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7fa8400bec30 con 0x7fa848103b40 2024-04-22T13:31:53.388 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:53.388 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:53.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 msgr2=0x7fa8340a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 0x7fa8340a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa838005fd0 tx=0x7fa83800b410 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 msgr2=0x7fa84819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa84002c700 tx=0x7fa8400093f0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 shutdown_connections 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8340a39b0 0x7fa8340a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa848103b40 0x7fa84819f6e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa848103180 0x7fa848198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 --2- 172.21.15.87:0/1157680485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa848101f90 0x7fa848198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 >> 172.21.15.87:0/1157680485 conn(0x7fa8480fd500 msgr2=0x7fa848106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 shutdown_connections 2024-04-22T13:31:53.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.394+0000 7fa84da43640 1 -- 172.21.15.87:0/1157680485 wait complete. 2024-04-22T13:31:53.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:53.401 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 -- 172.21.15.87:0/216746968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc781029a0 msgr2=0x7fcc781092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/216746968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc781029a0 0x7fcc781092c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009ca0 tx=0x7fcc7002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 -- 172.21.15.87:0/216746968 shutdown_connections 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/216746968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc781029a0 0x7fcc781092c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/216746968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 0x7fcc78102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/216746968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc78100e90 0x7fcc781012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 -- 172.21.15.87:0/216746968 >> 172.21.15.87:0/216746968 conn(0x7fcc780fc400 msgr2=0x7fcc780fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 -- 172.21.15.87:0/216746968 shutdown_connections 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.510+0000 7fcc7e28e640 1 -- 172.21.15.87:0/216746968 wait complete. 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 Processor -- start 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 -- start start 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 0x7fcc7819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 0x7fcc7819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7810c9e0 con 0x7fcc78100e90 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7810c860 con 0x7fcc781029a0 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7e28e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7810cb60 con 0x7fcc78101fe0 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57958/0 (socket says 172.21.15.87:57958) 2024-04-22T13:31:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc77fff640 1 -- 172.21.15.87:0/2317620013 learned_addr learned my addr 172.21.15.87:0/2317620013 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc777fe640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 0x7fcc7819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.511+0000 7fcc7c82b640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 0x7fcc7819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 -- 172.21.15.87:0/2317620013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 msgr2=0x7fcc7819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 0x7fcc7819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 -- 172.21.15.87:0/2317620013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 msgr2=0x7fcc7819f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 0x7fcc7819f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 -- 172.21.15.87:0/2317620013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7819fdb0 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc7c82b640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 0x7fcc7819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc77fff640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c920 tx=0x7fcc6800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68007c70 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc68007e10 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7819ffe0 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68018820 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.512+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc781a04a0 con 0x7fcc78100e90 2024-04-22T13:31:53.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.513+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc781081a0 con 0x7fcc78100e90 2024-04-22T13:31:53.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.514+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fcc6800f450 con 0x7fcc78100e90 2024-04-22T13:31:53.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.514+0000 7fcc757fa640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 0x7fcc600a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.515+0000 7fcc777fe640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 0x7fcc600a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.515+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcc680d0470 con 0x7fcc78100e90 2024-04-22T13:31:53.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.515+0000 7fcc777fe640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 0x7fcc600a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc780699c0 tx=0x7fcc6c00b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.518+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc68014030 con 0x7fcc78100e90 2024-04-22T13:31:53.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.701+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc7802d850 con 0x7fcc78100e90 2024-04-22T13:31:53.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.702+0000 7fcc757fa640 1 -- 172.21.15.87:0/2317620013 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fcc6809d120 con 0x7fcc78100e90 2024-04-22T13:31:53.697 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:53.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.703+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 msgr2=0x7fcc600a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.703+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 0x7fcc600a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc780699c0 tx=0x7fcc6c00b390 comp rx=0 tx=0).stop 2024-04-22T13:31:53.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.703+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 msgr2=0x7fcc78069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.703+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c920 tx=0x7fcc6800cde0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 shutdown_connections 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcc600a39b0 0x7fcc600a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc781029a0 0x7fcc7819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc78101fe0 0x7fcc7819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 --2- 172.21.15.87:0/2317620013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc78100e90 0x7fcc78069c50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 >> 172.21.15.87:0/2317620013 conn(0x7fcc780fc400 msgr2=0x7fcc78105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 shutdown_connections 2024-04-22T13:31:53.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.704+0000 7fcc7e28e640 1 -- 172.21.15.87:0/2317620013 wait complete. 2024-04-22T13:31:53.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 -- 172.21.15.87:0/3300316494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c101fa0 msgr2=0x7f401c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 --2- 172.21.15.87:0/3300316494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c101fa0 0x7f401c1023c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f400c009900 tx=0x7f400c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 -- 172.21.15.87:0/3300316494 shutdown_connections 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 --2- 172.21.15.87:0/3300316494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c103b50 0x7f401c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 --2- 172.21.15.87:0/3300316494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 0x7f401c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 --2- 172.21.15.87:0/3300316494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c101fa0 0x7f401c1023c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 -- 172.21.15.87:0/3300316494 >> 172.21.15.87:0/3300316494 conn(0x7f401c0fd530 msgr2=0x7f401c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 -- 172.21.15.87:0/3300316494 shutdown_connections 2024-04-22T13:31:53.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.826+0000 7f4022074640 1 -- 172.21.15.87:0/3300316494 wait complete. 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 Processor -- start 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 -- start start 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c101fa0 0x7f401c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 0x7f401c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10daa0 con 0x7f401c103b50 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10d920 con 0x7f401c103190 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f4022074640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10dc20 con 0x7f401c101fa0 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f401bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f401bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57980/0 (socket says 172.21.15.87:57980) 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f401bfff640 1 -- 172.21.15.87:0/988642656 learned_addr learned my addr 172.21.15.87:0/988642656 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f401affd640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 0x7f401c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.827+0000 7f401b7fe640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c101fa0 0x7f401c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 -- 172.21.15.87:0/988642656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c101fa0 msgr2=0x7f401c075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c101fa0 0x7f401c075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 -- 172.21.15.87:0/988642656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 msgr2=0x7f401c074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 0x7f401c074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 -- 172.21.15.87:0/988642656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c075110 con 0x7f401c103b50 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f401bfff640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4014009fd0 tx=0x7f401400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014017070 con 0x7f401c103b50 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c1a2360 con 0x7f401c103b50 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4014003e20 con 0x7f401c103b50 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40140106c0 con 0x7f401c103b50 2024-04-22T13:31:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.828+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c1a28b0 con 0x7f401c103b50 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.829+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c1094d0 con 0x7f401c103b50 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.831+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f4014004140 con 0x7f401c103b50 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.831+0000 7f4018ff9640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 0x7f40040a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.832+0000 7f401b7fe640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 0x7f40040a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.832+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f40140cfcd0 con 0x7f401c103b50 2024-04-22T13:31:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.832+0000 7f401b7fe640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 0x7f40040a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f400c005ab0 tx=0x7f400c005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:53.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:53.835+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40140047e0 con 0x7f401c103b50 2024-04-22T13:31:54.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.070+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f401c02d5c0 con 0x7f401c103b50 2024-04-22T13:31:54.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.071+0000 7f4018ff9640 1 -- 172.21.15.87:0/988642656 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f401409c2a0 con 0x7f401c103b50 2024-04-22T13:31:54.067 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 msgr2=0x7f40040a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 0x7f40040a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f400c005ab0 tx=0x7f400c005fd0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 msgr2=0x7f401c074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4014009fd0 tx=0x7f401400b260 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 shutdown_connections 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f40040a39b0 0x7f40040a5e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c103b50 0x7f401c074880 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f401c103190 0x7f401c074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 --2- 172.21.15.87:0/988642656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f401c101fa0 0x7f401c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 >> 172.21.15.87:0/988642656 conn(0x7f401c0fd530 msgr2=0x7f401c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 shutdown_connections 2024-04-22T13:31:54.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.073+0000 7f4022074640 1 -- 172.21.15.87:0/988642656 wait complete. 2024-04-22T13:31:54.082 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:54.158 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: [22/Apr/2024:13:31:52] ENGINE Bus STARTING 2024-04-22T13:31:54.158 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: [22/Apr/2024:13:31:52] ENGINE Serving on https://172.21.15.87:7150 2024-04-22T13:31:54.161 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: [22/Apr/2024:13:31:52] ENGINE Serving on http://172.21.15.87:8765 2024-04-22T13:31:54.161 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: [22/Apr/2024:13:31:52] ENGINE Bus STARTED 2024-04-22T13:31:54.162 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: mgrmap e22: x(active, since 1.02021s), standbys: y, z 2024-04-22T13:31:54.162 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1157680485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:54.162 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:53 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2317620013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:54.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 -- 172.21.15.87:0/1948302576 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c102520 msgr2=0x7f573c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 --2- 172.21.15.87:0/1948302576 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c102520 0x7f573c102940 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5738009900 tx=0x7f573802bc80 comp rx=0 tx=0).stop 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 -- 172.21.15.87:0/1948302576 shutdown_connections 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 --2- 172.21.15.87:0/1948302576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 0x7f573c109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 --2- 172.21.15.87:0/1948302576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c1010e0 0x7f573c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 --2- 172.21.15.87:0/1948302576 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c102520 0x7f573c102940 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 -- 172.21.15.87:0/1948302576 >> 172.21.15.87:0/1948302576 conn(0x7f573c0fbb90 msgr2=0x7f573c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 -- 172.21.15.87:0/1948302576 shutdown_connections 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.212+0000 7f5743c11640 1 -- 172.21.15.87:0/1948302576 wait complete. 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 Processor -- start 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 -- start start 2024-04-22T13:31:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 0x7f573c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c102520 0x7f573c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f573c10c9d0 con 0x7f573c1010e0 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f573c10c850 con 0x7f573c102520 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f5743c11640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f573c10cb50 con 0x7f573c101aa0 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58018/0 (socket says 172.21.15.87:58018) 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 -- 172.21.15.87:0/248764488 learned_addr learned my addr 172.21.15.87:0/248764488 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57421ae640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c102520 0x7f573c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57411ac640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 0x7f573c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 -- 172.21.15.87:0/248764488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 msgr2=0x7f573c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 0x7f573c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 -- 172.21.15.87:0/248764488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c102520 msgr2=0x7f573c19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.213+0000 7f57419ad640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c102520 0x7f573c19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f57419ad640 1 -- 172.21.15.87:0/248764488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573c19fe40 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f57419ad640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5738006010 tx=0x7f5738004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738043020 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573c1a00d0 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5738004440 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738003bf0 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.214+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f573c1a0620 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.215+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f573c108340 con 0x7f573c1010e0 2024-04-22T13:31:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.216+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f5738046020 con 0x7f573c1010e0 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: [22/Apr/2024:13:31:52] ENGINE Bus STARTING 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: [22/Apr/2024:13:31:52] ENGINE Serving on https://172.21.15.87:7150 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: [22/Apr/2024:13:31:52] ENGINE Serving on http://172.21.15.87:8765 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: [22/Apr/2024:13:31:52] ENGINE Bus STARTED 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: mgrmap e22: x(active, since 1.02021s), standbys: y, z 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1157680485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:54.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:53 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2317620013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:54.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.217+0000 7f5732ffd640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 0x7f57280a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.217+0000 7f57411ac640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 0x7f57280a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.217+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f57380f2250 con 0x7f573c1010e0 2024-04-22T13:31:54.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.217+0000 7f57411ac640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 0x7f57280a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f573c1008d0 tx=0x7f572c005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.221+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f573804d020 con 0x7f573c1010e0 2024-04-22T13:31:54.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: [22/Apr/2024:13:31:52] ENGINE Bus STARTING 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: [22/Apr/2024:13:31:52] ENGINE Serving on https://172.21.15.87:7150 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: [22/Apr/2024:13:31:52] ENGINE Serving on http://172.21.15.87:8765 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: [22/Apr/2024:13:31:52] ENGINE Bus STARTED 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: mgrmap e22: x(active, since 1.02021s), standbys: y, z 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1157680485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:53 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2317620013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:54.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.410+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f573c062b50 con 0x7f573c1010e0 2024-04-22T13:31:54.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.410+0000 7f5732ffd640 1 -- 172.21.15.87:0/248764488 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f57380bee70 con 0x7f573c1010e0 2024-04-22T13:31:54.406 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 msgr2=0x7f57280a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 0x7f57280a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f573c1008d0 tx=0x7f572c005f90 comp rx=0 tx=0).stop 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 msgr2=0x7f573c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5738006010 tx=0x7f5738004040 comp rx=0 tx=0).stop 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 shutdown_connections 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57280a39b0 0x7f57280a5e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f573c102520 0x7f573c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f573c101aa0 0x7f573c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 --2- 172.21.15.87:0/248764488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f573c1010e0 0x7f573c100b60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 >> 172.21.15.87:0/248764488 conn(0x7f573c0fbb90 msgr2=0x7f573c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 shutdown_connections 2024-04-22T13:31:54.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.412+0000 7f5743c11640 1 -- 172.21.15.87:0/248764488 wait complete. 2024-04-22T13:31:54.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- 172.21.15.87:0/117397601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14101fc0 msgr2=0x7fbe14102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/117397601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14101fc0 0x7fbe14102440 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009900 tx=0x7fbe0802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- 172.21.15.87:0/117397601 shutdown_connections 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/117397601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 0x7fbe141092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/117397601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14101fc0 0x7fbe14102440 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/117397601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14100e70 0x7fbe14101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- 172.21.15.87:0/117397601 >> 172.21.15.87:0/117397601 conn(0x7fbe140fc3e0 msgr2=0x7fbe140fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- 172.21.15.87:0/117397601 shutdown_connections 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- 172.21.15.87:0/117397601 wait complete. 2024-04-22T13:31:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 Processor -- start 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.525+0000 7fbe1a5db640 1 -- start start 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 0x7fbe14197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 0x7fbe1419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410c9d0 con 0x7fbe14100e70 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410c850 con 0x7fbe14101fc0 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe1a5db640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410cb50 con 0x7fbe14102980 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58046/0 (socket says 172.21.15.87:58046) 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 -- 172.21.15.87:0/2948331061 learned_addr learned my addr 172.21.15.87:0/2948331061 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe18b78640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 0x7fbe1419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe137fe640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 0x7fbe14197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 -- 172.21.15.87:0/2948331061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 msgr2=0x7fbe1419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 0x7fbe1419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 -- 172.21.15.87:0/2948331061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 msgr2=0x7fbe14197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 0x7fbe14197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 -- 172.21.15.87:0/2948331061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1419ed10 con 0x7fbe14100e70 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe137fe640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 0x7fbe14197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.526+0000 7fbe13fff640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9f0 tx=0x7fbe0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.527+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400cd80 con 0x7fbe14100e70 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.527+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe040108e0 con 0x7fbe14100e70 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.527+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1419efa0 con 0x7fbe14100e70 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.527+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04018980 con 0x7fbe14100e70 2024-04-22T13:31:54.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.527+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1419f460 con 0x7fbe14100e70 2024-04-22T13:31:54.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.528+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe141082a0 con 0x7fbe14100e70 2024-04-22T13:31:54.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.529+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fbe04010430 con 0x7fbe14100e70 2024-04-22T13:31:54.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.529+0000 7fbe117fa640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 0x7fbdfc0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.530+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbe040d08e0 con 0x7fbe14100e70 2024-04-22T13:31:54.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.530+0000 7fbe137fe640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 0x7fbdfc0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.530+0000 7fbe137fe640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 0x7fbdfc0a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe08006010 tx=0x7fbe0802c7e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.534+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe0409d500 con 0x7fbe14100e70 2024-04-22T13:31:54.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.720+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe14062b80 con 0x7fbe14100e70 2024-04-22T13:31:54.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.721+0000 7fbe117fa640 1 -- 172.21.15.87:0/2948331061 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7fbe0409ce00 con 0x7fbe14100e70 2024-04-22T13:31:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:54.717 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 msgr2=0x7fbdfc0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 0x7fbdfc0a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe08006010 tx=0x7fbe0802c7e0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 msgr2=0x7fbe141970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9f0 tx=0x7fbe0400eeb0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 shutdown_connections 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbdfc0a39b0 0x7fbdfc0a5e70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe14102980 0x7fbe1419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe14101fc0 0x7fbe14197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 --2- 172.21.15.87:0/2948331061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbe14100e70 0x7fbe141970f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 >> 172.21.15.87:0/2948331061 conn(0x7fbe140fc3e0 msgr2=0x7fbe14105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 shutdown_connections 2024-04-22T13:31:54.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.723+0000 7fbe1a5db640 1 -- 172.21.15.87:0/2948331061 wait complete. 2024-04-22T13:31:54.731 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:54.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3154626998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 msgr2=0x7fa8b81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3154626998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b81012b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009900 tx=0x7fa8a802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3154626998 shutdown_connections 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3154626998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3154626998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3154626998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b81012b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3154626998 >> 172.21.15.87:0/3154626998 conn(0x7fa8b80fc400 msgr2=0x7fa8b80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:54.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3154626998 shutdown_connections 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3154626998 wait complete. 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 Processor -- start 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.840+0000 7fa8bcce0640 1 -- start start 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810c9c0 con 0x7fa8b8100e90 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810c840 con 0x7fa8b8101fe0 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8bcce0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810cb40 con 0x7fa8b81029a0 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58068/0 (socket says 172.21.15.87:58068) 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 -- 172.21.15.87:0/3422447390 learned_addr learned my addr 172.21.15.87:0/3422447390 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b5d9b640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b6d9d640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 -- 172.21.15.87:0/3422447390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 msgr2=0x7fa8b819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 -- 172.21.15.87:0/3422447390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 msgr2=0x7fa8b8197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 -- 172.21.15.87:0/3422447390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b819ed20 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b5d9b640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b6d9d640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8b659c640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8006010 tx=0x7fa8a8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8043020 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.841+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a8004440 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.842+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b819efb0 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.842+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8003bf0 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.842+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b819f470 con 0x7fa8b8100e90 2024-04-22T13:31:54.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.842+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b8108310 con 0x7fa8b8100e90 2024-04-22T13:31:54.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.844+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fa8a8046020 con 0x7fa8b8100e90 2024-04-22T13:31:54.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.845+0000 7fa8a77fe640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 0x7fa8a00a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:54.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.845+0000 7fa8b5d9b640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 0x7fa8a00a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:54.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.845+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa8a80f23c0 con 0x7fa8b8100e90 2024-04-22T13:31:54.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.845+0000 7fa8b5d9b640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 0x7fa8a00a5ef0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8198610 tx=0x7fa8ac00b410 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:54.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.849+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8a804d020 con 0x7fa8b8100e90 2024-04-22T13:31:54.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:54.918+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fa8a80baf50 con 0x7fa8b8100e90 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/988642656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/248764488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:54.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:54 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2948331061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:55.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.044+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8b8062b50 con 0x7fa8b8100e90 2024-04-22T13:31:55.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.044+0000 7fa8a77fe640 1 -- 172.21.15.87:0/3422447390 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fa8a80befe0 con 0x7fa8b8100e90 2024-04-22T13:31:55.041 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 msgr2=0x7fa8a00a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 0x7fa8a00a5ef0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8198610 tx=0x7fa8ac00b410 comp rx=0 tx=0).stop 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 msgr2=0x7fa8b8197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8006010 tx=0x7fa8a8004040 comp rx=0 tx=0).stop 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 shutdown_connections 2024-04-22T13:31:55.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa8a00a3a30 0x7fa8a00a5ef0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8b81029a0 0x7fa8b819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8b8101fe0 0x7fa8b8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 --2- 172.21.15.87:0/3422447390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8b8100e90 0x7fa8b8197100 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 >> 172.21.15.87:0/3422447390 conn(0x7fa8b80fc400 msgr2=0x7fa8b81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 shutdown_connections 2024-04-22T13:31:55.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.047+0000 7fa8bcce0640 1 -- 172.21.15.87:0/3422447390 wait complete. 2024-04-22T13:31:55.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/1519452110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac105340 msgr2=0x7f6cac107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/1519452110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac105340 0x7f6cac107730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009900 tx=0x7f6c9802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/1519452110 shutdown_connections 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/1519452110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 0x7f6cac10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/1519452110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac105340 0x7f6cac107730 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/1519452110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac102a10 0x7f6cac104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/1519452110 >> 172.21.15.87:0/1519452110 conn(0x7f6cac0fc400 msgr2=0x7f6cac0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/1519452110 shutdown_connections 2024-04-22T13:31:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.167+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/1519452110 wait complete. 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 Processor -- start 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 -- start start 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac105340 0x7f6cac198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 0x7f6cac19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10dac0 con 0x7f6cac102a10 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10d940 con 0x7f6cac107c70 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb2bc7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10dc40 con 0x7f6cac105340 2024-04-22T13:31:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58090/0 (socket says 172.21.15.87:58090) 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 -- 172.21.15.87:0/3063363420 learned_addr learned my addr 172.21.15.87:0/3063363420 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb1164640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 0x7f6cac19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cabfff640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac105340 0x7f6cac198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 -- 172.21.15.87:0/3063363420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac105340 msgr2=0x7f6cac198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac105340 0x7f6cac198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 -- 172.21.15.87:0/3063363420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 msgr2=0x7f6cac19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 0x7f6cac19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 -- 172.21.15.87:0/3063363420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac19fe20 con 0x7f6cac102a10 2024-04-22T13:31:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.168+0000 7f6cb0963640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000e9f0 tx=0x7f6ca000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.169+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000cd80 con 0x7f6cac102a10 2024-04-22T13:31:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.169+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ca00054d0 con 0x7f6cac102a10 2024-04-22T13:31:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.169+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0010690 con 0x7f6cac102a10 2024-04-22T13:31:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.169+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac1a0110 con 0x7f6cac102a10 2024-04-22T13:31:55.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.169+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac1a0630 con 0x7f6cac102a10 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/988642656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/248764488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:54 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2948331061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.170+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac0fdef0 con 0x7f6cac102a10 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.171+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f6ca0004e90 con 0x7f6cac102a10 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.172+0000 7f6ca9ffb640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 0x7f6c9c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.172+0000 7f6cabfff640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 0x7f6c9c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.172+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6ca00d0cd0 con 0x7f6cac102a10 2024-04-22T13:31:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.172+0000 7f6cabfff640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 0x7f6c9c0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005ab0 tx=0x7f6c98005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.176+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ca009d270 con 0x7f6cac102a10 2024-04-22T13:31:55.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:31:55.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/988642656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/248764488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:54 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2948331061' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:55.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.369+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6cac062b50 con 0x7f6cac102a10 2024-04-22T13:31:55.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.370+0000 7f6ca9ffb640 1 -- 172.21.15.87:0/3063363420 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f6ca009cb70 con 0x7f6cac102a10 2024-04-22T13:31:55.366 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 msgr2=0x7f6c9c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 0x7f6c9c0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005ab0 tx=0x7f6c98005fd0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 msgr2=0x7f6cac198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000e9f0 tx=0x7f6ca000edf0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 shutdown_connections 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c9c0a3e30 0x7f6c9c0a62f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cac107c70 0x7f6cac19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cac105340 0x7f6cac198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 --2- 172.21.15.87:0/3063363420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6cac102a10 0x7f6cac198200 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 >> 172.21.15.87:0/3063363420 conn(0x7f6cac0fc400 msgr2=0x7f6cac1048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 shutdown_connections 2024-04-22T13:31:55.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.372+0000 7f6cb2bc7640 1 -- 172.21.15.87:0/3063363420 wait complete. 2024-04-22T13:31:55.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 -- 172.21.15.87:0/697058796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c101180 msgr2=0x7fb69c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/697058796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c101180 0x7fb69c1015e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb69002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 -- 172.21.15.87:0/697058796 shutdown_connections 2024-04-22T13:31:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/697058796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 0x7fb69c101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/697058796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c101180 0x7fb69c1015e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/697058796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c100820 0x7fb69c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 -- 172.21.15.87:0/697058796 >> 172.21.15.87:0/697058796 conn(0x7fb69c0fc440 msgr2=0x7fb69c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 -- 172.21.15.87:0/697058796 shutdown_connections 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.508+0000 7fb6a1b52640 1 -- 172.21.15.87:0/697058796 wait complete. 2024-04-22T13:31:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 Processor -- start 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 -- start start 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c101180 0x7fb69c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 0x7fb69c1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c10fd50 con 0x7fb69c100820 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c10fbd0 con 0x7fb69c101180 2024-04-22T13:31:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb6a1b52640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c10fed0 con 0x7fb69c101b20 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb69b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb69b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58110/0 (socket says 172.21.15.87:58110) 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb69b7fe640 1 -- 172.21.15.87:0/3498790513 learned_addr learned my addr 172.21.15.87:0/3498790513 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb69affd640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c101180 0x7fb69c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.509+0000 7fb69bfff640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 0x7fb69c1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 -- 172.21.15.87:0/3498790513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 msgr2=0x7fb69c1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 0x7fb69c1a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 -- 172.21.15.87:0/3498790513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c101180 msgr2=0x7fb69c19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c101180 0x7fb69c19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 -- 172.21.15.87:0/3498790513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c1a2040 con 0x7fb69c100820 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb69b7fe640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00e9f0 tx=0x7fb68c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c00cd80 con 0x7fb69c100820 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c1a2330 con 0x7fb69c100820 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb68c0108e0 con 0x7fb69c100820 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c018980 con 0x7fb69c100820 2024-04-22T13:31:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.510+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c1a2850 con 0x7fb69c100820 2024-04-22T13:31:55.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.511+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69c10b570 con 0x7fb69c100820 2024-04-22T13:31:55.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.512+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fb68c010430 con 0x7fb69c100820 2024-04-22T13:31:55.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.513+0000 7fb698ff9640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 0x7fb6840a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.513+0000 7fb69affd640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 0x7fb6840a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.513+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb68c0d0c10 con 0x7fb69c100820 2024-04-22T13:31:55.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.513+0000 7fb69affd640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 0x7fb6840a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb69002c7e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.517+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb68c09cb60 con 0x7fb69c100820 2024-04-22T13:31:55.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.709+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb69c062b50 con 0x7fb69c100820 2024-04-22T13:31:55.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.709+0000 7fb698ff9640 1 -- 172.21.15.87:0/3498790513 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fb68c0bd0f0 con 0x7fb69c100820 2024-04-22T13:31:55.705 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 msgr2=0x7fb6840a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 0x7fb6840a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb69002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 msgr2=0x7fb69c19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00e9f0 tx=0x7fb68c00eeb0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 shutdown_connections 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb6840a3db0 0x7fb6840a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb69c101b20 0x7fb69c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb69c101180 0x7fb69c19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 --2- 172.21.15.87:0/3498790513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb69c100820 0x7fb69c19a420 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.711+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 >> 172.21.15.87:0/3498790513 conn(0x7fb69c0fc440 msgr2=0x7fb69c0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.712+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 shutdown_connections 2024-04-22T13:31:55.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.712+0000 7fb6a1b52640 1 -- 172.21.15.87:0/3498790513 wait complete. 2024-04-22T13:31:55.719 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T13:31:55.719 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 -- 172.21.15.87:0/1829650523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018102980 msgr2=0x7f60181092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 --2- 172.21.15.87:0/1829650523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018102980 0x7f60181092a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6010009ca0 tx=0x7f601002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 -- 172.21.15.87:0/1829650523 shutdown_connections 2024-04-22T13:31:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 --2- 172.21.15.87:0/1829650523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018102980 0x7f60181092a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 --2- 172.21.15.87:0/1829650523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018101fc0 0x7f6018102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 --2- 172.21.15.87:0/1829650523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 -- 172.21.15.87:0/1829650523 >> 172.21.15.87:0/1829650523 conn(0x7f60180fc3e0 msgr2=0x7f60180fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 -- 172.21.15.87:0/1829650523 shutdown_connections 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.827+0000 7f601e055640 1 -- 172.21.15.87:0/1829650523 wait complete. 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 Processor -- start 2024-04-22T13:31:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 -- start start 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018102980 0x7f601819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f601810db10 con 0x7f6018101fc0 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f601810d990 con 0x7f6018100e70 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f601e055640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f601810dc90 con 0x7f6018102980 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f6016ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f6016ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58134/0 (socket says 172.21.15.87:58134) 2024-04-22T13:31:55.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f6016ffd640 1 -- 172.21.15.87:0/3487658431 learned_addr learned my addr 172.21.15.87:0/3487658431 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f60177fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.828+0000 7f6017fff640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018102980 0x7f601819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 -- 172.21.15.87:0/3487658431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018102980 msgr2=0x7f601819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018102980 0x7f601819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 -- 172.21.15.87:0/3487658431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 msgr2=0x7f6018198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 -- 172.21.15.87:0/3487658431 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601819fd50 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f60177fe640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6016ffd640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f600c00d770 tx=0x7f600c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c017070 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f600c009c30 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601819ffe0 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c003cc0 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.829+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181a0500 con 0x7f6018101fc0 2024-04-22T13:31:55.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.830+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6018108180 con 0x7f6018101fc0 2024-04-22T13:31:55.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.831+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f600c003e60 con 0x7f6018101fc0 2024-04-22T13:31:55.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.832+0000 7f6014ff9640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 0x7f60000a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:55.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.832+0000 7f60177fe640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 0x7f60000a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:55.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.832+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f600c0d0230 con 0x7f6018101fc0 2024-04-22T13:31:55.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.832+0000 7f60177fe640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 0x7f60000a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f600800bfd0 tx=0x7f600800b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:55.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:55.837+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f600c009de0 con 0x7f6018101fc0 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3422447390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3063363420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:55.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:55.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:55.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3498790513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.072+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f601802d770 con 0x7f6018101fc0 2024-04-22T13:31:56.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.073+0000 7f6014ff9640 1 -- 172.21.15.87:0/3487658431 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f600c09c180 con 0x7f6018101fc0 2024-04-22T13:31:56.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:56.069 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:31:56.069 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:31:56.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 msgr2=0x7f60000a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 0x7f60000a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f600800bfd0 tx=0x7f600800b390 comp rx=0 tx=0).stop 2024-04-22T13:31:56.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 msgr2=0x7f6018198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f600c00d770 tx=0x7f600c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:31:56.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 shutdown_connections 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f60000a3db0 0x7f60000a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6018102980 0x7f601819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6018101fc0 0x7f6018198670 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 --2- 172.21.15.87:0/3487658431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6018100e70 0x7f6018198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 >> 172.21.15.87:0/3487658431 conn(0x7f60180fc3e0 msgr2=0x7f6018105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 shutdown_connections 2024-04-22T13:31:56.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.075+0000 7f601e055640 1 -- 172.21.15.87:0/3487658431 wait complete. 2024-04-22T13:31:56.084 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T13:31:56.084 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T13:31:56.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:31:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T13:31:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3422447390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3063363420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:56.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3498790513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- 172.21.15.87:0/3657657008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 msgr2=0x7f9a10101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 --2- 172.21.15.87:0/3657657008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a10101280 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009ca0 tx=0x7f9a0802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- 172.21.15.87:0/3657657008 shutdown_connections 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 --2- 172.21.15.87:0/3657657008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102a00 0x7f9a10109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 --2- 172.21.15.87:0/3657657008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102040 0x7f9a101024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 --2- 172.21.15.87:0/3657657008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a10101280 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- 172.21.15.87:0/3657657008 >> 172.21.15.87:0/3657657008 conn(0x7f9a100fc440 msgr2=0x7f9a100fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- 172.21.15.87:0/3657657008 shutdown_connections 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- 172.21.15.87:0/3657657008 wait complete. 2024-04-22T13:31:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 Processor -- start 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.191+0000 7f9a18385640 1 -- start start 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102040 0x7f9a1019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102a00 0x7f9a1019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010c9f0 con 0x7f9a10100e60 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010c870 con 0x7f9a10102a00 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010cb70 con 0x7f9a10102040 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58144/0 (socket says 172.21.15.87:58144) 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 -- 172.21.15.87:0/3635648854 learned_addr learned my addr 172.21.15.87:0/3635648854 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a15920640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102040 0x7f9a1019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16922640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102a00 0x7f9a1019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 -- 172.21.15.87:0/3635648854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102040 msgr2=0x7f9a1019cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102040 0x7f9a1019cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 -- 172.21.15.87:0/3635648854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102a00 msgr2=0x7f9a1019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102a00 0x7f9a1019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 -- 172.21.15.87:0/3635648854 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1019fe40 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a16121640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9a08000f80 tx=0x7f9a080346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08041020 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a08034ca0 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.192+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a101a0070 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.193+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08039040 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.193+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a101a05b0 con 0x7f9a10100e60 2024-04-22T13:31:56.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.194+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a10108250 con 0x7f9a10100e60 2024-04-22T13:31:56.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.195+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f9a08044020 con 0x7f9a10100e60 2024-04-22T13:31:56.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.196+0000 7f9a077fe640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 0x7f99fc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.196+0000 7f9a15920640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 0x7f99fc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.196+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9a080f2530 con 0x7f9a10100e60 2024-04-22T13:31:56.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.196+0000 7f9a15920640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 0x7f99fc0a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a000095f0 tx=0x7f9a00005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.201+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a0804b020 con 0x7f9a10100e60 2024-04-22T13:31:56.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3422447390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3063363420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:56.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3498790513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:56.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.391+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a10062390 con 0x7f99fc0a3e30 2024-04-22T13:31:56.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.398+0000 7f9a077fe640 1 -- 172.21.15.87:0/3635648854 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258030 (secure 0 0 0) 0x7f9a10062390 con 0x7f99fc0a3e30 2024-04-22T13:31:56.397 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:56.398 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:31:56.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 msgr2=0x7f99fc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 0x7f99fc0a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a000095f0 tx=0x7f9a00005f90 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 msgr2=0x7f9a1006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9a08000f80 tx=0x7f9a080346d0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 shutdown_connections 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99fc0a3e30 0x7f99fc0a62f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a10102a00 0x7f9a1019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a10102040 0x7f9a1019cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 --2- 172.21.15.87:0/3635648854 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a10100e60 0x7f9a1006ae90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 >> 172.21.15.87:0/3635648854 conn(0x7f9a100fc440 msgr2=0x7f9a10105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 shutdown_connections 2024-04-22T13:31:56.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.402+0000 7f9a18385640 1 -- 172.21.15.87:0/3635648854 wait complete. 2024-04-22T13:31:56.410 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T13:31:56.033581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185627,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":992,"num_read_kb":654,"num_write":467,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369324,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572500,"statfs":{"total":1151940427776,"available":1151562240000,"internally_reserved":0,"allocated":12087296,"data_stored":8919659,"data_compressed":211086,"data_compressed_allocated":278528,"data_compressed_original":3248128,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988831"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.497219+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:52.497219+0000","last_peered":"2024-04-22T13:31:52.497219+0000","last_clean":"2024-04-22T13:31:52.497219+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:52.497219+0000","last_undegraded":"2024-04-22T13:31:52.497219+0000","last_fullsized":"2024-04-22T13:31:52.497219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.266785+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:52.266785+0000","last_peered":"2024-04-22T13:31:52.266785+0000","last_clean":"2024-04-22T13:31:52.266785+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:52.266785+0000","last_undegraded":"2024-04-22T13:31:52.266785+0000","last_fullsized":"2024-04-22T13:31:52.266785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":311,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.521755+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:52.521755+0000","last_peered":"2024-04-22T13:31:52.521755+0000","last_clean":"2024-04-22T13:31:52.521755+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:52.521755+0000","last_undegraded":"2024-04-22T13:31:52.521755+0000","last_fullsized":"2024-04-22T13:31:52.521755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":255,"num_read_kb":191,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.499232+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:52.499232+0000","last_peered":"2024-04-22T13:31:52.499232+0000","last_clean":"2024-04-22T13:31:52.499232+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:52.499232+0000","last_undegraded":"2024-04-22T13:31:52.499232+0000","last_fullsized":"2024-04-22T13:31:52.499232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111358+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.111358+0000","last_peered":"2024-04-22T13:31:52.111358+0000","last_clean":"2024-04-22T13:31:52.111358+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.111358+0000","last_undegraded":"2024-04-22T13:31:52.111358+0000","last_fullsized":"2024-04-22T13:31:52.111358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025859+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025859+0000","last_peered":"2024-04-22T13:31:52.025859+0000","last_clean":"2024-04-22T13:31:52.025859+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025859+0000","last_undegraded":"2024-04-22T13:31:52.025859+0000","last_fullsized":"2024-04-22T13:31:52.025859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025651+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025651+0000","last_peered":"2024-04-22T13:31:52.025651+0000","last_clean":"2024-04-22T13:31:52.025651+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025651+0000","last_undegraded":"2024-04-22T13:31:52.025651+0000","last_fullsized":"2024-04-22T13:31:52.025651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498569+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:52.498569+0000","last_peered":"2024-04-22T13:31:52.498569+0000","last_clean":"2024-04-22T13:31:52.498569+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:52.498569+0000","last_undegraded":"2024-04-22T13:31:52.498569+0000","last_fullsized":"2024-04-22T13:31:52.498569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210431+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:52.210431+0000","last_peered":"2024-04-22T13:31:52.210431+0000","last_clean":"2024-04-22T13:31:52.210431+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:52.210431+0000","last_undegraded":"2024-04-22T13:31:52.210431+0000","last_fullsized":"2024-04-22T13:31:52.210431+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111243+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111243+0000","last_peered":"2024-04-22T13:31:52.111243+0000","last_clean":"2024-04-22T13:31:52.111243+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111243+0000","last_undegraded":"2024-04-22T13:31:52.111243+0000","last_fullsized":"2024-04-22T13:31:52.111243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.225725+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:52.225725+0000","last_peered":"2024-04-22T13:31:52.225725+0000","last_clean":"2024-04-22T13:31:52.225725+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:52.225725+0000","last_undegraded":"2024-04-22T13:31:52.225725+0000","last_fullsized":"2024-04-22T13:31:52.225725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112291+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112291+0000","last_peered":"2024-04-22T13:31:52.112291+0000","last_clean":"2024-04-22T13:31:52.112291+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112291+0000","last_undegraded":"2024-04-22T13:31:52.112291+0000","last_fullsized":"2024-04-22T13:31:52.112291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196033+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196033+0000","last_peered":"2024-04-22T13:31:52.196033+0000","last_clean":"2024-04-22T13:31:52.196033+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196033+0000","last_undegraded":"2024-04-22T13:31:52.196033+0000","last_fullsized":"2024-04-22T13:31:52.196033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210820+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210820+0000","last_peered":"2024-04-22T13:31:52.210820+0000","last_clean":"2024-04-22T13:31:52.210820+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210820+0000","last_undegraded":"2024-04-22T13:31:52.210820+0000","last_fullsized":"2024-04-22T13:31:52.210820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209903+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209903+0000","last_peered":"2024-04-22T13:31:52.209903+0000","last_clean":"2024-04-22T13:31:52.209903+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209903+0000","last_undegraded":"2024-04-22T13:31:52.209903+0000","last_fullsized":"2024-04-22T13:31:52.209903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021839+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021839+0000","last_peered":"2024-04-22T13:31:52.021839+0000","last_clean":"2024-04-22T13:31:52.021839+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021839+0000","last_undegraded":"2024-04-22T13:31:52.021839+0000","last_fullsized":"2024-04-22T13:31:52.021839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209885+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209885+0000","last_peered":"2024-04-22T13:31:52.209885+0000","last_clean":"2024-04-22T13:31:52.209885+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209885+0000","last_undegraded":"2024-04-22T13:31:52.209885+0000","last_fullsized":"2024-04-22T13:31:52.209885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210676+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210676+0000","last_peered":"2024-04-22T13:31:52.210676+0000","last_clean":"2024-04-22T13:31:52.210676+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210676+0000","last_undegraded":"2024-04-22T13:31:52.210676+0000","last_fullsized":"2024-04-22T13:31:52.210676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210117+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210117+0000","last_peered":"2024-04-22T13:31:52.210117+0000","last_clean":"2024-04-22T13:31:52.210117+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210117+0000","last_undegraded":"2024-04-22T13:31:52.210117+0000","last_fullsized":"2024-04-22T13:31:52.210117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209919+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209919+0000","last_peered":"2024-04-22T13:31:52.209919+0000","last_clean":"2024-04-22T13:31:52.209919+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209919+0000","last_undegraded":"2024-04-22T13:31:52.209919+0000","last_fullsized":"2024-04-22T13:31:52.209919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209867+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209867+0000","last_peered":"2024-04-22T13:31:52.209867+0000","last_clean":"2024-04-22T13:31:52.209867+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209867+0000","last_undegraded":"2024-04-22T13:31:52.209867+0000","last_fullsized":"2024-04-22T13:31:52.209867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210274+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210274+0000","last_peered":"2024-04-22T13:31:52.210274+0000","last_clean":"2024-04-22T13:31:52.210274+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210274+0000","last_undegraded":"2024-04-22T13:31:52.210274+0000","last_fullsized":"2024-04-22T13:31:52.210274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022501+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022501+0000","last_peered":"2024-04-22T13:31:52.022501+0000","last_clean":"2024-04-22T13:31:52.022501+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022501+0000","last_undegraded":"2024-04-22T13:31:52.022501+0000","last_fullsized":"2024-04-22T13:31:52.022501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595025,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":856,"num_read_kb":535,"num_write":284,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785075,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1187057,"data_compressed":43378,"data_compressed_allocated":57344,"data_compressed_original":667648,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1058052,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":519761,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531119,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466028,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":1056255,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65118,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66825,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199386,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:31:56.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 -- 172.21.15.87:0/1932935169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc107fe0 msgr2=0x7f1cbc1083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/1932935169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc1083c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009900 tx=0x7f1cb802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 -- 172.21.15.87:0/1932935169 shutdown_connections 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/1932935169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc069a80 0x7f1cbc10eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/1932935169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 0x7f1cbc069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/1932935169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc1083c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 -- 172.21.15.87:0/1932935169 >> 172.21.15.87:0/1932935169 conn(0x7f1cbc077e30 msgr2=0x7f1cbc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 -- 172.21.15.87:0/1932935169 shutdown_connections 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 -- 172.21.15.87:0/1932935169 wait complete. 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.558+0000 7f1cc47f4640 1 Processor -- start 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 -- start start 2024-04-22T13:31:56.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 0x7f1cbc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc112140 con 0x7f1cbc069a80 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc111fc0 con 0x7f1cbc107fe0 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc47f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc1122c0 con 0x7f1cbc0690e0 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58154/0 (socket says 172.21.15.87:58154) 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 -- 172.21.15.87:0/2056999006 learned_addr learned my addr 172.21.15.87:0/2056999006 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc2590640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 0x7f1cbc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc2d91640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 -- 172.21.15.87:0/2056999006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 msgr2=0x7f1cbc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 0x7f1cbc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 -- 172.21.15.87:0/2056999006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 msgr2=0x7f1cbc19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 -- 172.21.15.87:0/2056999006 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc19d6f0 con 0x7f1cbc069a80 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc2d91640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cc1d8f640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00d950 tx=0x7f1cac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.559+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac017030 con 0x7f1cbc069a80 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.560+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc19d9e0 con 0x7f1cbc069a80 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.560+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cac004020 con 0x7f1cbc069a80 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.560+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0047a0 con 0x7f1cbc069a80 2024-04-22T13:31:56.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.560+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc1a5270 con 0x7f1cbc069a80 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.561+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cbc068de0 con 0x7f1cbc069a80 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.562+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f1cac009a30 con 0x7f1cbc069a80 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.562+0000 7f1cb37fe640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 0x7f1ca80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.562+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1cac0d0390 con 0x7f1cbc069a80 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.562+0000 7f1cc2590640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 0x7f1ca80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.563+0000 7f1cc2590640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 0x7f1ca80a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006010 tx=0x7f1cb802c6c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.567+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cac09c930 con 0x7f1cbc069a80 2024-04-22T13:31:56.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.748+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1cbc062390 con 0x7f1ca80a3db0 2024-04-22T13:31:56.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.756+0000 7f1cb37fe640 1 -- 172.21.15.87:0/2056999006 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258030 (secure 0 0 0) 0x7f1cbc062390 con 0x7f1ca80a3db0 2024-04-22T13:31:56.754 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:56.755 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 msgr2=0x7f1ca80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 0x7f1ca80a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006010 tx=0x7f1cb802c6c0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 msgr2=0x7f1cbc198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00d950 tx=0x7f1cac00de10 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 shutdown_connections 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ca80a3db0 0x7f1ca80a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cbc107fe0 0x7f1cbc19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cbc069a80 0x7f1cbc198c40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 --2- 172.21.15.87:0/2056999006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1cbc0690e0 0x7f1cbc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 >> 172.21.15.87:0/2056999006 conn(0x7f1cbc077e30 msgr2=0x7f1cbc100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 shutdown_connections 2024-04-22T13:31:56.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.759+0000 7f1cc47f4640 1 -- 172.21.15.87:0/2056999006 wait complete. 2024-04-22T13:31:56.766 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T13:31:56.033581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185627,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":992,"num_read_kb":654,"num_write":467,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369324,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572500,"statfs":{"total":1151940427776,"available":1151562240000,"internally_reserved":0,"allocated":12087296,"data_stored":8919659,"data_compressed":211086,"data_compressed_allocated":278528,"data_compressed_original":3248128,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988831"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.497219+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:52.497219+0000","last_peered":"2024-04-22T13:31:52.497219+0000","last_clean":"2024-04-22T13:31:52.497219+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:52.497219+0000","last_undegraded":"2024-04-22T13:31:52.497219+0000","last_fullsized":"2024-04-22T13:31:52.497219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.266785+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:52.266785+0000","last_peered":"2024-04-22T13:31:52.266785+0000","last_clean":"2024-04-22T13:31:52.266785+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:52.266785+0000","last_undegraded":"2024-04-22T13:31:52.266785+0000","last_fullsized":"2024-04-22T13:31:52.266785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":311,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.521755+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:52.521755+0000","last_peered":"2024-04-22T13:31:52.521755+0000","last_clean":"2024-04-22T13:31:52.521755+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:52.521755+0000","last_undegraded":"2024-04-22T13:31:52.521755+0000","last_fullsized":"2024-04-22T13:31:52.521755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":255,"num_read_kb":191,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.499232+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:52.499232+0000","last_peered":"2024-04-22T13:31:52.499232+0000","last_clean":"2024-04-22T13:31:52.499232+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:52.499232+0000","last_undegraded":"2024-04-22T13:31:52.499232+0000","last_fullsized":"2024-04-22T13:31:52.499232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111358+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.111358+0000","last_peered":"2024-04-22T13:31:52.111358+0000","last_clean":"2024-04-22T13:31:52.111358+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.111358+0000","last_undegraded":"2024-04-22T13:31:52.111358+0000","last_fullsized":"2024-04-22T13:31:52.111358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025859+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025859+0000","last_peered":"2024-04-22T13:31:52.025859+0000","last_clean":"2024-04-22T13:31:52.025859+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025859+0000","last_undegraded":"2024-04-22T13:31:52.025859+0000","last_fullsized":"2024-04-22T13:31:52.025859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025651+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025651+0000","last_peered":"2024-04-22T13:31:52.025651+0000","last_clean":"2024-04-22T13:31:52.025651+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025651+0000","last_undegraded":"2024-04-22T13:31:52.025651+0000","last_fullsized":"2024-04-22T13:31:52.025651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498569+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:52.498569+0000","last_peered":"2024-04-22T13:31:52.498569+0000","last_clean":"2024-04-22T13:31:52.498569+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:52.498569+0000","last_undegraded":"2024-04-22T13:31:52.498569+0000","last_fullsized":"2024-04-22T13:31:52.498569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210431+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:52.210431+0000","last_peered":"2024-04-22T13:31:52.210431+0000","last_clean":"2024-04-22T13:31:52.210431+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:52.210431+0000","last_undegraded":"2024-04-22T13:31:52.210431+0000","last_fullsized":"2024-04-22T13:31:52.210431+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111243+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111243+0000","last_peered":"2024-04-22T13:31:52.111243+0000","last_clean":"2024-04-22T13:31:52.111243+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111243+0000","last_undegraded":"2024-04-22T13:31:52.111243+0000","last_fullsized":"2024-04-22T13:31:52.111243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.225725+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:52.225725+0000","last_peered":"2024-04-22T13:31:52.225725+0000","last_clean":"2024-04-22T13:31:52.225725+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:52.225725+0000","last_undegraded":"2024-04-22T13:31:52.225725+0000","last_fullsized":"2024-04-22T13:31:52.225725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112291+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112291+0000","last_peered":"2024-04-22T13:31:52.112291+0000","last_clean":"2024-04-22T13:31:52.112291+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112291+0000","last_undegraded":"2024-04-22T13:31:52.112291+0000","last_fullsized":"2024-04-22T13:31:52.112291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196033+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196033+0000","last_peered":"2024-04-22T13:31:52.196033+0000","last_clean":"2024-04-22T13:31:52.196033+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196033+0000","last_undegraded":"2024-04-22T13:31:52.196033+0000","last_fullsized":"2024-04-22T13:31:52.196033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210820+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210820+0000","last_peered":"2024-04-22T13:31:52.210820+0000","last_clean":"2024-04-22T13:31:52.210820+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210820+0000","last_undegraded":"2024-04-22T13:31:52.210820+0000","last_fullsized":"2024-04-22T13:31:52.210820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209903+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209903+0000","last_peered":"2024-04-22T13:31:52.209903+0000","last_clean":"2024-04-22T13:31:52.209903+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209903+0000","last_undegraded":"2024-04-22T13:31:52.209903+0000","last_fullsized":"2024-04-22T13:31:52.209903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021839+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021839+0000","last_peered":"2024-04-22T13:31:52.021839+0000","last_clean":"2024-04-22T13:31:52.021839+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021839+0000","last_undegraded":"2024-04-22T13:31:52.021839+0000","last_fullsized":"2024-04-22T13:31:52.021839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209885+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209885+0000","last_peered":"2024-04-22T13:31:52.209885+0000","last_clean":"2024-04-22T13:31:52.209885+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209885+0000","last_undegraded":"2024-04-22T13:31:52.209885+0000","last_fullsized":"2024-04-22T13:31:52.209885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210676+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210676+0000","last_peered":"2024-04-22T13:31:52.210676+0000","last_clean":"2024-04-22T13:31:52.210676+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210676+0000","last_undegraded":"2024-04-22T13:31:52.210676+0000","last_fullsized":"2024-04-22T13:31:52.210676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210117+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210117+0000","last_peered":"2024-04-22T13:31:52.210117+0000","last_clean":"2024-04-22T13:31:52.210117+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210117+0000","last_undegraded":"2024-04-22T13:31:52.210117+0000","last_fullsized":"2024-04-22T13:31:52.210117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209919+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209919+0000","last_peered":"2024-04-22T13:31:52.209919+0000","last_clean":"2024-04-22T13:31:52.209919+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209919+0000","last_undegraded":"2024-04-22T13:31:52.209919+0000","last_fullsized":"2024-04-22T13:31:52.209919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209867+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209867+0000","last_peered":"2024-04-22T13:31:52.209867+0000","last_clean":"2024-04-22T13:31:52.209867+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209867+0000","last_undegraded":"2024-04-22T13:31:52.209867+0000","last_fullsized":"2024-04-22T13:31:52.209867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210274+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210274+0000","last_peered":"2024-04-22T13:31:52.210274+0000","last_clean":"2024-04-22T13:31:52.210274+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210274+0000","last_undegraded":"2024-04-22T13:31:52.210274+0000","last_fullsized":"2024-04-22T13:31:52.210274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022501+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022501+0000","last_peered":"2024-04-22T13:31:52.022501+0000","last_clean":"2024-04-22T13:31:52.022501+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022501+0000","last_undegraded":"2024-04-22T13:31:52.022501+0000","last_fullsized":"2024-04-22T13:31:52.022501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595025,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":856,"num_read_kb":535,"num_write":284,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785075,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1187057,"data_compressed":43378,"data_compressed_allocated":57344,"data_compressed_original":667648,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1058052,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":519761,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531119,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466028,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":1056255,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65118,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66825,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199386,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:31:56.770 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T13:31:56.770 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 -- 172.21.15.87:0/698431891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d30106f50 msgr2=0x7f9d30107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 --2- 172.21.15.87:0/698431891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d30106f50 0x7f9d30107330 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009930 tx=0x7f9d2402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 -- 172.21.15.87:0/698431891 shutdown_connections 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 --2- 172.21.15.87:0/698431891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30100e80 0x7f9d3010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 --2- 172.21.15.87:0/698431891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d301004c0 0x7f9d30100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 --2- 172.21.15.87:0/698431891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d30106f50 0x7f9d30107330 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.910+0000 7f9d3848c640 1 -- 172.21.15.87:0/698431891 >> 172.21.15.87:0/698431891 conn(0x7f9d30077450 msgr2=0x7f9d300fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.911+0000 7f9d3848c640 1 -- 172.21.15.87:0/698431891 shutdown_connections 2024-04-22T13:31:56.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.911+0000 7f9d3848c640 1 -- 172.21.15.87:0/698431891 wait complete. 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.911+0000 7f9d3848c640 1 Processor -- start 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.911+0000 7f9d3848c640 1 -- start start 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d30100e80 0x7f9d3019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30106f50 0x7f9d301985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d30112280 con 0x7f9d301004c0 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d30112100 con 0x7f9d30100e80 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d3848c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d30112400 con 0x7f9d30106f50 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58166/0 (socket says 172.21.15.87:58166) 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 -- 172.21.15.87:0/970376607 learned_addr learned my addr 172.21.15.87:0/970376607 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d35a27640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d30100e80 0x7f9d3019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36a29640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30106f50 0x7f9d301985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 -- 172.21.15.87:0/970376607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30106f50 msgr2=0x7f9d301985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30106f50 0x7f9d301985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 -- 172.21.15.87:0/970376607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d30100e80 msgr2=0x7f9d3019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:56.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d30100e80 0x7f9d3019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:56.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 -- 172.21.15.87:0/970376607 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d30198d20 con 0x7f9d301004c0 2024-04-22T13:31:56.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.912+0000 7f9d36228640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005940 tx=0x7f9d24005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.914+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24043020 con 0x7f9d301004c0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.914+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d24004350 con 0x7f9d301004c0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.914+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d30198fb0 con 0x7f9d301004c0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.914+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24003a80 con 0x7f9d301004c0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.914+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d301a52b0 con 0x7f9d301004c0 2024-04-22T13:31:56.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.915+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d30069230 con 0x7f9d301004c0 2024-04-22T13:31:56.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.916+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f9d24046020 con 0x7f9d301004c0 2024-04-22T13:31:56.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.917+0000 7f9d237fe640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 0x7f9d1c0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:56.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.917+0000 7f9d35a27640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 0x7f9d1c0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:56.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.917+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9d240f2340 con 0x7f9d301004c0 2024-04-22T13:31:56.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.918+0000 7f9d35a27640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 0x7f9d1c0a63c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d300ffa10 tx=0x7f9d2c005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:56.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.926+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9d240ba960 con 0x7f9d301004c0 2024-04-22T13:31:56.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:56.932+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d2404d030 con 0x7f9d301004c0 2024-04-22T13:31:57.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.106+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d3010a510 con 0x7f9d1c0a3f00 2024-04-22T13:31:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.113+0000 7f9d237fe640 1 -- 172.21.15.87:0/970376607 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258030 (secure 0 0 0) 0x7f9d2c01b3f0 con 0x7f9d1c0a3f00 2024-04-22T13:31:57.111 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 msgr2=0x7f9d1c0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 0x7f9d1c0a63c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d300ffa10 tx=0x7f9d2c005f90 comp rx=0 tx=0).stop 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 msgr2=0x7f9d3019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005940 tx=0x7f9d24005dc0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 shutdown_connections 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d1c0a3f00 0x7f9d1c0a63c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d30106f50 0x7f9d301985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d30100e80 0x7f9d3019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 --2- 172.21.15.87:0/970376607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d301004c0 0x7f9d3019e3f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 >> 172.21.15.87:0/970376607 conn(0x7f9d30077450 msgr2=0x7f9d30105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 shutdown_connections 2024-04-22T13:31:57.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.116+0000 7f9d3848c640 1 -- 172.21.15.87:0/970376607 wait complete. 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:31:57.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3487658431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:57.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:56 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.125 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T13:31:56.033581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185627,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":992,"num_read_kb":654,"num_write":467,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369324,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572500,"statfs":{"total":1151940427776,"available":1151562240000,"internally_reserved":0,"allocated":12087296,"data_stored":8919659,"data_compressed":211086,"data_compressed_allocated":278528,"data_compressed_original":3248128,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.988831"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.497219+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:52.497219+0000","last_peered":"2024-04-22T13:31:52.497219+0000","last_clean":"2024-04-22T13:31:52.497219+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:52.497219+0000","last_undegraded":"2024-04-22T13:31:52.497219+0000","last_fullsized":"2024-04-22T13:31:52.497219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.266785+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:52.266785+0000","last_peered":"2024-04-22T13:31:52.266785+0000","last_clean":"2024-04-22T13:31:52.266785+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:52.266785+0000","last_undegraded":"2024-04-22T13:31:52.266785+0000","last_fullsized":"2024-04-22T13:31:52.266785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":311,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.521755+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:52.521755+0000","last_peered":"2024-04-22T13:31:52.521755+0000","last_clean":"2024-04-22T13:31:52.521755+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:52.521755+0000","last_undegraded":"2024-04-22T13:31:52.521755+0000","last_fullsized":"2024-04-22T13:31:52.521755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":255,"num_read_kb":191,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.499232+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:52.499232+0000","last_peered":"2024-04-22T13:31:52.499232+0000","last_clean":"2024-04-22T13:31:52.499232+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:52.499232+0000","last_undegraded":"2024-04-22T13:31:52.499232+0000","last_fullsized":"2024-04-22T13:31:52.499232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111358+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.111358+0000","last_peered":"2024-04-22T13:31:52.111358+0000","last_clean":"2024-04-22T13:31:52.111358+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.111358+0000","last_undegraded":"2024-04-22T13:31:52.111358+0000","last_fullsized":"2024-04-22T13:31:52.111358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025859+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025859+0000","last_peered":"2024-04-22T13:31:52.025859+0000","last_clean":"2024-04-22T13:31:52.025859+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025859+0000","last_undegraded":"2024-04-22T13:31:52.025859+0000","last_fullsized":"2024-04-22T13:31:52.025859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025651+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025651+0000","last_peered":"2024-04-22T13:31:52.025651+0000","last_clean":"2024-04-22T13:31:52.025651+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025651+0000","last_undegraded":"2024-04-22T13:31:52.025651+0000","last_fullsized":"2024-04-22T13:31:52.025651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498569+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:52.498569+0000","last_peered":"2024-04-22T13:31:52.498569+0000","last_clean":"2024-04-22T13:31:52.498569+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:52.498569+0000","last_undegraded":"2024-04-22T13:31:52.498569+0000","last_fullsized":"2024-04-22T13:31:52.498569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210431+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:52.210431+0000","last_peered":"2024-04-22T13:31:52.210431+0000","last_clean":"2024-04-22T13:31:52.210431+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:52.210431+0000","last_undegraded":"2024-04-22T13:31:52.210431+0000","last_fullsized":"2024-04-22T13:31:52.210431+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111243+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111243+0000","last_peered":"2024-04-22T13:31:52.111243+0000","last_clean":"2024-04-22T13:31:52.111243+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111243+0000","last_undegraded":"2024-04-22T13:31:52.111243+0000","last_fullsized":"2024-04-22T13:31:52.111243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.225725+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:52.225725+0000","last_peered":"2024-04-22T13:31:52.225725+0000","last_clean":"2024-04-22T13:31:52.225725+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:52.225725+0000","last_undegraded":"2024-04-22T13:31:52.225725+0000","last_fullsized":"2024-04-22T13:31:52.225725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112291+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112291+0000","last_peered":"2024-04-22T13:31:52.112291+0000","last_clean":"2024-04-22T13:31:52.112291+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112291+0000","last_undegraded":"2024-04-22T13:31:52.112291+0000","last_fullsized":"2024-04-22T13:31:52.112291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196033+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196033+0000","last_peered":"2024-04-22T13:31:52.196033+0000","last_clean":"2024-04-22T13:31:52.196033+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196033+0000","last_undegraded":"2024-04-22T13:31:52.196033+0000","last_fullsized":"2024-04-22T13:31:52.196033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210820+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210820+0000","last_peered":"2024-04-22T13:31:52.210820+0000","last_clean":"2024-04-22T13:31:52.210820+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210820+0000","last_undegraded":"2024-04-22T13:31:52.210820+0000","last_fullsized":"2024-04-22T13:31:52.210820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209903+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209903+0000","last_peered":"2024-04-22T13:31:52.209903+0000","last_clean":"2024-04-22T13:31:52.209903+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209903+0000","last_undegraded":"2024-04-22T13:31:52.209903+0000","last_fullsized":"2024-04-22T13:31:52.209903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021839+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021839+0000","last_peered":"2024-04-22T13:31:52.021839+0000","last_clean":"2024-04-22T13:31:52.021839+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021839+0000","last_undegraded":"2024-04-22T13:31:52.021839+0000","last_fullsized":"2024-04-22T13:31:52.021839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209885+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209885+0000","last_peered":"2024-04-22T13:31:52.209885+0000","last_clean":"2024-04-22T13:31:52.209885+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209885+0000","last_undegraded":"2024-04-22T13:31:52.209885+0000","last_fullsized":"2024-04-22T13:31:52.209885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210676+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210676+0000","last_peered":"2024-04-22T13:31:52.210676+0000","last_clean":"2024-04-22T13:31:52.210676+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210676+0000","last_undegraded":"2024-04-22T13:31:52.210676+0000","last_fullsized":"2024-04-22T13:31:52.210676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210117+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210117+0000","last_peered":"2024-04-22T13:31:52.210117+0000","last_clean":"2024-04-22T13:31:52.210117+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210117+0000","last_undegraded":"2024-04-22T13:31:52.210117+0000","last_fullsized":"2024-04-22T13:31:52.210117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209919+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209919+0000","last_peered":"2024-04-22T13:31:52.209919+0000","last_clean":"2024-04-22T13:31:52.209919+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209919+0000","last_undegraded":"2024-04-22T13:31:52.209919+0000","last_fullsized":"2024-04-22T13:31:52.209919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209867+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209867+0000","last_peered":"2024-04-22T13:31:52.209867+0000","last_clean":"2024-04-22T13:31:52.209867+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209867+0000","last_undegraded":"2024-04-22T13:31:52.209867+0000","last_fullsized":"2024-04-22T13:31:52.209867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210274+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210274+0000","last_peered":"2024-04-22T13:31:52.210274+0000","last_clean":"2024-04-22T13:31:52.210274+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210274+0000","last_undegraded":"2024-04-22T13:31:52.210274+0000","last_fullsized":"2024-04-22T13:31:52.210274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022501+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022501+0000","last_peered":"2024-04-22T13:31:52.022501+0000","last_clean":"2024-04-22T13:31:52.022501+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022501+0000","last_undegraded":"2024-04-22T13:31:52.022501+0000","last_fullsized":"2024-04-22T13:31:52.022501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595025,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":856,"num_read_kb":535,"num_write":284,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785075,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1187057,"data_compressed":43378,"data_compressed_allocated":57344,"data_compressed_original":667648,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1058052,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":519761,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531119,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466028,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":1056255,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65118,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66825,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199386,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:31:57.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3487658431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:57.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:57.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:56 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi087:/etc/ceph/ceph.conf 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi120:/etc/ceph/ceph.conf 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3487658431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:31:57.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:31:57.215 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:56 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:31:57.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 -- 172.21.15.87:0/3592844179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4101b60 msgr2=0x7f15c410dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 --2- 172.21.15.87:0/3592844179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4101b60 0x7f15c410dc90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009900 tx=0x7f15b402bc80 comp rx=0 tx=0).stop 2024-04-22T13:31:57.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 -- 172.21.15.87:0/3592844179 shutdown_connections 2024-04-22T13:31:57.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 --2- 172.21.15.87:0/3592844179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4101b60 0x7f15c410dc90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 --2- 172.21.15.87:0/3592844179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 --2- 172.21.15.87:0/3592844179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4105ee0 0x7f15c41062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 -- 172.21.15.87:0/3592844179 >> 172.21.15.87:0/3592844179 conn(0x7f15c4077c20 msgr2=0x7f15c4078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 -- 172.21.15.87:0/3592844179 shutdown_connections 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.268+0000 7f15c8e06640 1 -- 172.21.15.87:0/3592844179 wait complete. 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 Processor -- start 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 -- start start 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4101b60 0x7f15c4199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f15c41110f0 con 0x7f15c4105ee0 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15c4110f70 con 0x7f15c4101b60 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c8e06640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15c4111270 con 0x7f15c41011a0 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58180/0 (socket says 172.21.15.87:58180) 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 -- 172.21.15.87:0/2439036350 learned_addr learned my addr 172.21.15.87:0/2439036350 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c259c640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c1d9b640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4101b60 0x7f15c4199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 -- 172.21.15.87:0/2439036350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 msgr2=0x7f15c4199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 -- 172.21.15.87:0/2439036350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4101b60 msgr2=0x7f15c4199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4101b60 0x7f15c4199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c2d9d640 1 -- 172.21.15.87:0/2439036350 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c41a4b50 con 0x7f15c4105ee0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.269+0000 7f15c259c640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15c2d9d640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009230 tx=0x7f15b402c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4043020 con 0x7f15c4105ee0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b402cae0 con 0x7f15c4105ee0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4003970 con 0x7f15c4105ee0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c41a4d20 con 0x7f15c4105ee0 2024-04-22T13:31:57.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.270+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c41a51e0 con 0x7f15c4105ee0 2024-04-22T13:31:57.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.271+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c4069230 con 0x7f15c4105ee0 2024-04-22T13:31:57.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.272+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f15b4046020 con 0x7f15c4105ee0 2024-04-22T13:31:57.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.272+0000 7f15b37fe640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 0x7f15ac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.273+0000 7f15c259c640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 0x7f15ac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.273+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f15b40f2d40 con 0x7f15c4105ee0 2024-04-22T13:31:57.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.273+0000 7f15c259c640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 0x7f15ac0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00af30 tx=0x7f15bc00a2d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.276+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15b404d020 con 0x7f15c4105ee0 2024-04-22T13:31:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.454+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15c4109350 con 0x7f15c4105ee0 2024-04-22T13:31:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.455+0000 7f15b37fe640 1 -- 172.21.15.87:0/2439036350 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7f15b40bf310 con 0x7f15c4105ee0 2024-04-22T13:31:57.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:57.451 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 msgr2=0x7f15ac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 0x7f15ac0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00af30 tx=0x7f15bc00a2d0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 msgr2=0x7f15c41a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009230 tx=0x7f15b402c700 comp rx=0 tx=0).stop 2024-04-22T13:31:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 shutdown_connections 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15ac0a3e40 0x7f15ac0a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15c4105ee0 0x7f15c41a4610 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15c4101b60 0x7f15c4199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 --2- 172.21.15.87:0/2439036350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15c41011a0 0x7f15c4199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 >> 172.21.15.87:0/2439036350 conn(0x7f15c4077c20 msgr2=0x7f15c40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 shutdown_connections 2024-04-22T13:31:57.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.457+0000 7f15c8e06640 1 -- 172.21.15.87:0/2439036350 wait complete. 2024-04-22T13:31:57.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:31:57.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:31:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 -- 172.21.15.87:0/3643206134 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52581021a0 msgr2=0x7f5258102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3643206134 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52581021a0 0x7f5258102580 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5248009900 tx=0x7f524802bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 -- 172.21.15.87:0/3643206134 shutdown_connections 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3643206134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5258107270 0x7f525810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3643206134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258102b50 0x7f5258106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3643206134 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52581021a0 0x7f5258102580 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 -- 172.21.15.87:0/3643206134 >> 172.21.15.87:0/3643206134 conn(0x7f52580fd9c0 msgr2=0x7f52580ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 -- 172.21.15.87:0/3643206134 shutdown_connections 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.571+0000 7f525ddfb640 1 -- 172.21.15.87:0/3643206134 wait complete. 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 Processor -- start 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 -- start start 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52581021a0 0x7f525819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 0x7f525819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f52581121e0 con 0x7f5258198630 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5258112060 con 0x7f5258107270 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f525ddfb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5258112360 con 0x7f52581021a0 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f5257fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f5257fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58198/0 (socket says 172.21.15.87:58198) 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f5257fff640 1 -- 172.21.15.87:0/3038161396 learned_addr learned my addr 172.21.15.87:0/3038161396 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f52577fe640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52581021a0 0x7f525819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.572+0000 7f524ffff640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 0x7f525819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 -- 172.21.15.87:0/3038161396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52581021a0 msgr2=0x7f525819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52581021a0 0x7f525819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 -- 172.21.15.87:0/3038161396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 msgr2=0x7f525819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 0x7f525819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 -- 172.21.15.87:0/3038161396 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5258199150 con 0x7f5258198630 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f524ffff640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 0x7f525819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:57.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f5257fff640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb10 tx=0x7f525000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250004b20 con 0x7f5258198630 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5250004cc0 con 0x7f5258198630 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5258199440 con 0x7f5258198630 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250010670 con 0x7f5258198630 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.573+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52581a5690 con 0x7f5258198630 2024-04-22T13:31:57.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.574+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52580691c0 con 0x7f5258198630 2024-04-22T13:31:57.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.575+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5250005020 con 0x7f5258198630 2024-04-22T13:31:57.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.576+0000 7f52557fa640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 0x7f52440a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.576+0000 7f52577fe640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 0x7f52440a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.576+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f52500d0fc0 con 0x7f5258198630 2024-04-22T13:31:57.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.576+0000 7f52577fe640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 0x7f52440a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f524802c6b0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.580+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f525009d480 con 0x7f5258198630 2024-04-22T13:31:57.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.759+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5258102b50 con 0x7f5258198630 2024-04-22T13:31:57.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.760+0000 7f52557fa640 1 -- 172.21.15.87:0/3038161396 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f525009cd80 con 0x7f5258198630 2024-04-22T13:31:57.756 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 msgr2=0x7f52440a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 0x7f52440a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f524802c6b0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 msgr2=0x7f5258198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb10 tx=0x7f525000c2d0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 shutdown_connections 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52440a3ec0 0x7f52440a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5258198630 0x7f5258198ae0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5258107270 0x7f525819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 --2- 172.21.15.87:0/3038161396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52581021a0 0x7f525819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 >> 172.21.15.87:0/3038161396 conn(0x7f52580fd9c0 msgr2=0x7f52580fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 shutdown_connections 2024-04-22T13:31:57.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.762+0000 7f525ddfb640 1 -- 172.21.15.87:0/3038161396 wait complete. 2024-04-22T13:31:57.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:31:57.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- 172.21.15.87:0/1685185020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2406b170 msgr2=0x7f3b24106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 --2- 172.21.15.87:0/1685185020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2406b170 0x7f3b24106ba0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009900 tx=0x7f3b1402bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- 172.21.15.87:0/1685185020 shutdown_connections 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 --2- 172.21.15.87:0/1685185020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b24107280 0x7f3b2410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 --2- 172.21.15.87:0/1685185020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2406b170 0x7f3b24106ba0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 --2- 172.21.15.87:0/1685185020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b2406a7c0 0x7f3b2406aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- 172.21.15.87:0/1685185020 >> 172.21.15.87:0/1685185020 conn(0x7f3b24076c50 msgr2=0x7f3b24077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- 172.21.15.87:0/1685185020 shutdown_connections 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- 172.21.15.87:0/1685185020 wait complete. 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 Processor -- start 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.875+0000 7f3b2a975640 1 -- start start 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 0x7f3b241986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 0x7f3b24198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3b241121d0 con 0x7f3b2419cf20 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112050 con 0x7f3b24107280 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b2a975640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112350 con 0x7f3b2406a7c0 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58220/0 (socket says 172.21.15.87:58220) 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 -- 172.21.15.87:0/3139001721 learned_addr learned my addr 172.21.15.87:0/3139001721 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b237fe640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 0x7f3b24198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b23fff640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 0x7f3b241986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 -- 172.21.15.87:0/3139001721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 msgr2=0x7f3b241986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 0x7f3b241986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 -- 172.21.15.87:0/3139001721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 msgr2=0x7f3b24198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 0x7f3b24198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 -- 172.21.15.87:0/3139001721 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2419da10 con 0x7f3b2419cf20 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b237fe640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 0x7f3b24198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:57.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b23fff640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 0x7f3b241986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:57.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b28f12640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00cce0 tx=0x7f3b1c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.876+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00f450 con 0x7f3b2419cf20 2024-04-22T13:31:57.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.877+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b1c00faa0 con 0x7f3b2419cf20 2024-04-22T13:31:57.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.877+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c018b70 con 0x7f3b2419cf20 2024-04-22T13:31:57.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.877+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2419dd00 con 0x7f3b2419cf20 2024-04-22T13:31:57.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.877+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b241a5470 con 0x7f3b2419cf20 2024-04-22T13:31:57.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.878+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b24103680 con 0x7f3b2419cf20 2024-04-22T13:31:57.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.879+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3b1c018d10 con 0x7f3b2419cf20 2024-04-22T13:31:57.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.879+0000 7f3b217fa640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 0x7f3b100a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:57.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.880+0000 7f3b23fff640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 0x7f3b100a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:57.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.880+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f3b1c0d0f60 con 0x7f3b2419cf20 2024-04-22T13:31:57.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.880+0000 7f3b23fff640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 0x7f3b100a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009b40 tx=0x7f3b0c005e50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:57.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:57.884+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b1c014030 con 0x7f3b2419cf20 2024-04-22T13:31:58.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.066+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3b241999b0 con 0x7f3b2419cf20 2024-04-22T13:31:58.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.067+0000 7f3b217fa640 1 -- 172.21.15.87:0/3139001721 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f3b1c09d420 con 0x7f3b2419cf20 2024-04-22T13:31:58.063 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:31:58.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 msgr2=0x7f3b100a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 0x7f3b100a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009b40 tx=0x7f3b0c005e50 comp rx=0 tx=0).stop 2024-04-22T13:31:58.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 msgr2=0x7f3b2419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00cce0 tx=0x7f3b1c00a620 comp rx=0 tx=0).stop 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 shutdown_connections 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3b100a3e40 0x7f3b100a6300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3b2419cf20 0x7f3b2419d3d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b24107280 0x7f3b24198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 --2- 172.21.15.87:0/3139001721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b2406a7c0 0x7f3b241986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 >> 172.21.15.87:0/3139001721 conn(0x7f3b24076c50 msgr2=0x7f3b240ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 shutdown_connections 2024-04-22T13:31:58.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.069+0000 7f3b2a975640 1 -- 172.21.15.87:0/3139001721 wait complete. 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2439036350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:58.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:57 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3038161396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:58.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:31:58.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2439036350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:57 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3038161396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:58.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 -- 172.21.15.87:0/2224900578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd750106170 msgr2=0x7fd75010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 --2- 172.21.15.87:0/2224900578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd750106170 0x7fd75010dd40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd740009ca0 tx=0x7fd74002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 -- 172.21.15.87:0/2224900578 shutdown_connections 2024-04-22T13:31:58.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 --2- 172.21.15.87:0/2224900578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd750106170 0x7fd75010dd40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 --2- 172.21.15.87:0/2224900578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750101a50 0x7fd750105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 --2- 172.21.15.87:0/2224900578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 0x7fd750101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 -- 172.21.15.87:0/2224900578 >> 172.21.15.87:0/2224900578 conn(0x7fd750077eb0 msgr2=0x7fd7500782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 -- 172.21.15.87:0/2224900578 shutdown_connections 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.181+0000 7fd7582c1640 1 -- 172.21.15.87:0/2224900578 wait complete. 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 Processor -- start 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 -- start start 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 0x7fd7501995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750106170 0x7fd750199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd7501110e0 con 0x7fd75019a090 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd750110f60 con 0x7fd7501010a0 2024-04-22T13:31:58.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd7582c1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd750111260 con 0x7fd750106170 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58234/0 (socket says 172.21.15.87:58234) 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 -- 172.21.15.87:0/1246388801 learned_addr learned my addr 172.21.15.87:0/1246388801 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75605d640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 0x7fd7501995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75585c640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750106170 0x7fd750199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 -- 172.21.15.87:0/1246388801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750106170 msgr2=0x7fd750199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750106170 0x7fd750199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 -- 172.21.15.87:0/1246388801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 msgr2=0x7fd7501995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 0x7fd7501995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.182+0000 7fd75685e640 1 -- 172.21.15.87:0/1246388801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7501a4e10 con 0x7fd75019a090 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd75685e640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd740005e30 tx=0x7fd740034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740039040 con 0x7fd75019a090 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd740044070 con 0x7fd75019a090 2024-04-22T13:31:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740003e60 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7501a50a0 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.183+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7501a55e0 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.184+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7500691c0 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.185+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd7400324c0 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.185+0000 7fd74f7fe640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 0x7fd7380a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.185+0000 7fd75605d640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 0x7fd7380a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.185+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd7400f2a90 con 0x7fd75019a090 2024-04-22T13:31:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.186+0000 7fd75605d640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 0x7fd7380a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd74800bbe0 tx=0x7fd74800b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.190+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7400bf060 con 0x7fd75019a090 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: Updating smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: Updating smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2439036350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:31:58.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:57 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3038161396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:31:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.364+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd7501093c0 con 0x7fd75019a090 2024-04-22T13:31:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.365+0000 7fd74f7fe640 1 -- 172.21.15.87:0/1246388801 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fd7400be960 con 0x7fd75019a090 2024-04-22T13:31:58.360 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:31:58.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 msgr2=0x7fd7380a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 0x7fd7380a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd74800bbe0 tx=0x7fd74800b390 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 msgr2=0x7fd7501a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd740005e30 tx=0x7fd740034760 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 shutdown_connections 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7380a3ec0 0x7fd7380a6380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd75019a090 0x7fd7501a4640 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd750106170 0x7fd750199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 --2- 172.21.15.87:0/1246388801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7501010a0 0x7fd7501995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 >> 172.21.15.87:0/1246388801 conn(0x7fd750077eb0 msgr2=0x7fd7500feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 shutdown_connections 2024-04-22T13:31:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.367+0000 7fd7582c1640 1 -- 172.21.15.87:0/1246388801 wait complete. 2024-04-22T13:31:58.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 -- 172.21.15.87:0/1965294436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102b40 msgr2=0x7fbff8106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 --2- 172.21.15.87:0/1965294436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102b40 0x7fbff8106b80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009900 tx=0x7fbfec02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 -- 172.21.15.87:0/1965294436 shutdown_connections 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 --2- 172.21.15.87:0/1965294436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff8107260 0x7fbff810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 --2- 172.21.15.87:0/1965294436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102b40 0x7fbff8106b80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 --2- 172.21.15.87:0/1965294436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8102190 0x7fbff8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 -- 172.21.15.87:0/1965294436 >> 172.21.15.87:0/1965294436 conn(0x7fbff80793e0 msgr2=0x7fbff80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 -- 172.21.15.87:0/1965294436 shutdown_connections 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.476+0000 7fbffddf4640 1 -- 172.21.15.87:0/1965294436 wait complete. 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 Processor -- start 2024-04-22T13:31:58.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 -- start start 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8107260 0x7fbff8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 0x7fbff819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8112180 con 0x7fbff8102190 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8112000 con 0x7fbff8107260 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbffddf4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8112300 con 0x7fbff819cf60 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58252/0 (socket says 172.21.15.87:58252) 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 -- 172.21.15.87:0/3521565123 learned_addr learned my addr 172.21.15.87:0/3521565123 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff6ffd640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8107260 0x7fbff8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff7fff640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 0x7fbff819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 -- 172.21.15.87:0/3521565123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 msgr2=0x7fbff819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 0x7fbff819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 -- 172.21.15.87:0/3521565123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8107260 msgr2=0x7fbff8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8107260 0x7fbff8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff77fe640 1 -- 172.21.15.87:0/3521565123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff819da50 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.477+0000 7fbff7fff640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 0x7fbff819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbff77fe640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c890 tx=0x7fbfe800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004220 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff819dd40 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe80043c0 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004a00 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.478+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a5550 con 0x7fbff8102190 2024-04-22T13:31:58.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.479+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff8069230 con 0x7fbff8102190 2024-04-22T13:31:58.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.480+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fbfe80079b0 con 0x7fbff8102190 2024-04-22T13:31:58.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.481+0000 7fbff4ff9640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 0x7fbfe00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.481+0000 7fbff6ffd640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 0x7fbfe00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.481+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbfe80d0790 con 0x7fbff8102190 2024-04-22T13:31:58.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.481+0000 7fbff6ffd640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 0x7fbfe00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbfec005ec0 tx=0x7fbfec005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.485+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfe8014030 con 0x7fbff8102190 2024-04-22T13:31:58.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.699+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbff810a4c0 con 0x7fbff8102190 2024-04-22T13:31:58.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.700+0000 7fbff4ff9640 1 -- 172.21.15.87:0/3521565123 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fbfe809cc50 con 0x7fbff8102190 2024-04-22T13:31:58.696 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:58.696 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:31:58.696 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 msgr2=0x7fbfe00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 0x7fbfe00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbfec005ec0 tx=0x7fbfec005fd0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 msgr2=0x7fbff81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800c890 tx=0x7fbfe800cd50 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 shutdown_connections 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe00a3e40 0x7fbfe00a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbff819cf60 0x7fbff819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbff8107260 0x7fbff8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 --2- 172.21.15.87:0/3521565123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbff8102190 0x7fbff81986a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 >> 172.21.15.87:0/3521565123 conn(0x7fbff80793e0 msgr2=0x7fbff80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 shutdown_connections 2024-04-22T13:31:58.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.702+0000 7fbffddf4640 1 -- 172.21.15.87:0/3521565123 wait complete. 2024-04-22T13:31:58.709 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.812+0000 7fe6e285c640 1 -- 172.21.15.87:0/2958625394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 msgr2=0x7fe6dc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.812+0000 7fe6e285c640 1 --2- 172.21.15.87:0/2958625394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc102570 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009900 tx=0x7fe6cc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 -- 172.21.15.87:0/2958625394 shutdown_connections 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 --2- 172.21.15.87:0/2958625394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 --2- 172.21.15.87:0/2958625394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc102b40 0x7fe6dc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 --2- 172.21.15.87:0/2958625394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc102570 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 -- 172.21.15.87:0/2958625394 >> 172.21.15.87:0/2958625394 conn(0x7fe6dc0fd9f0 msgr2=0x7fe6dc0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 -- 172.21.15.87:0/2958625394 shutdown_connections 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 -- 172.21.15.87:0/2958625394 wait complete. 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 Processor -- start 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 -- start start 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.813+0000 7fe6e285c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc074cd0 0x7fe6dc1a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc112220 con 0x7fe6dc102190 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc1120a0 con 0x7fe6dc074cd0 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc1123a0 con 0x7fe6dc107260 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58274/0 (socket says 172.21.15.87:58274) 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 -- 172.21.15.87:0/3766036266 learned_addr learned my addr 172.21.15.87:0/3766036266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6db7fe640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 -- 172.21.15.87:0/3766036266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 msgr2=0x7fe6dc074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 -- 172.21.15.87:0/3766036266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc074cd0 msgr2=0x7fe6dc1a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc074cd0 0x7fe6dc1a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 -- 172.21.15.87:0/3766036266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6dc1a70e0 con 0x7fe6dc102190 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6db7fe640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6dbfff640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc000c00 tx=0x7fe6cc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc043020 con 0x7fe6dc102190 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc0052b0 con 0x7fe6dc102190 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6dc1a7370 con 0x7fe6dc102190 2024-04-22T13:31:58.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc003c60 con 0x7fe6dc102190 2024-04-22T13:31:58.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.814+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6dc1a78c0 con 0x7fe6dc102190 2024-04-22T13:31:58.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.815+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6dc068de0 con 0x7fe6dc102190 2024-04-22T13:31:58.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.817+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe6cc046020 con 0x7fe6dc102190 2024-04-22T13:31:58.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.817+0000 7fe6d97fa640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 0x7fe6c40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:58.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.818+0000 7fe6db7fe640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 0x7fe6c40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:58.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.818+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe6cc0f3050 con 0x7fe6dc102190 2024-04-22T13:31:58.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.818+0000 7fe6db7fe640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 0x7fe6c40a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc075c90 tx=0x7fe6d0005e50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:58.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:58.821+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6cc04d020 con 0x7fe6dc102190 2024-04-22T13:31:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:58 smithi087 ceph-mon[27999]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3139001721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1246388801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3521565123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:58.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.000+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6dc10a4c0 con 0x7fe6dc102190 2024-04-22T13:31:58.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.001+0000 7fe6d97fa640 1 -- 172.21.15.87:0/3766036266 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7fe6cc0bf620 con 0x7fe6dc102190 2024-04-22T13:31:58.998 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:31:58.998 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 msgr2=0x7fe6c40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 0x7fe6c40a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc075c90 tx=0x7fe6d0005e50 comp rx=0 tx=0).stop 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 msgr2=0x7fe6dc074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc000c00 tx=0x7fe6cc02c7e0 comp rx=0 tx=0).stop 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 shutdown_connections 2024-04-22T13:31:58.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c40a3e40 0x7fe6c40a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc074cd0 0x7fe6dc1a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6dc107260 0x7fe6dc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 --2- 172.21.15.87:0/3766036266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6dc102190 0x7fe6dc074250 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 >> 172.21.15.87:0/3766036266 conn(0x7fe6dc0fd9f0 msgr2=0x7fe6dc0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.003+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 shutdown_connections 2024-04-22T13:31:59.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.004+0000 7fe6e285c640 1 -- 172.21.15.87:0/3766036266 wait complete. 2024-04-22T13:31:59.011 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T13:31:59.011 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T13:31:59.020 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T13:31:59.020 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:31:59.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-22T13:31:59.021 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T13:31:59.021 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:31:59.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:31:59.021 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:31:59.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 -- 172.21.15.87:0/2706089504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 msgr2=0x7f6df0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 --2- 172.21.15.87:0/2706089504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df0101470 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009900 tx=0x7f6de002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:59.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 -- 172.21.15.87:0/3923038645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8106170 msgr2=0x7f73c810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/3923038645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8106170 0x7f73c810dd40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009ca0 tx=0x7f73c002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 -- 172.21.15.87:0/2706089504 shutdown_connections 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 --2- 172.21.15.87:0/2706089504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 0x7f6df010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 --2- 172.21.15.87:0/2706089504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df0101a40 0x7f6df0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 --2- 172.21.15.87:0/2706089504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df0101470 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 -- 172.21.15.87:0/2706089504 >> 172.21.15.87:0/2706089504 conn(0x7f6df0077f50 msgr2=0x7f6df0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 -- 172.21.15.87:0/3923038645 shutdown_connections 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/3923038645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8106170 0x7f73c810dd40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/3923038645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8101aa0 0x7f73c8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/3923038645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c81010f0 0x7f73c81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 -- 172.21.15.87:0/3923038645 >> 172.21.15.87:0/3923038645 conn(0x7f73c8077fd0 msgr2=0x7f73c80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 -- 172.21.15.87:0/2706089504 shutdown_connections 2024-04-22T13:31:59.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f6df551e640 1 -- 172.21.15.87:0/2706089504 wait complete. 2024-04-22T13:31:59.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 -- 172.21.15.87:0/3923038645 shutdown_connections 2024-04-22T13:31:59.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.129+0000 7f73ce4b3640 1 -- 172.21.15.87:0/3923038645 wait complete. 2024-04-22T13:31:59.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 Processor -- start 2024-04-22T13:31:59.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 Processor -- start 2024-04-22T13:31:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 -- start start 2024-04-22T13:31:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 -- start start 2024-04-22T13:31:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 0x7f73c8199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c819a010 0x7f73c81a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81110c0 con 0x7f73c81010f0 2024-04-22T13:31:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 0x7f6df019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8110f40 con 0x7f73c819a010 2024-04-22T13:31:59.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73ce4b3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8111240 con 0x7f73c8106170 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 0x7f6df0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01110c0 con 0x7f6df0101090 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0110f40 con 0x7f6df0106160 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6df551e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0111240 con 0x7f6df01985d0 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73c7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f6deeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73c7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58310/0 (socket says 172.21.15.87:58310) 2024-04-22T13:31:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73c7fff640 1 -- 172.21.15.87:0/534951969 learned_addr learned my addr 172.21.15.87:0/534951969 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.130+0000 7f73c77fe640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 0x7f73c8199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58326/0 (socket says 172.21.15.87:58326) 2024-04-22T13:31:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 -- 172.21.15.87:0/3235778096 learned_addr learned my addr 172.21.15.87:0/3235778096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6def7fe640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 0x7f6df0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73cca50640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c819a010 0x7f73c81a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 0x7f6df019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 -- 172.21.15.87:0/534951969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 msgr2=0x7f73c8199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 0x7f73c8199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 -- 172.21.15.87:0/534951969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c819a010 msgr2=0x7f73c81a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c819a010 0x7f73c81a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 -- 172.21.15.87:0/534951969 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c81a4d90 con 0x7f73c81010f0 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c77fe640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 0x7f73c8199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:59.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 -- 172.21.15.87:0/3235778096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 msgr2=0x7f6df0198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 0x7f6df0198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 -- 172.21.15.87:0/3235778096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 msgr2=0x7f6df019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 0x7f6df019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 -- 172.21.15.87:0/3235778096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df01990f0 con 0x7f6df0101090 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6def7fe640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 0x7f6df0198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c7fff640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f73b800ea60 tx=0x7f73b800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6deeffd640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6de0005940 tx=0x7f6de00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b800cde0 con 0x7f73c81010f0 2024-04-22T13:31:59.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c81a5080 con 0x7f73c81010f0 2024-04-22T13:31:59.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73b80035f0 con 0x7f73c81010f0 2024-04-22T13:31:59.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8010690 con 0x7f73c81010f0 2024-04-22T13:31:59.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6dd7fff640 1 -- 172.21.15.87:0/3235778096 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0043020 con 0x7f6df0101090 2024-04-22T13:31:59.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c81a5620 con 0x7f73c81010f0 2024-04-22T13:31:59.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6dd7fff640 1 -- 172.21.15.87:0/3235778096 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de002caf0 con 0x7f6df0101090 2024-04-22T13:31:59.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.131+0000 7f6dd7fff640 1 -- 172.21.15.87:0/3235778096 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0003af0 con 0x7f6df0101090 2024-04-22T13:31:59.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.132+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df0199380 con 0x7f6df0101090 2024-04-22T13:31:59.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.132+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df0102cf0 con 0x7f6df0101090 2024-04-22T13:31:59.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.133+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c804fce0 con 0x7f73c81010f0 2024-04-22T13:31:59.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f73b80037a0 con 0x7f73c81010f0 2024-04-22T13:31:59.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f6dd7fff640 1 -- 172.21.15.87:0/3235778096 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f6de0046020 con 0x7f6df0101090 2024-04-22T13:31:59.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f6df81e0740 1 Processor -- start 2024-04-22T13:31:59.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f73c57fa640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 0x7f73b00a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f6df81e0740 1 -- start start 2024-04-22T13:31:59.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.134+0000 7f73c77fe640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 0x7f73b00a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f73b80d0cc0 con 0x7f73c81010f0 2024-04-22T13:31:59.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c78840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5bd3f00 con 0x55d3f5c78e10 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5b71590 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6df81e0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5bcfec0 con 0x55d3f5c70290 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dd7fff640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 0x7f6dd80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 0x7f6dd80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dee7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6deeffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c78840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dee7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58328/0 (socket says 172.21.15.87:58328) 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dee7fc640 1 -- 172.21.15.87:0/1525134136 learned_addr learned my addr 172.21.15.87:0/1525134136 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:46114/0 (socket says 172.21.15.87:46114) 2024-04-22T13:31:59.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dd7fff640 1 -- 172.21.15.87:0/3235778096 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6de00f29c0 con 0x7f6df0101090 2024-04-22T13:31:59.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f73c77fe640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 0x7f73b00a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c806a150 tx=0x7f73bc005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 -- 172.21.15.87:0/1525134136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 msgr2=0x55d3f5c78840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c78840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 -- 172.21.15.87:0/1525134136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 msgr2=0x55d3f5c7b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 -- 172.21.15.87:0/1525134136 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3f59ae7e0 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.135+0000 7f6def7fe640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 0x7f6dd80a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005fd0 tx=0x7f6de8011690 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6def7fe640 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005df0 tx=0x7f6de8011f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc89d6b2a8b8c964 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6dd57fa640 1 -- 172.21.15.87:0/1525134136 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8020c80 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6dd57fa640 1 -- 172.21.15.87:0/1525134136 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de8020e20 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6dd57fa640 1 -- 172.21.15.87:0/1525134136 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de80046d0 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 -- 172.21.15.87:0/1525134136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 msgr2=0x55d3f5c6fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005df0 tx=0x7f6de8011f90 comp rx=0 tx=0).stop 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 -- 172.21.15.87:0/1525134136 shutdown_connections 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c7b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c78840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 --2- 172.21.15.87:0/1525134136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c6fcc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.136+0000 7f6df81e0740 1 -- 172.21.15.87:0/1525134136 >> 172.21.15.87:0/1525134136 conn(0x55d3f5c69fb0 msgr2=0x55d3f5c6a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- 172.21.15.87:0/1525134136 shutdown_connections 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- 172.21.15.87:0/1525134136 wait complete. 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- start start 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 Processor -- start 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5c7bd20 con 0x55d3f5c78e10 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5b8d580 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d3f5c7bea0 con 0x55d3f5c70290 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6df81e0740 10 client.0 _ll_register_callbacks cb 0x55d3f5b93650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58344/0 (socket says 172.21.15.87:58344) 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 -- 172.21.15.87:0/2411864990 learned_addr learned my addr 172.21.15.87:0/2411864990 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6deeffd640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 -- 172.21.15.87:0/2411864990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 msgr2=0x55d3f5c763d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c763d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.137+0000 7f6def7fe640 1 -- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 msgr2=0x55d3f5c75e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.138+0000 7f6def7fe640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c75e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.138+0000 7f6def7fe640 1 -- 172.21.15.87:0/2411864990 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3f5a0ef70 con 0x55d3f5c78e10 2024-04-22T13:31:59.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.138+0000 7f6def7fe640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005c10 tx=0x7f6de8020c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39e1d5d54bf218e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.138+0000 7f6dd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:59.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.140+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73b809d180 con 0x7f73c81010f0 2024-04-22T13:31:59.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.140+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de801a040 con 0x55d3f5c78e10 2024-04-22T13:31:59.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.140+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6de8003e00 con 0x55d3f5c78e10 2024-04-22T13:31:59.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.140+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de8020e20 con 0x55d3f5c78e10 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6de8005040 con 0x55d3f5c78e10 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 10 client.15507 fetch_fsmap learned FSMap version 21 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d3f58a6170 con 0x55d3f5c78e10 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de8020e20 con 0x55d3f5c78e10 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6de8030070 con 0x55d3f5c78e10 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 10 client.15507 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 10 client.15507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 20 client.15507 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5c8b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 4 client.15507 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3f5be1920 con 0x55d3f5c892d0 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.141+0000 7f6deeffd640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5c8b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.142+0000 7f6dd2ff5640 10 client.15507 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.142+0000 7f6deeffd640 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5c8b6c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd36e61849a0e274 server_cookie=d46b5a4fd6427581 in_seq=0 out_seq=0 2024-04-22T13:31:59.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.142+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/2411864990 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3f5be1920 con 0x55d3f5c892d0 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.142+0000 7f6dd2ff5640 10 client.15507 handle_command_reply: tid=0 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.142+0000 7f6df81e0740 1 client.15507 shutdown 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 msgr2=0x55d3f5c76910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005c10 tx=0x7f6de8020c50 comp rx=0 tx=0).stop 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 shutdown_connections 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5c8b6c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c76910 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c763d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6dd2ff5640 0 client.15507 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 --2- 172.21.15.87:0/2411864990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c75e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 >> 172.21.15.87:0/2411864990 conn(0x55d3f5c69fb0 msgr2=0x55d3f5c6bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 shutdown_connections 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 1 -- 172.21.15.87:0/2411864990 wait complete. 2024-04-22T13:31:59.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.143+0000 7f6df81e0740 20 client.15507 trim_cache size 0 max 16384 2024-04-22T13:31:59.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 Processor -- start 2024-04-22T13:31:59.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 -- start start 2024-04-22T13:31:59.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c785c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c84c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01ae0c0 con 0x55d3f5c78e10 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01ab990 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6df81e0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01a9800 con 0x55d3f5c70290 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6def7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58346/0 (socket says 172.21.15.87:58346) 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6def7fe640 1 -- 172.21.15.87:0/748741547 learned_addr learned my addr 172.21.15.87:0/748741547 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c84c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.160+0000 7f6deeffd640 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c785c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 -- 172.21.15.87:0/748741547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 msgr2=0x55d3f5c84c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c84c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 -- 172.21.15.87:0/748741547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 msgr2=0x55d3f5c785c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c785c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 -- 172.21.15.87:0/748741547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3f5c743e0 con 0x55d3f5c78e10 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6def7fe640 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6de800fdc0 tx=0x7f6de8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f236e58e1b59cfa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6dd4ff9640 1 -- 172.21.15.87:0/748741547 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de803f020 con 0x55d3f5c78e10 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6dd4ff9640 1 -- 172.21.15.87:0/748741547 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de800fe50 con 0x55d3f5c78e10 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.161+0000 7f6dd4ff9640 1 -- 172.21.15.87:0/748741547 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8003c00 con 0x55d3f5c78e10 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- 172.21.15.87:0/748741547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 msgr2=0x55d3f5c6ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6de800fdc0 tx=0x7f6de8011f90 comp rx=0 tx=0).stop 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- 172.21.15.87:0/748741547 shutdown_connections 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c78e10 0x55d3f5c6ba90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c70290 0x55d3f5c84c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- 172.21.15.87:0/748741547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c785c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- 172.21.15.87:0/748741547 >> 172.21.15.87:0/748741547 conn(0x55d3f5c69fb0 msgr2=0x55d3f5c6b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- 172.21.15.87:0/748741547 shutdown_connections 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- 172.21.15.87:0/748741547 wait complete. 2024-04-22T13:31:59.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- start start 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 Processor -- start 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c70290 0x55d3f5c7d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c78e10 0x55d3f5cee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01a9e50 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01ab2c0 con 0x55d3f5c70290 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01aa520 con 0x55d3f5c78e10 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6df81e0740 10 client.0 _ll_register_callbacks cb 0x7f6de400fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6deeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6deeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58362/0 (socket says 172.21.15.87:58362) 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.162+0000 7f6deeffd640 1 -- 172.21.15.87:0/656462976 learned_addr learned my addr 172.21.15.87:0/656462976 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c70290 0x55d3f5c7d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6def7fe640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c78e10 0x55d3f5cee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 -- 172.21.15.87:0/656462976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c78e10 msgr2=0x55d3f5cee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c78e10 0x55d3f5cee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 -- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c70290 msgr2=0x55d3f5c7d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c70290 0x55d3f5c7d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 -- 172.21.15.87:0/656462976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3f5ceec30 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6deeffd640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6de0005ec0 tx=0x7f6de0045650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b22e7d9b96e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00c2c40 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6de0003c90 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de800fe50 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6de00c7500 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6df81e0740 10 client.15513 fetch_fsmap learned FSMap version 21 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d3f5cefe70 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de800fe50 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.163+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6de00a90b0 con 0x55d3f5c6f8e0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 10 client.15513 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 10 client.15513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 20 client.15513 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5cf25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 4 client.15513 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d3f5bdb790 con 0x55d3f5c892d0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5cf25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6dd2ff5640 10 client.15513 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.164+0000 7f6dee7fc640 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5cf25a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1893b70eb1872e7 server_cookie=ab872bc4c07af16f in_seq=0 out_seq=0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6dd2ff5640 1 -- 172.21.15.87:0/656462976 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d3f5bdb790 con 0x55d3f5c892d0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6dd2ff5640 10 client.15513 handle_command_reply: tid=0 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 client.15513 shutdown 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 msgr2=0x55d3f5c7cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6de0005ec0 tx=0x7f6de0045650 comp rx=0 tx=0).stop 2024-04-22T13:31:59.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 shutdown_connections 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3f5c892d0 0x55d3f5cf25a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3f5c78e10 0x55d3f5cee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6dd2ff5640 0 client.15513 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3f5c70290 0x55d3f5c7d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 --2- 172.21.15.87:0/656462976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3f5c6f8e0 0x55d3f5c7cf40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 >> 172.21.15.87:0/656462976 conn(0x55d3f5c69fb0 msgr2=0x55d3f5c6bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.165+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 shutdown_connections 2024-04-22T13:31:59.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df81e0740 1 -- 172.21.15.87:0/656462976 wait complete. 2024-04-22T13:31:59.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df81e0740 20 client.15513 trim_cache size 0 max 16384 2024-04-22T13:31:59.169 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:59.169 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:31:59.169 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "10e18270-9999-40e7-86bf-ac6dda09d099", 2024-04-22T13:31:59.170 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:31:59.170 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 msgr2=0x7f6dd80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 0x7f6dd80a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005fd0 tx=0x7f6de8011690 comp rx=0 tx=0).stop 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 msgr2=0x7f6df019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6de0005940 tx=0x7f6de00045e0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 shutdown_connections 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6dd80a3ec0 0x7f6dd80a6380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6df01985d0 0x7f6df0198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6df0106160 0x7f6df019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 --2- 172.21.15.87:0/3235778096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6df0101090 0x7f6df019e420 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 >> 172.21.15.87:0/3235778096 conn(0x7f6df0077f50 msgr2=0x7f6df0069520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 shutdown_connections 2024-04-22T13:31:59.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.166+0000 7f6df551e640 1 -- 172.21.15.87:0/3235778096 wait complete. 2024-04-22T13:31:59.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:58 smithi120 ceph-mon[31101]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:59.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3139001721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:59.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1246388801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:59.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3521565123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:59.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:31:59.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:31:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:58 smithi132 ceph-mon[31531]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:31:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3139001721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:31:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1246388801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:31:59.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3521565123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:31:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 -- 172.21.15.87:0/3386559439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc601031c0 msgr2=0x7fbc601035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 --2- 172.21.15.87:0/3386559439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc601031c0 0x7fbc601035a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009900 tx=0x7fbc5402bd60 comp rx=0 tx=0).stop 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 -- 172.21.15.87:0/3386559439 shutdown_connections 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 --2- 172.21.15.87:0/3386559439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 0x7fbc6010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 --2- 172.21.15.87:0/3386559439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc60103ae0 0x7fbc6010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 --2- 172.21.15.87:0/3386559439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc601031c0 0x7fbc601035a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 -- 172.21.15.87:0/3386559439 >> 172.21.15.87:0/3386559439 conn(0x7fbc60078000 msgr2=0x7fbc60078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 -- 172.21.15.87:0/3386559439 shutdown_connections 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.282+0000 7fbc6581f640 1 -- 172.21.15.87:0/3386559439 wait complete. 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 Processor -- start 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 -- start start 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 0x7fbc60198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 0x7fbc6019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60112170 con 0x7fbc60103ae0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60111ff0 con 0x7fbc6010c500 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc6581f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601122f0 con 0x7fbc601031c0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58372/0 (socket says 172.21.15.87:58372) 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/33940306 learned_addr learned my addr 172.21.15.87:0/33940306 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.283+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 0x7fbc6019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5effd640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 0x7fbc60198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/33940306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 msgr2=0x7fbc60198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 0x7fbc60198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/33940306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 msgr2=0x7fbc6019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 0x7fbc6019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/33940306 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6019d6b0 con 0x7fbc60103ae0 2024-04-22T13:31:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5effd640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 0x7fbc60198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d950 tx=0x7fbc5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc6481d640 1 -- 172.21.15.87:0/33940306 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50017070 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6019d9a0 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc6481d640 1 -- 172.21.15.87:0/33940306 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc50009c30 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc6481d640 1 -- 172.21.15.87:0/33940306 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000f620 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.284+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc601a5230 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.286+0000 7fbc6481d640 1 -- 172.21.15.87:0/33940306 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fbc50010460 con 0x7fbc60103ae0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.286+0000 7fbc6840d740 1 Processor -- start 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.286+0000 7fbc6840d740 1 -- start start 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f723c2e1e0 con 0x55f723cc92b0 2024-04-22T13:31:59.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f723bcb390 con 0x55f723cc9bd0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6840d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f723c2a1a0 con 0x55f723cd6e90 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6481d640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 0x7fbc480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58388/0 (socket says 172.21.15.87:58388) 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:46182/0 (socket says 172.21.15.87:46182) 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 -- 172.21.15.87:0/3005997417 learned_addr learned my addr 172.21.15.87:0/3005997417 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 0x7fbc480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc6481d640 1 -- 172.21.15.87:0/33940306 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbc500d06a0 con 0x7fbc60103ae0 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 -- 172.21.15.87:0/3005997417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 msgr2=0x55f723cd9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 -- 172.21.15.87:0/3005997417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 msgr2=0x55f723cc9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 -- 172.21.15.87:0/3005997417 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f723a0a7d0 con 0x55f723cc9bd0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5effd640 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc54006010 tx=0x7fbc5402c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df50df14efa65c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.287+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 0x7fbc480a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc580092f0 tx=0x7fbc58004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc45ffb640 1 -- 172.21.15.87:0/3005997417 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54041020 con 0x55f723cc9bd0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc45ffb640 1 -- 172.21.15.87:0/3005997417 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5402c910 con 0x55f723cc9bd0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc45ffb640 1 -- 172.21.15.87:0/3005997417 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54005420 con 0x55f723cc9bd0 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 -- 172.21.15.87:0/3005997417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 msgr2=0x55f723cd6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc54006010 tx=0x7fbc5402c780 comp rx=0 tx=0).stop 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 -- 172.21.15.87:0/3005997417 shutdown_connections 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723cd6950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 --2- 172.21.15.87:0/3005997417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cc9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 -- 172.21.15.87:0/3005997417 >> 172.21.15.87:0/3005997417 conn(0x55f723cc3cb0 msgr2=0x55f723cc4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 -- 172.21.15.87:0/3005997417 shutdown_connections 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.288+0000 7fbc6840d740 1 -- 172.21.15.87:0/3005997417 wait complete. 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 -- start start 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 Processor -- start 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 0x55f723cd3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 0x55f723cd4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f723cd9bf0 con 0x55f723cd6e90 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f723be72f0 con 0x55f723cc92b0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f723cd9d70 con 0x55f723cc9bd0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc6840d740 10 client.0 _ll_register_callbacks cb 0x55f723bf4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58390/0 (socket says 172.21.15.87:58390) 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc5f7fe640 1 -- 172.21.15.87:0/4133701436 learned_addr learned my addr 172.21.15.87:0/4133701436 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.289+0000 7fbc5effd640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 0x55f723cd3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 0x55f723cd4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 -- 172.21.15.87:0/4133701436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 msgr2=0x55f723cd4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 0x55f723cd4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 -- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 msgr2=0x55f723cd3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 0x55f723cd3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 -- 172.21.15.87:0/4133701436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7239403f0 con 0x55f723cd6e90 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5effd640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 0x55f723cd3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 0x55f723cd4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc580092a0 tx=0x7fbc58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9cd86ff41943eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58024070 con 0x55f723cd6e90 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbc58028400 con 0x55f723cd6e90 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5402c910 con 0x55f723cd6e90 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbc58028600 con 0x55f723cd6e90 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc6840d740 10 client.15525 fetch_fsmap learned FSMap version 21 2024-04-22T13:31:59.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f7239000f0 con 0x55f723cd6e90 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc5402c910 con 0x55f723cd6e90 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbc58030030 con 0x55f723cd6e90 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc6840d740 10 client.15525 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.290+0000 7fbc6840d740 10 client.15525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc6840d740 20 client.15525 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723ce6dc0 0x55f723ce91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc6840d740 4 client.15525 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f723c3bc00 con 0x55f723ce6dc0 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc5effd640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723ce6dc0 0x55f723ce91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc5effd640 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723ce6dc0 0x55f723ce91f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a1086f1de0038e server_cookie=74edd6a9819a89ae in_seq=0 out_seq=0 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.291+0000 7fbc3effd640 10 client.15525 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc3effd640 1 -- 172.21.15.87:0/4133701436 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f723c3bc00 con 0x55f723ce6dc0 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc3effd640 10 client.15525 handle_command_reply: tid=0 2024-04-22T13:31:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 client.15525 shutdown 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 msgr2=0x55f723cd4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc580092a0 tx=0x7fbc58004a40 comp rx=0 tx=0).stop 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 shutdown_connections 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723ce6dc0 0x55f723ce91f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cd6e90 0x55f723cd4890 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc9bd0 0x55f723cd4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 --2- 172.21.15.87:0/4133701436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc92b0 0x55f723cd3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 >> 172.21.15.87:0/4133701436 conn(0x55f723cc3cb0 msgr2=0x55f723cc5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc3effd640 0 client.15525 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 shutdown_connections 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 1 -- 172.21.15.87:0/4133701436 wait complete. 2024-04-22T13:31:59.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.292+0000 7fbc6840d740 20 client.15525 trim_cache size 0 max 16384 2024-04-22T13:31:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 Processor -- start 2024-04-22T13:31:59.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 -- start start 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723ce2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601ae1c0 con 0x55f723cc92b0 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601aba90 con 0x55f723cc9bd0 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc6840d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601a9920 con 0x55f723cd6e90 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58404/0 (socket says 172.21.15.87:58404) 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 -- 172.21.15.87:0/2924891949 learned_addr learned my addr 172.21.15.87:0/2924891949 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723ce2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 -- 172.21.15.87:0/2924891949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 msgr2=0x55f723cd33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 -- 172.21.15.87:0/2924891949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 msgr2=0x55f723ce2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723ce2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 -- 172.21.15.87:0/2924891949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f723cd2e20 con 0x55f723cc92b0 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.310+0000 7fbc5effd640 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009230 tx=0x7fbc5402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3f6df5099ce6b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc457fa640 1 -- 172.21.15.87:0/2924891949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54041040 con 0x55f723cc92b0 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc457fa640 1 -- 172.21.15.87:0/2924891949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc54056070 con 0x55f723cc92b0 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc457fa640 1 -- 172.21.15.87:0/2924891949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54050750 con 0x55f723cc92b0 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 -- 172.21.15.87:0/2924891949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 msgr2=0x55f723cd6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009230 tx=0x7fbc5402e040 comp rx=0 tx=0).stop 2024-04-22T13:31:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 -- 172.21.15.87:0/2924891949 shutdown_connections 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cd6e90 0x55f723cd33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cc9bd0 0x55f723ce2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 --2- 172.21.15.87:0/2924891949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc92b0 0x55f723cd6540 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 -- 172.21.15.87:0/2924891949 >> 172.21.15.87:0/2924891949 conn(0x55f723cc3cb0 msgr2=0x55f723cc24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 -- 172.21.15.87:0/2924891949 shutdown_connections 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.311+0000 7fbc6840d740 1 -- 172.21.15.87:0/2924891949 wait complete. 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 -- start start 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 Processor -- start 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 0x55f723cdf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 0x55f723cd0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601a9f70 con 0x55f723cc9bd0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601ab3c0 con 0x55f723cd6e90 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601aa620 con 0x55f723cc92b0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc6840d740 10 client.0 _ll_register_callbacks cb 0x7fbc58009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58420/0 (socket says 172.21.15.87:58420) 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/425513478 learned_addr learned my addr 172.21.15.87:0/425513478 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5effd640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 0x55f723cdf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 0x55f723cd0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/425513478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 msgr2=0x55f723cdf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 0x55f723cdf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 msgr2=0x55f723cd0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 0x55f723cd0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5e7fc640 1 -- 172.21.15.87:0/425513478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f723cd07a0 con 0x55f723cc9bd0 2024-04-22T13:31:59.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5f7fe640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 0x55f723cd0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.312+0000 7fbc5effd640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 0x55f723cdf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc5e7fc640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbc50096fd0 tx=0x7fbc500209d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82b6a0d13d4ac92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50020050 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbc50016170 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc54056070 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbc50009de0 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 10 client.15531 fetch_fsmap learned FSMap version 21 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f723d4cc30 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc54056070 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbc500cdfc0 con 0x55f723cc9bd0 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 10 client.15531 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 10 client.15531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:31:59.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 20 client.15531 populate_metadata read hostname 'smithi087' 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723d4d400 0x55f723d4f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 4 client.15531 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.313+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f723c35a70 con 0x55f723d4d400 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc5effd640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723d4d400 0x55f723d4f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc3effd640 10 client.15531 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc5effd640 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723d4d400 0x55f723d4f820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=388eb12d5dea3cf7 server_cookie=2a73bc117eb20067 in_seq=0 out_seq=0 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc3effd640 1 -- 172.21.15.87:0/425513478 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f723c35a70 con 0x55f723d4d400 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc3effd640 10 client.15531 handle_command_reply: tid=0 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 client.15531 shutdown 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 msgr2=0x55f723cdf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbc50096fd0 tx=0x7fbc500209d0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 shutdown_connections 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f723d4d400 0x55f723d4f820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f723cd6e90 0x55f723cd0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc3effd640 0 client.15531 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f723cc9bd0 0x55f723cdf540 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 --2- 172.21.15.87:0/425513478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f723cc92b0 0x55f723cdf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.314+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 >> 172.21.15.87:0/425513478 conn(0x55f723cc3cb0 msgr2=0x55f723cc2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 shutdown_connections 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6840d740 1 -- 172.21.15.87:0/425513478 wait complete. 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6840d740 20 client.15531 trim_cache size 0 max 16384 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:31:59.320 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: "10e18270-9999-40e7-86bf-ac6dda09d099": { 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "10e18270-9999-40e7-86bf-ac6dda09d099", 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:31:59.321 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:31:59.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 msgr2=0x7fbc480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 0x7fbc480a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc580092f0 tx=0x7fbc58004ab0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 msgr2=0x7fbc60198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d950 tx=0x7fbc5000de10 comp rx=0 tx=0).stop 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 shutdown_connections 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbc480a3e40 0x7fbc480a6300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc6010c500 0x7fbc6019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc60103ae0 0x7fbc60198c50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 --2- 172.21.15.87:0/33940306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc601031c0 0x7fbc60198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 >> 172.21.15.87:0/33940306 conn(0x7fbc60078000 msgr2=0x7fbc600fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 shutdown_connections 2024-04-22T13:31:59.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.315+0000 7fbc6581f640 1 -- 172.21.15.87:0/33940306 wait complete. 2024-04-22T13:31:59.324 INFO:tasks.cephfs.filesystem:scrub status for tag:10e18270-9999-40e7-86bf-ac6dda09d099 - {'path': '/', 'tag': '10e18270-9999-40e7-86bf-ac6dda09d099', 'options': 'recursive,force'} 2024-04-22T13:31:59.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.352+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f73c8101aa0 con 0x7f73b00a3ee0 2024-04-22T13:31:59.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.417+0000 7f73c57fa640 1 -- 172.21.15.87:0/534951969 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f73c8101aa0 con 0x7f73b00a3ee0 2024-04-22T13:31:59.413 INFO:teuthology.orchestra.run.smithi087.stdout:Schedule set for path / 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 msgr2=0x7f73b00a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 0x7f73b00a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c806a150 tx=0x7f73bc005f90 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 msgr2=0x7f73c8199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f73b800ea60 tx=0x7f73b800ef20 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 shutdown_connections 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ee0 0x7f73b00a63a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c819a010 0x7f73c81a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8106170 0x7f73c8199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 --2- 172.21.15.87:0/534951969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c81010f0 0x7f73c8199540 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 >> 172.21.15.87:0/534951969 conn(0x7f73c8077fd0 msgr2=0x7f73c80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 shutdown_connections 2024-04-22T13:31:59.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.419+0000 7f73ce4b3640 1 -- 172.21.15.87:0/534951969 wait complete. 2024-04-22T13:31:59.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 -- 172.21.15.87:0/991188631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340102b70 msgr2=0x7fb340106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 --2- 172.21.15.87:0/991188631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340102b70 0x7fb340106bb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb330009900 tx=0x7fb33002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 -- 172.21.15.87:0/991188631 shutdown_connections 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 --2- 172.21.15.87:0/991188631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb340107290 0x7fb34010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 --2- 172.21.15.87:0/991188631 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340102b70 0x7fb340106bb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 --2- 172.21.15.87:0/991188631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 -- 172.21.15.87:0/991188631 >> 172.21.15.87:0/991188631 conn(0x7fb3400fda20 msgr2=0x7fb3400ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 -- 172.21.15.87:0/991188631 shutdown_connections 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.572+0000 7fb3484a5640 1 -- 172.21.15.87:0/991188631 wait complete. 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 Processor -- start 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 -- start start 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb34019cf90 0x7fb34019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb340112230 con 0x7fb340107290 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401120b0 con 0x7fb3401021c0 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb3484a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3401123b0 con 0x7fb34019cf90 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37832/0 (socket says 172.21.15.87:37832) 2024-04-22T13:31:59.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 -- 172.21.15.87:0/3133799718 learned_addr learned my addr 172.21.15.87:0/3133799718 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb346241640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 -- 172.21.15.87:0/3133799718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb34019cf90 msgr2=0x7fb34019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb34019cf90 0x7fb34019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 -- 172.21.15.87:0/3133799718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 msgr2=0x7fb3401986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 -- 172.21.15.87:0/3133799718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34019da80 con 0x7fb340107290 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb346241640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.573+0000 7fb345a40640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb330005ec0 tx=0x7fb33002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330043020 con 0x7fb340107290 2024-04-22T13:31:59.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34019dd10 con 0x7fb340107290 2024-04-22T13:31:59.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3300052b0 con 0x7fb340107290 2024-04-22T13:31:59.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330003d40 con 0x7fb340107290 2024-04-22T13:31:59.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3401a5560 con 0x7fb340107290 2024-04-22T13:31:59.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.574+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb34004fce0 con 0x7fb340107290 2024-04-22T13:31:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.576+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb330046020 con 0x7fb340107290 2024-04-22T13:31:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.576+0000 7fb3377fe640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 0x7fb32c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.576+0000 7fb346241640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 0x7fb32c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.576+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb3300f33a0 con 0x7fb340107290 2024-04-22T13:31:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.577+0000 7fb346241640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 0x7fb32c0a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0058c0 tx=0x7fb33c005d30 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.580+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb33004d020 con 0x7fb340107290 2024-04-22T13:31:59.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.782+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb340102b70 con 0x7fb32c0a3ee0 2024-04-22T13:31:59.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.846+0000 7fb3377fe640 1 -- 172.21.15.87:0/3133799718 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb340102b70 con 0x7fb32c0a3ee0 2024-04-22T13:31:59.842 INFO:teuthology.orchestra.run.smithi087.stdout:Retention added to path / 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 msgr2=0x7fb32c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 0x7fb32c0a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0058c0 tx=0x7fb33c005d30 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 msgr2=0x7fb340198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb330005ec0 tx=0x7fb33002ca10 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 shutdown_connections 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb32c0a3ee0 0x7fb32c0a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb34019cf90 0x7fb34019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb340107290 0x7fb340198c10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 --2- 172.21.15.87:0/3133799718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3401021c0 0x7fb3401986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 >> 172.21.15.87:0/3133799718 conn(0x7fb3400fda20 msgr2=0x7fb3400fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 shutdown_connections 2024-04-22T13:31:59.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.848+0000 7fb3484a5640 1 -- 172.21.15.87:0/3133799718 wait complete. 2024-04-22T13:31:59.855 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T13:31:59.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/4065315896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 msgr2=0x7fb7e406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/4065315896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e406ae10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009900 tx=0x7fb7e002bc30 comp rx=0 tx=0).stop 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/4065315896 shutdown_connections 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/4065315896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e41072e0 0x7fb7e410ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/4065315896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e406b3e0 0x7fb7e4106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/4065315896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e406ae10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/4065315896 >> 172.21.15.87:0/4065315896 conn(0x7fb7e4077ba0 msgr2=0x7fb7e4077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/4065315896 shutdown_connections 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/4065315896 wait complete. 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 Processor -- start 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.959+0000 7fb7eb6c8640 1 -- start start 2024-04-22T13:31:59.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 0x7fb7e4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 0x7fb7e419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e4112220 con 0x7fb7e406aa30 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41120a0 con 0x7fb7e41072e0 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41123a0 con 0x7fb7e419cfa0 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37844/0 (socket says 172.21.15.87:37844) 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 -- 172.21.15.87:0/2383731870 learned_addr learned my addr 172.21.15.87:0/2383731870 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e8c63640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 0x7fb7e4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9c65640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 0x7fb7e419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 -- 172.21.15.87:0/2383731870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 msgr2=0x7fb7e419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 0x7fb7e419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 -- 172.21.15.87:0/2383731870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 msgr2=0x7fb7e4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 0x7fb7e4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 -- 172.21.15.87:0/2383731870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e419da90 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9c65640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 0x7fb7e419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e8c63640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 0x7fb7e4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7e9464640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0005940 tx=0x7fb7e00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0043020 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7e002caf0 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e419dd20 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0003af0 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.960+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e41a5540 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.961+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e4103690 con 0x7fb7e406aa30 2024-04-22T13:31:59.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.962+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb7e0046020 con 0x7fb7e406aa30 2024-04-22T13:31:59.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.963+0000 7fb7da7fc640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 0x7fb7d00a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:31:59.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.963+0000 7fb7e8c63640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 0x7fb7d00a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:31:59.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.963+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb7e00f2b80 con 0x7fb7e406aa30 2024-04-22T13:31:59.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.963+0000 7fb7e8c63640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 0x7fb7d00a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40058c0 tx=0x7fb7d4005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:31:59.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:31:59.967+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7e004d020 con 0x7fb7e406aa30 2024-04-22T13:32:00.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.168+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fb7e406b3e0 con 0x7fb7d00a3ee0 2024-04-22T13:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.176+0000 7fb7da7fc640 1 -- 172.21.15.87:0/2383731870 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fb7e406b3e0 con 0x7fb7d00a3ee0 2024-04-22T13:32:00.171 INFO:teuthology.orchestra.run.smithi087.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T13:31:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 msgr2=0x7fb7d00a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 0x7fb7d00a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40058c0 tx=0x7fb7d4005850 comp rx=0 tx=0).stop 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 msgr2=0x7fb7e41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0005940 tx=0x7fb7e00045e0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 shutdown_connections 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb7d00a3ee0 0x7fb7d00a63a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7e419cfa0 0x7fb7e419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e41072e0 0x7fb7e4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 --2- 172.21.15.87:0/2383731870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7e406aa30 0x7fb7e41986e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 >> 172.21.15.87:0/2383731870 conn(0x7fb7e4077ba0 msgr2=0x7fb7e40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 shutdown_connections 2024-04-22T13:32:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.178+0000 7fb7eb6c8640 1 -- 172.21.15.87:0/2383731870 wait complete. 2024-04-22T13:32:00.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-22T13:32:00.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3766036266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:00.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:32:00.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.184 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:32:00.184 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:31:59 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T13:32:00.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-22T13:32:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3766036266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:32:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:32:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:31:59 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/2598084406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 msgr2=0x7f8bbc1083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/2598084406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc1083c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009930 tx=0x7f8bac02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/2598084406 shutdown_connections 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/2598084406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc10edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/2598084406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 0x7f8bbc06abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/2598084406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc1083c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/2598084406 >> 172.21.15.87:0/2598084406 conn(0x7f8bbc076c50 msgr2=0x7f8bbc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/2598084406 shutdown_connections 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.289+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/2598084406 wait complete. 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 Processor -- start 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 -- start start 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 0x7f8bbc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112170 con 0x7f8bbc107fe0 2024-04-22T13:32:00.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc111ff0 con 0x7f8bbc06a780 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bc0cc1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1122f0 con 0x7f8bbc06b120 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bbad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bbad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37860/0 (socket says 172.21.15.87:37860) 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.290+0000 7f8bbad9d640 1 -- 172.21.15.87:0/1327698212 learned_addr learned my addr 172.21.15.87:0/1327698212 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bba59c640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 0x7f8bbc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bb3fff640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 -- 172.21.15.87:0/1327698212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 msgr2=0x7f8bbc198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 -- 172.21.15.87:0/1327698212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 msgr2=0x7f8bbc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 0x7f8bbc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 -- 172.21.15.87:0/1327698212 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc19d650 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bb3fff640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bbad9d640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400cce0 tx=0x7f8bb400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb400f450 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bb400faa0 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc19d940 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4018b70 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.291+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc1a51d0 con 0x7f8bbc107fe0 2024-04-22T13:32:00.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.292+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bbc103680 con 0x7f8bbc107fe0 2024-04-22T13:32:00.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.293+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8bb4018d10 con 0x7f8bbc107fe0 2024-04-22T13:32:00.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.294+0000 7f8bb37fe640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 0x7f8ba80a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.294+0000 7f8bba59c640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 0x7f8ba80a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.294+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8bb40d0f30 con 0x7f8bbc107fe0 2024-04-22T13:32:00.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.294+0000 7f8bba59c640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 0x7f8ba80a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac004120 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:00.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.298+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bb4014030 con 0x7f8bbc107fe0 2024-04-22T13:32:00.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-22T13:32:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3766036266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:32:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:32:00.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:31:59 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:00.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.494+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8bbc062390 con 0x7f8ba80a3ee0 2024-04-22T13:32:00.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.501+0000 7f8bb37fe640 1 -- 172.21.15.87:0/1327698212 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8bbc062390 con 0x7f8ba80a3ee0 2024-04-22T13:32:00.497 INFO:teuthology.orchestra.run.smithi087.stdout:/ 1m 6m3h 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 msgr2=0x7f8ba80a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 0x7f8ba80a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac004120 comp rx=0 tx=0).stop 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 msgr2=0x7f8bbc19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400cce0 tx=0x7f8bb400a620 comp rx=0 tx=0).stop 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 shutdown_connections 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ba80a3ee0 0x7f8ba80a63a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8bbc107fe0 0x7f8bbc19cf20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bbc06b120 0x7f8bbc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 --2- 172.21.15.87:0/1327698212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bbc06a780 0x7f8bbc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 >> 172.21.15.87:0/1327698212 conn(0x7f8bbc076c50 msgr2=0x7f8bbc0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:00.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 shutdown_connections 2024-04-22T13:32:00.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.503+0000 7f8bc0cc1640 1 -- 172.21.15.87:0/1327698212 wait complete. 2024-04-22T13:32:00.510 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-22T13:32:00.535 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-22T13:32:00.547 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T13:32:00.565 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:32:00.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T13:32:00.578 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-22T13:32:00.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:32:00.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- 172.21.15.87:0/896658726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 msgr2=0x7f57b4101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/896658726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b4101620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f57a8009930 tx=0x7f57a802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- 172.21.15.87:0/896658726 shutdown_connections 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/896658726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b4101b60 0x7f57b410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/896658726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b4101620 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/896658726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 0x7f57b4106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- 172.21.15.87:0/896658726 >> 172.21.15.87:0/896658726 conn(0x7f57b4077e30 msgr2=0x7f57b4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- 172.21.15.87:0/896658726 shutdown_connections 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- 172.21.15.87:0/896658726 wait complete. 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 Processor -- start 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.694+0000 7f57ba2c8640 1 -- start start 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b4101b60 0x7f57b4199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 0x7f57b41a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f57b41110c0 con 0x7f57b41011a0 2024-04-22T13:32:00.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57b4110f40 con 0x7f57b4101b60 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57b4111240 con 0x7f57b4105e30 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37876/0 (socket says 172.21.15.87:37876) 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 -- 172.21.15.87:0/3726633238 learned_addr learned my addr 172.21.15.87:0/3726633238 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b8865640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 0x7f57b41a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 -- 172.21.15.87:0/3726633238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 msgr2=0x7f57b41a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 0x7f57b41a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 -- 172.21.15.87:0/3726633238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b4101b60 msgr2=0x7f57b4199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b4101b60 0x7f57b4199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 -- 172.21.15.87:0/3726633238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b41a4ba0 con 0x7f57b41011a0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b3fff640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c890 tx=0x7f57a400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4004220 con 0x7f57b41011a0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a40043c0 con 0x7f57b41011a0 2024-04-22T13:32:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b41a4d70 con 0x7f57b41011a0 2024-04-22T13:32:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4004a00 con 0x7f57b41011a0 2024-04-22T13:32:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.695+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b41a51d0 con 0x7f57b41011a0 2024-04-22T13:32:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.696+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b4068de0 con 0x7f57b41011a0 2024-04-22T13:32:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.697+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f57a40079b0 con 0x7f57b41011a0 2024-04-22T13:32:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.698+0000 7f57b17fa640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 0x7f579c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.698+0000 7f57b37fe640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 0x7f579c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.698+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f57a40d07d0 con 0x7f57b41011a0 2024-04-22T13:32:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.699+0000 7f57b37fe640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 0x7f579c0a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57a80096c0 tx=0x7f57a8000bc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:00.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.703+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57a4014030 con 0x7f57b41011a0 2024-04-22T13:32:00.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.886+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57b402d5c0 con 0x7f57b41011a0 2024-04-22T13:32:00.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.888+0000 7f57b17fa640 1 -- 172.21.15.87:0/3726633238 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7f57a409cc90 con 0x7f57b41011a0 2024-04-22T13:32:00.884 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:00.884 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 msgr2=0x7f579c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 0x7f579c0a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57a80096c0 tx=0x7f57a8000bc0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 msgr2=0x7f57b41995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f57a400c890 tx=0x7f57a400cd50 comp rx=0 tx=0).stop 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 shutdown_connections 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f579c0a3e40 0x7f579c0a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57b4105e30 0x7f57b41a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57b4101b60 0x7f57b4199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 --2- 172.21.15.87:0/3726633238 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57b41011a0 0x7f57b41995e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:00.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 >> 172.21.15.87:0/3726633238 conn(0x7f57b4077e30 msgr2=0x7f57b40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:00.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 shutdown_connections 2024-04-22T13:32:00.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:00.890+0000 7f57ba2c8640 1 -- 172.21.15.87:0/3726633238 wait complete. 2024-04-22T13:32:00.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:32:00.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/1890753044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c1072b0 msgr2=0x7fc09c10eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/1890753044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c1072b0 0x7fc09c10eec0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc094009ca0 tx=0x7fc09402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/1890753044 shutdown_connections 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/1890753044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c1072b0 0x7fc09c10eec0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/1890753044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c1014a0 0x7fc09c106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/1890753044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c100af0 0x7fc09c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/1890753044 >> 172.21.15.87:0/1890753044 conn(0x7fc09c0760e0 msgr2=0x7fc09c0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/1890753044 shutdown_connections 2024-04-22T13:32:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.007+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/1890753044 wait complete. 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 Processor -- start 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 -- start start 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c100af0 0x7fc09c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 0x7fc09c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc09c112250 con 0x7fc09c19d010 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc09c1120d0 con 0x7fc09c1072b0 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc0a0ab3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc09c1123d0 con 0x7fc09c100af0 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37896/0 (socket says 172.21.15.87:37896) 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 -- 172.21.15.87:0/2534010634 learned_addr learned my addr 172.21.15.87:0/2534010634 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09a59c640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c100af0 0x7fc09c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc093fff640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 0x7fc09c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 -- 172.21.15.87:0/2534010634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c100af0 msgr2=0x7fc09c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c100af0 0x7fc09c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 -- 172.21.15.87:0/2534010634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 msgr2=0x7fc09c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 0x7fc09c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc09ad9d640 1 -- 172.21.15.87:0/2534010634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09c19db00 con 0x7fc09c19d010 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.008+0000 7fc093fff640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 0x7fc09c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc09ad9d640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc094005e30 tx=0x7fc094034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc094039040 con 0x7fc09c19d010 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc094044070 con 0x7fc09c19d010 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09c19dd90 con 0x7fc09c19d010 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc094003e60 con 0x7fc09c19d010 2024-04-22T13:32:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.009+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09c1a5550 con 0x7fc09c19d010 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.010+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc09c0691c0 con 0x7fc09c19d010 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.011+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc0940324c0 con 0x7fc09c19d010 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.011+0000 7fc0937fe640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 0x7fc0880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.012+0000 7fc09a59c640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 0x7fc0880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.012+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc0940f29c0 con 0x7fc09c19d010 2024-04-22T13:32:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.012+0000 7fc09a59c640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 0x7fc0880a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00bbe0 tx=0x7fc08c00b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.016+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0940f4e20 con 0x7fc09c19d010 2024-04-22T13:32:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.195+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc09c1014a0 con 0x7fc09c19d010 2024-04-22T13:32:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.196+0000 7fc0937fe640 1 -- 172.21.15.87:0/2534010634 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fc09404e020 con 0x7fc09c19d010 2024-04-22T13:32:01.192 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:32:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 msgr2=0x7fc0880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 0x7fc0880a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00bbe0 tx=0x7fc08c00b390 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 msgr2=0x7fc09c19d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc094005e30 tx=0x7fc094034760 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 shutdown_connections 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc0880a3e40 0x7fc0880a6300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc09c19d010 0x7fc09c19d4c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.197+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c1072b0 0x7fc09c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.198+0000 7fc0a0ab3640 1 --2- 172.21.15.87:0/2534010634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc09c100af0 0x7fc09c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.198+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 >> 172.21.15.87:0/2534010634 conn(0x7fc09c0760e0 msgr2=0x7fc09c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.198+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 shutdown_connections 2024-04-22T13:32:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.198+0000 7fc0a0ab3640 1 -- 172.21.15.87:0/2534010634 wait complete. 2024-04-22T13:32:01.203 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-mon[27999]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-mon[27999]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3726633238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:32:00.855+0000 7f06ed1f0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:32:00.855+0000 7f06ed1f0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:32:00.855+0000 7f06ed1f0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:32:00.855+0000 7f06ed1f0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:32:01.204 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:32:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:32:00.855+0000 7f06ed1f0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T13:32:01.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:32:01.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:00 smithi132 ceph-mon[31531]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:00 smithi132 ceph-mon[31531]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:00 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:32:01.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:00 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3726633238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 -- 172.21.15.87:0/3403295031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68102190 msgr2=0x7fda68102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 --2- 172.21.15.87:0/3403295031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68102190 0x7fda68102570 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fda58009900 tx=0x7fda5802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 -- 172.21.15.87:0/3403295031 shutdown_connections 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 --2- 172.21.15.87:0/3403295031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68107260 0x7fda6810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 --2- 172.21.15.87:0/3403295031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda68102b40 0x7fda68106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 --2- 172.21.15.87:0/3403295031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68102190 0x7fda68102570 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 -- 172.21.15.87:0/3403295031 >> 172.21.15.87:0/3403295031 conn(0x7fda680793a0 msgr2=0x7fda680797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 -- 172.21.15.87:0/3403295031 shutdown_connections 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.311+0000 7fda6ebef640 1 -- 172.21.15.87:0/3403295031 wait complete. 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 Processor -- start 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 -- start start 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 0x7fda6819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda681985a0 0x7fda68198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda681121c0 con 0x7fda68107260 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda68112040 con 0x7fda68102190 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6ebef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda68112340 con 0x7fda681985a0 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37912/0 (socket says 172.21.15.87:37912) 2024-04-22T13:32:01.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 -- 172.21.15.87:0/4010334174 learned_addr learned my addr 172.21.15.87:0/4010334174 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6c98b640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 0x7fda6819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6d18c640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda681985a0 0x7fda68198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 -- 172.21.15.87:0/4010334174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda681985a0 msgr2=0x7fda68198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda681985a0 0x7fda68198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 -- 172.21.15.87:0/4010334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 msgr2=0x7fda6819e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 0x7fda6819e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 -- 172.21.15.87:0/4010334174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda681990c0 con 0x7fda68107260 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda6c98b640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 0x7fda6819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda67fff640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c970 tx=0x7fda5000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50007bb0 con 0x7fda68107260 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda50007d50 con 0x7fda68107260 2024-04-22T13:32:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.312+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50018730 con 0x7fda68107260 2024-04-22T13:32:01.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.313+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda681993b0 con 0x7fda68107260 2024-04-22T13:32:01.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.313+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda681a5600 con 0x7fda68107260 2024-04-22T13:32:01.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.314+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda680691c0 con 0x7fda68107260 2024-04-22T13:32:01.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.315+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fda5000f990 con 0x7fda68107260 2024-04-22T13:32:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.315+0000 7fda65ffb640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 0x7fda540a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.315+0000 7fda6c98b640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 0x7fda540a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.316+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fda500d0b30 con 0x7fda68107260 2024-04-22T13:32:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.316+0000 7fda6c98b640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 0x7fda540a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda58005ab0 tx=0x7fda58005a20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.320+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda50014030 con 0x7fda68107260 2024-04-22T13:32:01.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:00 smithi120 ceph-mon[31101]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:00 smithi120 ceph-mon[31101]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:01.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:00 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:32:01.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:00 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3726633238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:01.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.497+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fda6810a4f0 con 0x7fda68107260 2024-04-22T13:32:01.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.497+0000 7fda65ffb640 1 -- 172.21.15.87:0/4010334174 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fda5009c9a0 con 0x7fda68107260 2024-04-22T13:32:01.493 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:32:01.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.499+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 msgr2=0x7fda540a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.499+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 0x7fda540a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda58005ab0 tx=0x7fda58005a20 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.499+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 msgr2=0x7fda6819e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.499+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c970 tx=0x7fda5000ce30 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 shutdown_connections 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda540a3ec0 0x7fda540a6380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda681985a0 0x7fda68198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda68107260 0x7fda6819e930 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 --2- 172.21.15.87:0/4010334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda68102190 0x7fda6819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 >> 172.21.15.87:0/4010334174 conn(0x7fda680793a0 msgr2=0x7fda68100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 shutdown_connections 2024-04-22T13:32:01.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.500+0000 7fda6ebef640 1 -- 172.21.15.87:0/4010334174 wait complete. 2024-04-22T13:32:01.507 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:32:01.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 -- 172.21.15.87:0/2914741243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5130101090 msgr2=0x7f5130101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 --2- 172.21.15.87:0/2914741243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5130101090 0x7f5130101470 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5120009900 tx=0x7f512002bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 -- 172.21.15.87:0/2914741243 shutdown_connections 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 --2- 172.21.15.87:0/2914741243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f513010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 --2- 172.21.15.87:0/2914741243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101a40 0x7f5130105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 --2- 172.21.15.87:0/2914741243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5130101090 0x7f5130101470 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.613+0000 7f51361ef640 1 -- 172.21.15.87:0/2914741243 >> 172.21.15.87:0/2914741243 conn(0x7f5130077f50 msgr2=0x7f5130078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- 172.21.15.87:0/2914741243 shutdown_connections 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- 172.21.15.87:0/2914741243 wait complete. 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 Processor -- start 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- start start 2024-04-22T13:32:01.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101090 0x7f51301995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f5130199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51301110c0 con 0x7f513019a0b0 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5130110f40 con 0x7f5130106160 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f51361ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5130111240 con 0x7f5130101090 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f512ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f512ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37934/0 (socket says 172.21.15.87:37934) 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.614+0000 7f512ffff640 1 -- 172.21.15.87:0/3791061708 learned_addr learned my addr 172.21.15.87:0/3791061708 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512f7fe640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101090 0x7f51301995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f5127fff640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f5130199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 -- 172.21.15.87:0/3791061708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101090 msgr2=0x7f51301995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101090 0x7f51301995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 -- 172.21.15.87:0/3791061708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 msgr2=0x7f5130199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f5130199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 -- 172.21.15.87:0/3791061708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51301a4e30 con 0x7f513019a0b0 2024-04-22T13:32:01.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f5127fff640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f5130199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512ffff640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f512800eb10 tx=0x7f512800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128004b20 con 0x7f513019a0b0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5128004cc0 con 0x7f513019a0b0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51301a5120 con 0x7f513019a0b0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128010670 con 0x7f513019a0b0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.615+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51301a5620 con 0x7f513019a0b0 2024-04-22T13:32:01.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.616+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5130068d70 con 0x7f513019a0b0 2024-04-22T13:32:01.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.617+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5128005020 con 0x7f513019a0b0 2024-04-22T13:32:01.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.618+0000 7f512d7fa640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 0x7f511c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.618+0000 7f512f7fe640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 0x7f511c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.618+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f51280d10b0 con 0x7f513019a0b0 2024-04-22T13:32:01.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.618+0000 7f512f7fe640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 0x7f511c0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5120000c00 tx=0x7f5120004040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.622+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f512809d570 con 0x7f513019a0b0 2024-04-22T13:32:01.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.803+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5130101a40 con 0x7f513019a0b0 2024-04-22T13:32:01.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.803+0000 7f512d7fa640 1 -- 172.21.15.87:0/3791061708 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f512809ce70 con 0x7f513019a0b0 2024-04-22T13:32:01.799 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 msgr2=0x7f511c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 0x7f511c0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5120000c00 tx=0x7f5120004040 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 msgr2=0x7f51301a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f512800eb10 tx=0x7f512800c2d0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 shutdown_connections 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f511c0a3e40 0x7f511c0a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f513019a0b0 0x7f51301a4660 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5130106160 0x7f5130199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 --2- 172.21.15.87:0/3791061708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5130101090 0x7f51301995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 >> 172.21.15.87:0/3791061708 conn(0x7f5130077f50 msgr2=0x7f51300ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 shutdown_connections 2024-04-22T13:32:01.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.805+0000 7f51361ef640 1 -- 172.21.15.87:0/3791061708 wait complete. 2024-04-22T13:32:01.811 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:32:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- 172.21.15.87:0/4073265130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 msgr2=0x7f9db4106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 --2- 172.21.15.87:0/4073265130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4106210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009930 tx=0x7f9da402bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- 172.21.15.87:0/4073265130 shutdown_connections 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 --2- 172.21.15.87:0/4073265130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db4101b60 0x7f9db410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 --2- 172.21.15.87:0/4073265130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db41011a0 0x7f9db4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 --2- 172.21.15.87:0/4073265130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4106210 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- 172.21.15.87:0/4073265130 >> 172.21.15.87:0/4073265130 conn(0x7f9db4077e30 msgr2=0x7f9db4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- 172.21.15.87:0/4073265130 shutdown_connections 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- 172.21.15.87:0/4073265130 wait complete. 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 Processor -- start 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.915+0000 7f9db94e0640 1 -- start start 2024-04-22T13:32:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db41011a0 0x7f9db41973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db4101b60 0x7f9db4197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9db4111140 con 0x7f9db4105e30 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9db4110fc0 con 0x7f9db41011a0 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9db41112c0 con 0x7f9db4101b60 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37952/0 (socket says 172.21.15.87:37952) 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 -- 172.21.15.87:0/71177339 learned_addr learned my addr 172.21.15.87:0/71177339 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db27fc640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db4101b60 0x7f9db4197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db2ffd640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db41011a0 0x7f9db41973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 -- 172.21.15.87:0/71177339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db4101b60 msgr2=0x7f9db4197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db4101b60 0x7f9db4197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 -- 172.21.15.87:0/71177339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db41011a0 msgr2=0x7f9db41973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db41011a0 0x7f9db41973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 -- 172.21.15.87:0/71177339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db41a6c60 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db37fe640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00eaa0 tx=0x7f9dac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac004d20 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db41a6f50 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dac005020 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac0116b0 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.916+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db41a7420 con 0x7f9db4105e30 2024-04-22T13:32:01.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.917+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db4102570 con 0x7f9db4105e30 2024-04-22T13:32:01.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.919+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9dac0051d0 con 0x7f9db4105e30 2024-04-22T13:32:01.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.919+0000 7f9dabfff640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 0x7f9da00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:01.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.919+0000 7f9db2ffd640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 0x7f9da00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:01.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.919+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9dac0d0f30 con 0x7f9db4105e30 2024-04-22T13:32:01.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.919+0000 7f9db2ffd640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 0x7f9da00a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9da4005ab0 tx=0x7f9da4005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:01.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:01.923+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dac016030 con 0x7f9db4105e30 2024-04-22T13:32:02.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.103+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9db4068de0 con 0x7f9db4105e30 2024-04-22T13:32:02.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.104+0000 7f9dabfff640 1 -- 172.21.15.87:0/71177339 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7f9db4068de0 con 0x7f9db4105e30 2024-04-22T13:32:02.100 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:02.100 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:32:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 msgr2=0x7f9da00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 0x7f9da00a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9da4005ab0 tx=0x7f9da4005fd0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 msgr2=0x7f9db4197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00eaa0 tx=0x7f9dac00eea0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 shutdown_connections 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9da00a3e40 0x7f9da00a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9db4105e30 0x7f9db4197e70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9db4101b60 0x7f9db4197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 --2- 172.21.15.87:0/71177339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db41011a0 0x7f9db41973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 >> 172.21.15.87:0/71177339 conn(0x7f9db4077e30 msgr2=0x7f9db40fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 shutdown_connections 2024-04-22T13:32:02.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.106+0000 7f9db94e0640 1 -- 172.21.15.87:0/71177339 wait complete. 2024-04-22T13:32:02.112 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T13:32:02.112 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.112 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.112 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2534010634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.113 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4010334174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.113 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3791061708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.114 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:32:02.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:32:02.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T13:32:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2534010634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4010334174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3791061708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.219+0000 7f36b4fab640 1 -- 172.21.15.87:0/2484844387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 msgr2=0x7f36b010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.219+0000 7f36b4fab640 1 --2- 172.21.15.87:0/2484844387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b010e9b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009900 tx=0x7f36a802bd60 comp rx=0 tx=0).stop 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- 172.21.15.87:0/2484844387 shutdown_connections 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- 172.21.15.87:0/2484844387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b010e9b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- 172.21.15.87:0/2484844387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b0103af0 0x7f36b010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- 172.21.15.87:0/2484844387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b01031d0 0x7f36b01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- 172.21.15.87:0/2484844387 >> 172.21.15.87:0/2484844387 conn(0x7f36b0077f50 msgr2=0x7f36b0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- 172.21.15.87:0/2484844387 shutdown_connections 2024-04-22T13:32:02.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- 172.21.15.87:0/2484844387 wait complete. 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 Processor -- start 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- start start 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 0x7f36b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 0x7f36b0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f36b01121a0 con 0x7f36b010c510 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36b0112020 con 0x7f36b0103af0 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.220+0000 7f36b4fab640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f36b0112320 con 0x7f36b01031d0 2024-04-22T13:32:02.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37964/0 (socket says 172.21.15.87:37964) 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 -- 172.21.15.87:0/3437445903 learned_addr learned my addr 172.21.15.87:0/3437445903 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36ae59c640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 0x7f36b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36add9b640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 0x7f36b0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 -- 172.21.15.87:0/3437445903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 msgr2=0x7f36b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 0x7f36b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 -- 172.21.15.87:0/3437445903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 msgr2=0x7f36b0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 0x7f36b0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 -- 172.21.15.87:0/3437445903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b019d6d0 con 0x7f36b010c510 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36add9b640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 0x7f36b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36ae59c640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 0x7f36b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36aed9d640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009230 tx=0x7f36a80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8043020 con 0x7f36b010c510 2024-04-22T13:32:02.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a802dcb0 con 0x7f36b010c510 2024-04-22T13:32:02.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b019d960 con 0x7f36b010c510 2024-04-22T13:32:02.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8003a90 con 0x7f36b010c510 2024-04-22T13:32:02.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.221+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b01a5250 con 0x7f36b010c510 2024-04-22T13:32:02.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.222+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b0068d70 con 0x7f36b010c510 2024-04-22T13:32:02.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.223+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f36a8046020 con 0x7f36b010c510 2024-04-22T13:32:02.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.224+0000 7f369f7fe640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 0x7f36980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.224+0000 7f36ae59c640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 0x7f36980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.224+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f36a80f3000 con 0x7f36b010c510 2024-04-22T13:32:02.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.224+0000 7f36ae59c640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 0x7f36980a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36a0005fd0 tx=0x7f36a0009970 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.227+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36a804d020 con 0x7f36b010c510 2024-04-22T13:32:02.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.407+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36b0107690 con 0x7f36b010c510 2024-04-22T13:32:02.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.408+0000 7f369f7fe640 1 -- 172.21.15.87:0/3437445903 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f36a80bf5d0 con 0x7f36b010c510 2024-04-22T13:32:02.404 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 msgr2=0x7f36980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 0x7f36980a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f36a0005fd0 tx=0x7f36a0009970 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 msgr2=0x7f36b019cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009230 tx=0x7f36a80045e0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 shutdown_connections 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36980a3e40 0x7f36980a6300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36b010c510 0x7f36b019cfa0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36b0103af0 0x7f36b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 --2- 172.21.15.87:0/3437445903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36b01031d0 0x7f36b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 >> 172.21.15.87:0/3437445903 conn(0x7f36b0077f50 msgr2=0x7f36b00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 shutdown_connections 2024-04-22T13:32:02.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.410+0000 7f36b4fab640 1 -- 172.21.15.87:0/3437445903 wait complete. 2024-04-22T13:32:02.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2534010634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4010334174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3791061708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 -- 172.21.15.87:0/3455639031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f439406b1d0 msgr2=0x7f4394106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 --2- 172.21.15.87:0/3455639031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f439406b1d0 0x7f4394106b90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4380009900 tx=0x7f438002bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 -- 172.21.15.87:0/3455639031 shutdown_connections 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 --2- 172.21.15.87:0/3455639031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4394107270 0x7f439410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 --2- 172.21.15.87:0/3455639031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f439406b1d0 0x7f4394106b90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 --2- 172.21.15.87:0/3455639031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439406a820 0x7f439406ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 -- 172.21.15.87:0/3455639031 >> 172.21.15.87:0/3455639031 conn(0x7f4394077eb0 msgr2=0x7f43940782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 -- 172.21.15.87:0/3455639031 shutdown_connections 2024-04-22T13:32:02.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 -- 172.21.15.87:0/3455639031 wait complete. 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.522+0000 7f439ae70640 1 Processor -- start 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 -- start start 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 0x7f4394198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439419cff0 0x7f439419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f43941121b0 con 0x7f4394107270 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4394112030 con 0x7f439406a820 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4394112330 con 0x7f439419cff0 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37968/0 (socket says 172.21.15.87:37968) 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 -- 172.21.15.87:0/2451720779 learned_addr learned my addr 172.21.15.87:0/2451720779 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439940d640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439419cff0 0x7f439419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4398c0c640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 0x7f4394198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 -- 172.21.15.87:0/2451720779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439419cff0 msgr2=0x7f439419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439419cff0 0x7f439419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 -- 172.21.15.87:0/2451720779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 msgr2=0x7f4394198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 0x7f4394198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 -- 172.21.15.87:0/2451720779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439419dae0 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4398c0c640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 0x7f4394198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4393fff640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4380036040 tx=0x7f43800045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380043020 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f438002cb70 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439419dd70 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.523+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380003bf0 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.524+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43941a5590 con 0x7f4394107270 2024-04-22T13:32:02.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.525+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43940691c0 con 0x7f4394107270 2024-04-22T13:32:02.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.526+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f4380046020 con 0x7f4394107270 2024-04-22T13:32:02.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.527+0000 7f4391ffb640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 0x7f43840a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.527+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f43800f2f20 con 0x7f4394107270 2024-04-22T13:32:02.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.527+0000 7f4398c0c640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 0x7f43840a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.527+0000 7f4398c0c640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 0x7f43840a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4388009c20 tx=0x7f4388005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.530+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f438004d020 con 0x7f4394107270 2024-04-22T13:32:02.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.707+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f439410a4f0 con 0x7f4394107270 2024-04-22T13:32:02.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.708+0000 7f4391ffb640 1 -- 172.21.15.87:0/2451720779 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f43800bf4f0 con 0x7f4394107270 2024-04-22T13:32:02.704 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:32:02.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 msgr2=0x7f43840a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 0x7f43840a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4388009c20 tx=0x7f4388005f90 comp rx=0 tx=0).stop 2024-04-22T13:32:02.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 msgr2=0x7f4394198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4380036040 tx=0x7f43800045e0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 shutdown_connections 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43840a3e40 0x7f43840a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439419cff0 0x7f439419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4394107270 0x7f4394198c70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 --2- 172.21.15.87:0/2451720779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f439406a820 0x7f4394198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 >> 172.21.15.87:0/2451720779 conn(0x7f4394077eb0 msgr2=0x7f43940ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 shutdown_connections 2024-04-22T13:32:02.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.710+0000 7f439ae70640 1 -- 172.21.15.87:0/2451720779 wait complete. 2024-04-22T13:32:02.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- 172.21.15.87:0/545933734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c641010a0 msgr2=0x7f6c64101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 --2- 172.21.15.87:0/545933734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c641010a0 0x7f6c64101480 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009900 tx=0x7f6c5402bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- 172.21.15.87:0/545933734 shutdown_connections 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 --2- 172.21.15.87:0/545933734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c64106170 0x7f6c6410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 --2- 172.21.15.87:0/545933734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c64101a50 0x7f6c64105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 --2- 172.21.15.87:0/545933734 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c641010a0 0x7f6c64101480 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- 172.21.15.87:0/545933734 >> 172.21.15.87:0/545933734 conn(0x7f6c64077eb0 msgr2=0x7f6c640782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- 172.21.15.87:0/545933734 shutdown_connections 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- 172.21.15.87:0/545933734 wait complete. 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 Processor -- start 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.823+0000 7f6c6a826640 1 -- start start 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c641010a0 0x7f6c64198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6419cff0 0x7f6c6419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6c641121f0 con 0x7f6c64106170 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64112070 con 0x7f6c6419cff0 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c6a826640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64112370 con 0x7f6c641010a0 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37998/0 (socket says 172.21.15.87:37998) 2024-04-22T13:32:02.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 -- 172.21.15.87:0/329291659 learned_addr learned my addr 172.21.15.87:0/329291659 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c63fff640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c641010a0 0x7f6c64198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c68dc3640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6419cff0 0x7f6c6419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 -- 172.21.15.87:0/329291659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c641010a0 msgr2=0x7f6c64198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c641010a0 0x7f6c64198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 -- 172.21.15.87:0/329291659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6419cff0 msgr2=0x7f6c6419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6419cff0 0x7f6c6419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 -- 172.21.15.87:0/329291659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6419dae0 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.824+0000 7f6c637fe640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d740 tx=0x7f6c5800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.825+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58017070 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.825+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6419ddd0 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.825+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c5800de50 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.825+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004ce0 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.825+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c641a55c0 con 0x7f6c64106170 2024-04-22T13:32:02.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.826+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c640691c0 con 0x7f6c64106170 2024-04-22T13:32:02.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.827+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f6c58004e80 con 0x7f6c64106170 2024-04-22T13:32:02.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.827+0000 7f6c617fa640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 0x7f6c4c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:02.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.827+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6c580d0760 con 0x7f6c64106170 2024-04-22T13:32:02.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.827+0000 7f6c63fff640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 0x7f6c4c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:02.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.828+0000 7f6c63fff640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 0x7f6c4c0a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c54005e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:02.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.831+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c5809c5d0 con 0x7f6c64106170 2024-04-22T13:32:02.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:02.953+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f6c58098c40 con 0x7f6c64106170 2024-04-22T13:32:02.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/71177339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:02.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3437445903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:02.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2451720779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:03.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.013+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c64101a50 con 0x7f6c64106170 2024-04-22T13:32:03.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.013+0000 7f6c617fa640 1 -- 172.21.15.87:0/329291659 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f6c580ca600 con 0x7f6c64106170 2024-04-22T13:32:03.009 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 msgr2=0x7f6c4c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 0x7f6c4c0a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c54005e40 comp rx=0 tx=0).stop 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 msgr2=0x7f6c64198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d740 tx=0x7f6c5800dc00 comp rx=0 tx=0).stop 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 shutdown_connections 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6c4c0a3e40 0x7f6c4c0a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6419cff0 0x7f6c6419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c64106170 0x7f6c64198c70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 --2- 172.21.15.87:0/329291659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c641010a0 0x7f6c64198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 >> 172.21.15.87:0/329291659 conn(0x7f6c64077eb0 msgr2=0x7f6c640ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 shutdown_connections 2024-04-22T13:32:03.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.015+0000 7f6c6a826640 1 -- 172.21.15.87:0/329291659 wait complete. 2024-04-22T13:32:03.022 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-22T13:32:03.022 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:32:03.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/3669728111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 msgr2=0x7fcb6810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/3669728111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb6810dcb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009ca0 tx=0x7fcb6002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/3669728111 shutdown_connections 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/3669728111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb6810dcb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/3669728111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68101130 0x7fcb68101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/3669728111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68106f10 0x7fcb681072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/3669728111 >> 172.21.15.87:0/3669728111 conn(0x7fcb68077e30 msgr2=0x7fcb68078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/3669728111 shutdown_connections 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/3669728111 wait complete. 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.125+0000 7fcb6e4e0640 1 Processor -- start 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 -- start start 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68101130 0x7fcb681995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68106f10 0x7fcb681a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcb681110f0 con 0x7fcb68101ad0 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcb68110f70 con 0x7fcb68106f10 2024-04-22T13:32:03.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6e4e0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcb68111270 con 0x7fcb68101130 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38008/0 (socket says 172.21.15.87:38008) 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 -- 172.21.15.87:0/743557249 learned_addr learned my addr 172.21.15.87:0/743557249 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb67fff640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68101130 0x7fcb681995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb6ca7d640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68106f10 0x7fcb681a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 -- 172.21.15.87:0/743557249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68101130 msgr2=0x7fcb681995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68101130 0x7fcb681995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 -- 172.21.15.87:0/743557249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68106f10 msgr2=0x7fcb681a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68106f10 0x7fcb681a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 -- 172.21.15.87:0/743557249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb681a4b70 con 0x7fcb68101ad0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.126+0000 7fcb677fe640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d950 tx=0x7fcb5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.127+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c017070 con 0x7fcb68101ad0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.127+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb681a4d40 con 0x7fcb68101ad0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.127+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5c009c30 con 0x7fcb68101ad0 2024-04-22T13:32:03.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.127+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00f620 con 0x7fcb68101ad0 2024-04-22T13:32:03.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.127+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb681a51b0 con 0x7fcb68101ad0 2024-04-22T13:32:03.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.128+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb68069230 con 0x7fcb68101ad0 2024-04-22T13:32:03.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.129+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fcb5c010460 con 0x7fcb68101ad0 2024-04-22T13:32:03.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.129+0000 7fcb657fa640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 0x7fcb500a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.129+0000 7fcb67fff640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 0x7fcb500a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.129+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcb5c0d0620 con 0x7fcb68101ad0 2024-04-22T13:32:03.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.130+0000 7fcb67fff640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 0x7fcb500a6390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005fd0 tx=0x7fcb5800b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.134+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb5c009de0 con 0x7fcb68101ad0 2024-04-22T13:32:03.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/71177339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:03.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3437445903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:03.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2451720779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:03.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.309+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb68062390 con 0x7fcb500a3ed0 2024-04-22T13:32:03.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.316+0000 7fcb657fa640 1 -- 172.21.15.87:0/743557249 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258041 (secure 0 0 0) 0x7fcb68062390 con 0x7fcb500a3ed0 2024-04-22T13:32:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:03.316 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:32:03.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 msgr2=0x7fcb500a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 0x7fcb500a6390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005fd0 tx=0x7fcb5800b390 comp rx=0 tx=0).stop 2024-04-22T13:32:03.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 msgr2=0x7fcb68199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d950 tx=0x7fcb5c00de10 comp rx=0 tx=0).stop 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 shutdown_connections 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb500a3ed0 0x7fcb500a6390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68106f10 0x7fcb681a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb68101ad0 0x7fcb68199af0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 --2- 172.21.15.87:0/743557249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb68101130 0x7fcb681995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.319+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 >> 172.21.15.87:0/743557249 conn(0x7fcb68077e30 msgr2=0x7fcb680ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.320+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 shutdown_connections 2024-04-22T13:32:03.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.320+0000 7fcb6e4e0640 1 -- 172.21.15.87:0/743557249 wait complete. 2024-04-22T13:32:03.326 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T13:32:02.034703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189320,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1127,"num_read_kb":886,"num_write":495,"num_write_kb":4381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369348,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572476,"statfs":{"total":1151940427776,"available":1151562215424,"internally_reserved":0,"allocated":12140544,"data_stored":9139488,"data_compressed":223395,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.989953"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":159,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503560+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:56.503560+0000","last_peered":"2024-04-22T13:31:56.503560+0000","last_clean":"2024-04-22T13:31:56.503560+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:56.503560+0000","last_undegraded":"2024-04-22T13:31:56.503560+0000","last_fullsized":"2024-04-22T13:31:56.503560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503056+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:56.503056+0000","last_peered":"2024-04-22T13:31:56.503056+0000","last_clean":"2024-04-22T13:31:56.503056+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:56.503056+0000","last_undegraded":"2024-04-22T13:31:56.503056+0000","last_fullsized":"2024-04-22T13:31:56.503056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'37","reported_seq":325,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:57.528536+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:57.528536+0000","last_peered":"2024-04-22T13:31:57.528536+0000","last_clean":"2024-04-22T13:31:57.528536+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:57.528536+0000","last_undegraded":"2024-04-22T13:31:57.528536+0000","last_fullsized":"2024-04-22T13:31:57.528536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68029,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":267,"num_read_kb":200,"num_write":38,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503870+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:56.503870+0000","last_peered":"2024-04-22T13:31:56.503870+0000","last_clean":"2024-04-22T13:31:56.503870+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:56.503870+0000","last_undegraded":"2024-04-22T13:31:56.503870+0000","last_fullsized":"2024-04-22T13:31:56.503870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'5","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.339239+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.339239+0000","last_peered":"2024-04-22T13:31:52.339239+0000","last_clean":"2024-04-22T13:31:52.339239+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.339239+0000","last_undegraded":"2024-04-22T13:31:52.339239+0000","last_fullsized":"2024-04-22T13:31:52.339239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025909+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025909+0000","last_peered":"2024-04-22T13:31:52.025909+0000","last_clean":"2024-04-22T13:31:52.025909+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025909+0000","last_undegraded":"2024-04-22T13:31:52.025909+0000","last_fullsized":"2024-04-22T13:31:52.025909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025701+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025701+0000","last_peered":"2024-04-22T13:31:52.025701+0000","last_clean":"2024-04-22T13:31:52.025701+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025701+0000","last_undegraded":"2024-04-22T13:31:52.025701+0000","last_fullsized":"2024-04-22T13:31:52.025701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":154,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503404+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:56.503404+0000","last_peered":"2024-04-22T13:31:56.503404+0000","last_clean":"2024-04-22T13:31:56.503404+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:56.503404+0000","last_undegraded":"2024-04-22T13:31:56.503404+0000","last_fullsized":"2024-04-22T13:31:56.503404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:54.141174+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:54.141174+0000","last_peered":"2024-04-22T13:31:54.141174+0000","last_clean":"2024-04-22T13:31:54.141174+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:54.141174+0000","last_undegraded":"2024-04-22T13:31:54.141174+0000","last_fullsized":"2024-04-22T13:31:54.141174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111290+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111290+0000","last_peered":"2024-04-22T13:31:52.111290+0000","last_clean":"2024-04-22T13:31:52.111290+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111290+0000","last_undegraded":"2024-04-22T13:31:52.111290+0000","last_fullsized":"2024-04-22T13:31:52.111290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:53.032318+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:53.032318+0000","last_peered":"2024-04-22T13:31:53.032318+0000","last_clean":"2024-04-22T13:31:53.032318+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:53.032318+0000","last_undegraded":"2024-04-22T13:31:53.032318+0000","last_fullsized":"2024-04-22T13:31:53.032318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112338+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112338+0000","last_peered":"2024-04-22T13:31:52.112338+0000","last_clean":"2024-04-22T13:31:52.112338+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112338+0000","last_undegraded":"2024-04-22T13:31:52.112338+0000","last_fullsized":"2024-04-22T13:31:52.112338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196097+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196097+0000","last_peered":"2024-04-22T13:31:52.196097+0000","last_clean":"2024-04-22T13:31:52.196097+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196097+0000","last_undegraded":"2024-04-22T13:31:52.196097+0000","last_fullsized":"2024-04-22T13:31:52.196097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210877+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210877+0000","last_peered":"2024-04-22T13:31:52.210877+0000","last_clean":"2024-04-22T13:31:52.210877+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210877+0000","last_undegraded":"2024-04-22T13:31:52.210877+0000","last_fullsized":"2024-04-22T13:31:52.210877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209960+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209960+0000","last_peered":"2024-04-22T13:31:52.209960+0000","last_clean":"2024-04-22T13:31:52.209960+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209960+0000","last_undegraded":"2024-04-22T13:31:52.209960+0000","last_fullsized":"2024-04-22T13:31:52.209960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021921+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021921+0000","last_peered":"2024-04-22T13:31:52.021921+0000","last_clean":"2024-04-22T13:31:52.021921+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021921+0000","last_undegraded":"2024-04-22T13:31:52.021921+0000","last_fullsized":"2024-04-22T13:31:52.021921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209941+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209941+0000","last_peered":"2024-04-22T13:31:52.209941+0000","last_clean":"2024-04-22T13:31:52.209941+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209941+0000","last_undegraded":"2024-04-22T13:31:52.209941+0000","last_fullsized":"2024-04-22T13:31:52.209941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210723+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210723+0000","last_peered":"2024-04-22T13:31:52.210723+0000","last_clean":"2024-04-22T13:31:52.210723+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210723+0000","last_undegraded":"2024-04-22T13:31:52.210723+0000","last_fullsized":"2024-04-22T13:31:52.210723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210147+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210147+0000","last_peered":"2024-04-22T13:31:52.210147+0000","last_clean":"2024-04-22T13:31:52.210147+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210147+0000","last_undegraded":"2024-04-22T13:31:52.210147+0000","last_fullsized":"2024-04-22T13:31:52.210147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209999+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209999+0000","last_peered":"2024-04-22T13:31:52.209999+0000","last_clean":"2024-04-22T13:31:52.209999+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209999+0000","last_undegraded":"2024-04-22T13:31:52.209999+0000","last_fullsized":"2024-04-22T13:31:52.209999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209923+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209923+0000","last_peered":"2024-04-22T13:31:52.209923+0000","last_clean":"2024-04-22T13:31:52.209923+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209923+0000","last_undegraded":"2024-04-22T13:31:52.209923+0000","last_fullsized":"2024-04-22T13:31:52.209923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210319+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210319+0000","last_peered":"2024-04-22T13:31:52.210319+0000","last_clean":"2024-04-22T13:31:52.210319+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210319+0000","last_undegraded":"2024-04-22T13:31:52.210319+0000","last_fullsized":"2024-04-22T13:31:52.210319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022549+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022549+0000","last_peered":"2024-04-22T13:31:52.022549+0000","last_clean":"2024-04-22T13:31:52.022549+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022549+0000","last_undegraded":"2024-04-22T13:31:52.022549+0000","last_fullsized":"2024-04-22T13:31:52.022549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598718,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":890,"num_read_kb":556,"num_write":292,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1796154,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257467,"data_compressed":46637,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1058834,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":532323,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":534421,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466419,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188637,"data_compressed":44795,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68029,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70127,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133858,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202688,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:32:03.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:32:03.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/71177339' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:03.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3437445903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:03.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2451720779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 -- 172.21.15.87:0/3333351437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc102190 msgr2=0x7f36bc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/3333351437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc102190 0x7f36bc102570 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009900 tx=0x7f36ac02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 -- 172.21.15.87:0/3333351437 shutdown_connections 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/3333351437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc107260 0x7f36bc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/3333351437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc102b40 0x7f36bc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/3333351437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc102190 0x7f36bc102570 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 -- 172.21.15.87:0/3333351437 >> 172.21.15.87:0/3333351437 conn(0x7f36bc0793e0 msgr2=0x7f36bc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 -- 172.21.15.87:0/3333351437 shutdown_connections 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.469+0000 7f36c0a4e640 1 -- 172.21.15.87:0/3333351437 wait complete. 2024-04-22T13:32:03.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 Processor -- start 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 -- start start 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 0x7f36bc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc107260 0x7f36bc198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1121a0 con 0x7f36bc19d000 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc112020 con 0x7f36bc102190 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36c0a4e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc112320 con 0x7f36bc107260 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38028/0 (socket says 172.21.15.87:38028) 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 -- 172.21.15.87:0/2645707295 learned_addr learned my addr 172.21.15.87:0/2645707295 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36ba59c640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 0x7f36bc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36b9d9b640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc107260 0x7f36bc198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 -- 172.21.15.87:0/2645707295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc107260 msgr2=0x7f36bc198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc107260 0x7f36bc198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 -- 172.21.15.87:0/2645707295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 msgr2=0x7f36bc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 0x7f36bc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36bad9d640 1 -- 172.21.15.87:0/2645707295 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc19daf0 con 0x7f36bc19d000 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.470+0000 7f36ba59c640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 0x7f36bc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36bad9d640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f36b400eb10 tx=0x7f36b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4004b20 con 0x7f36bc19d000 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36bc19dde0 con 0x7f36bc19d000 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b4004cc0 con 0x7f36bc19d000 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4010670 con 0x7f36bc19d000 2024-04-22T13:32:03.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.471+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36bc1a5580 con 0x7f36bc19d000 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.472+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36bc069230 con 0x7f36bc19d000 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.473+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f36b4005020 con 0x7f36bc19d000 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.473+0000 7f36b37fe640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 0x7f36a80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.474+0000 7f36ba59c640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 0x7f36a80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.474+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f36b40d10e0 con 0x7f36bc19d000 2024-04-22T13:32:03.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.474+0000 7f36ba59c640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 0x7f36a80a6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005ab0 tx=0x7f36ac005a20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.478+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36b409d5b0 con 0x7f36bc19d000 2024-04-22T13:32:03.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.650+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36bc102b40 con 0x7f36a80a3ed0 2024-04-22T13:32:03.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.657+0000 7f36b37fe640 1 -- 172.21.15.87:0/2645707295 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258041 (secure 0 0 0) 0x7f36bc102b40 con 0x7f36a80a3ed0 2024-04-22T13:32:03.656 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:03.657 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:32:03.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 msgr2=0x7f36a80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 0x7f36a80a6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f36ac005ab0 tx=0x7f36ac005a20 comp rx=0 tx=0).stop 2024-04-22T13:32:03.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 msgr2=0x7f36bc19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f36b400eb10 tx=0x7f36b400c2d0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 shutdown_connections 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f36a80a3ed0 0x7f36a80a6390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36bc19d000 0x7f36bc19d4b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f36bc107260 0x7f36bc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 --2- 172.21.15.87:0/2645707295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36bc102190 0x7f36bc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 >> 172.21.15.87:0/2645707295 conn(0x7f36bc0793e0 msgr2=0x7f36bc0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 shutdown_connections 2024-04-22T13:32:03.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.660+0000 7f36c0a4e640 1 -- 172.21.15.87:0/2645707295 wait complete. 2024-04-22T13:32:03.667 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T13:32:02.034703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189320,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1127,"num_read_kb":886,"num_write":495,"num_write_kb":4381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369348,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572476,"statfs":{"total":1151940427776,"available":1151562215424,"internally_reserved":0,"allocated":12140544,"data_stored":9139488,"data_compressed":223395,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.989953"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":159,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503560+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:56.503560+0000","last_peered":"2024-04-22T13:31:56.503560+0000","last_clean":"2024-04-22T13:31:56.503560+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:56.503560+0000","last_undegraded":"2024-04-22T13:31:56.503560+0000","last_fullsized":"2024-04-22T13:31:56.503560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503056+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:56.503056+0000","last_peered":"2024-04-22T13:31:56.503056+0000","last_clean":"2024-04-22T13:31:56.503056+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:56.503056+0000","last_undegraded":"2024-04-22T13:31:56.503056+0000","last_fullsized":"2024-04-22T13:31:56.503056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'37","reported_seq":325,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:57.528536+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:57.528536+0000","last_peered":"2024-04-22T13:31:57.528536+0000","last_clean":"2024-04-22T13:31:57.528536+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:57.528536+0000","last_undegraded":"2024-04-22T13:31:57.528536+0000","last_fullsized":"2024-04-22T13:31:57.528536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68029,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":267,"num_read_kb":200,"num_write":38,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503870+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:56.503870+0000","last_peered":"2024-04-22T13:31:56.503870+0000","last_clean":"2024-04-22T13:31:56.503870+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:56.503870+0000","last_undegraded":"2024-04-22T13:31:56.503870+0000","last_fullsized":"2024-04-22T13:31:56.503870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'5","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.339239+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.339239+0000","last_peered":"2024-04-22T13:31:52.339239+0000","last_clean":"2024-04-22T13:31:52.339239+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.339239+0000","last_undegraded":"2024-04-22T13:31:52.339239+0000","last_fullsized":"2024-04-22T13:31:52.339239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025909+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025909+0000","last_peered":"2024-04-22T13:31:52.025909+0000","last_clean":"2024-04-22T13:31:52.025909+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025909+0000","last_undegraded":"2024-04-22T13:31:52.025909+0000","last_fullsized":"2024-04-22T13:31:52.025909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025701+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025701+0000","last_peered":"2024-04-22T13:31:52.025701+0000","last_clean":"2024-04-22T13:31:52.025701+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025701+0000","last_undegraded":"2024-04-22T13:31:52.025701+0000","last_fullsized":"2024-04-22T13:31:52.025701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":154,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503404+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:56.503404+0000","last_peered":"2024-04-22T13:31:56.503404+0000","last_clean":"2024-04-22T13:31:56.503404+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:56.503404+0000","last_undegraded":"2024-04-22T13:31:56.503404+0000","last_fullsized":"2024-04-22T13:31:56.503404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:54.141174+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:54.141174+0000","last_peered":"2024-04-22T13:31:54.141174+0000","last_clean":"2024-04-22T13:31:54.141174+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:54.141174+0000","last_undegraded":"2024-04-22T13:31:54.141174+0000","last_fullsized":"2024-04-22T13:31:54.141174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111290+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111290+0000","last_peered":"2024-04-22T13:31:52.111290+0000","last_clean":"2024-04-22T13:31:52.111290+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111290+0000","last_undegraded":"2024-04-22T13:31:52.111290+0000","last_fullsized":"2024-04-22T13:31:52.111290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:53.032318+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:53.032318+0000","last_peered":"2024-04-22T13:31:53.032318+0000","last_clean":"2024-04-22T13:31:53.032318+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:53.032318+0000","last_undegraded":"2024-04-22T13:31:53.032318+0000","last_fullsized":"2024-04-22T13:31:53.032318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112338+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112338+0000","last_peered":"2024-04-22T13:31:52.112338+0000","last_clean":"2024-04-22T13:31:52.112338+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112338+0000","last_undegraded":"2024-04-22T13:31:52.112338+0000","last_fullsized":"2024-04-22T13:31:52.112338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196097+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196097+0000","last_peered":"2024-04-22T13:31:52.196097+0000","last_clean":"2024-04-22T13:31:52.196097+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196097+0000","last_undegraded":"2024-04-22T13:31:52.196097+0000","last_fullsized":"2024-04-22T13:31:52.196097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210877+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210877+0000","last_peered":"2024-04-22T13:31:52.210877+0000","last_clean":"2024-04-22T13:31:52.210877+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210877+0000","last_undegraded":"2024-04-22T13:31:52.210877+0000","last_fullsized":"2024-04-22T13:31:52.210877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209960+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209960+0000","last_peered":"2024-04-22T13:31:52.209960+0000","last_clean":"2024-04-22T13:31:52.209960+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209960+0000","last_undegraded":"2024-04-22T13:31:52.209960+0000","last_fullsized":"2024-04-22T13:31:52.209960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021921+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021921+0000","last_peered":"2024-04-22T13:31:52.021921+0000","last_clean":"2024-04-22T13:31:52.021921+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021921+0000","last_undegraded":"2024-04-22T13:31:52.021921+0000","last_fullsized":"2024-04-22T13:31:52.021921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209941+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209941+0000","last_peered":"2024-04-22T13:31:52.209941+0000","last_clean":"2024-04-22T13:31:52.209941+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209941+0000","last_undegraded":"2024-04-22T13:31:52.209941+0000","last_fullsized":"2024-04-22T13:31:52.209941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210723+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210723+0000","last_peered":"2024-04-22T13:31:52.210723+0000","last_clean":"2024-04-22T13:31:52.210723+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210723+0000","last_undegraded":"2024-04-22T13:31:52.210723+0000","last_fullsized":"2024-04-22T13:31:52.210723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210147+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210147+0000","last_peered":"2024-04-22T13:31:52.210147+0000","last_clean":"2024-04-22T13:31:52.210147+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210147+0000","last_undegraded":"2024-04-22T13:31:52.210147+0000","last_fullsized":"2024-04-22T13:31:52.210147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209999+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209999+0000","last_peered":"2024-04-22T13:31:52.209999+0000","last_clean":"2024-04-22T13:31:52.209999+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209999+0000","last_undegraded":"2024-04-22T13:31:52.209999+0000","last_fullsized":"2024-04-22T13:31:52.209999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209923+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209923+0000","last_peered":"2024-04-22T13:31:52.209923+0000","last_clean":"2024-04-22T13:31:52.209923+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209923+0000","last_undegraded":"2024-04-22T13:31:52.209923+0000","last_fullsized":"2024-04-22T13:31:52.209923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210319+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210319+0000","last_peered":"2024-04-22T13:31:52.210319+0000","last_clean":"2024-04-22T13:31:52.210319+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210319+0000","last_undegraded":"2024-04-22T13:31:52.210319+0000","last_fullsized":"2024-04-22T13:31:52.210319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022549+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022549+0000","last_peered":"2024-04-22T13:31:52.022549+0000","last_clean":"2024-04-22T13:31:52.022549+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022549+0000","last_undegraded":"2024-04-22T13:31:52.022549+0000","last_fullsized":"2024-04-22T13:31:52.022549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598718,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":890,"num_read_kb":556,"num_write":292,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1796154,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257467,"data_compressed":46637,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1058834,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":532323,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":534421,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466419,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188637,"data_compressed":44795,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68029,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70127,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133858,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202688,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:32:03.671 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-22T13:32:03.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T13:32:03.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2826177086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac103af0 msgr2=0x7fe8ac10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2826177086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac103af0 0x7fe8ac10bfd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe898009900 tx=0x7fe89802bcd0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2826177086 shutdown_connections 2024-04-22T13:32:03.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2826177086 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 0x7fe8ac10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2826177086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac103af0 0x7fe8ac10bfd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2826177086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2826177086 >> 172.21.15.87:0/2826177086 conn(0x7fe8ac0771d0 msgr2=0x7fe8ac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2826177086 shutdown_connections 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.810+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2826177086 wait complete. 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 Processor -- start 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 -- start start 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac103af0 0x7fe8ac198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 0x7fe8ac19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac112230 con 0x7fe8ac1031d0 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac1120b0 con 0x7fe8ac103af0 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b2db6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac1123b0 con 0x7fe8ac10c5a0 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b0b52640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b0b52640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38042/0 (socket says 172.21.15.87:38042) 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b0b52640 1 -- 172.21.15.87:0/2916310454 learned_addr learned my addr 172.21.15.87:0/2916310454 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8abfff640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac103af0 0x7fe8ac198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.811+0000 7fe8b1353640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 0x7fe8ac19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 -- 172.21.15.87:0/2916310454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 msgr2=0x7fe8ac19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 0x7fe8ac19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 -- 172.21.15.87:0/2916310454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac103af0 msgr2=0x7fe8ac198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac103af0 0x7fe8ac198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 -- 172.21.15.87:0/2916310454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ac19d720 con 0x7fe8ac1031d0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b0b52640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000d950 tx=0x7fe8a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0017070 con 0x7fe8ac1031d0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a0009c30 con 0x7fe8ac1031d0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ac19da10 con 0x7fe8ac1031d0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a000f620 con 0x7fe8ac1031d0 2024-04-22T13:32:03.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.812+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ac1a52a0 con 0x7fe8ac1031d0 2024-04-22T13:32:03.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.813+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ac069230 con 0x7fe8ac1031d0 2024-04-22T13:32:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.814+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe8a0010460 con 0x7fe8ac1031d0 2024-04-22T13:32:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.815+0000 7fe8a9ffb640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 0x7fe89c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.815+0000 7fe8abfff640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 0x7fe89c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.815+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe8a00d0620 con 0x7fe8ac1031d0 2024-04-22T13:32:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.815+0000 7fe8abfff640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 0x7fe89c0a6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe898005ab0 tx=0x7fe898005a20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:03.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.819+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8a0009de0 con 0x7fe8ac1031d0 2024-04-22T13:32:03.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:03.994+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8ac062390 con 0x7fe89c0a3ed0 2024-04-22T13:32:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.001+0000 7fe8a9ffb640 1 -- 172.21.15.87:0/2916310454 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258041 (secure 0 0 0) 0x7fe8ac062390 con 0x7fe89c0a3ed0 2024-04-22T13:32:03.999 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:04.000 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 msgr2=0x7fe89c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 0x7fe89c0a6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe898005ab0 tx=0x7fe898005a20 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 msgr2=0x7fe8ac198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000d950 tx=0x7fe8a000de10 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 shutdown_connections 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe89c0a3ed0 0x7fe89c0a6390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe8ac10c5a0 0x7fe8ac19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8ac103af0 0x7fe8ac198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 --2- 172.21.15.87:0/2916310454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe8ac1031d0 0x7fe8ac198730 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 >> 172.21.15.87:0/2916310454 conn(0x7fe8ac0771d0 msgr2=0x7fe8ac0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 shutdown_connections 2024-04-22T13:32:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.004+0000 7fe8b2db6640 1 -- 172.21.15.87:0/2916310454 wait complete. 2024-04-22T13:32:04.010 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T13:32:02.034703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189320,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1127,"num_read_kb":886,"num_write":495,"num_write_kb":4381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180887,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369348,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572476,"statfs":{"total":1151940427776,"available":1151562215424,"internally_reserved":0,"allocated":12140544,"data_stored":9139488,"data_compressed":223395,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.989953"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022302+0000","last_change":"2024-04-22T13:29:47.342586+0000","last_active":"2024-04-22T13:31:52.022302+0000","last_peered":"2024-04-22T13:31:52.022302+0000","last_clean":"2024-04-22T13:31:52.022302+0000","last_became_active":"2024-04-22T13:29:47.342262+0000","last_became_peered":"2024-04-22T13:29:47.342262+0000","last_unstale":"2024-04-22T13:31:52.022302+0000","last_undegraded":"2024-04-22T13:31:52.022302+0000","last_fullsized":"2024-04-22T13:31:52.022302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:58:07.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210042+0000","last_change":"2024-04-22T13:29:47.340049+0000","last_active":"2024-04-22T13:31:52.210042+0000","last_peered":"2024-04-22T13:31:52.210042+0000","last_clean":"2024-04-22T13:31:52.210042+0000","last_became_active":"2024-04-22T13:29:47.335147+0000","last_became_peered":"2024-04-22T13:29:47.335147+0000","last_unstale":"2024-04-22T13:31:52.210042+0000","last_undegraded":"2024-04-22T13:31:52.210042+0000","last_fullsized":"2024-04-22T13:31:52.210042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:09:09.712735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210531+0000","last_change":"2024-04-22T13:29:47.339642+0000","last_active":"2024-04-22T13:31:52.210531+0000","last_peered":"2024-04-22T13:31:52.210531+0000","last_clean":"2024-04-22T13:31:52.210531+0000","last_became_active":"2024-04-22T13:29:47.339275+0000","last_became_peered":"2024-04-22T13:29:47.339275+0000","last_unstale":"2024-04-22T13:31:52.210531+0000","last_undegraded":"2024-04-22T13:31:52.210531+0000","last_fullsized":"2024-04-22T13:31:52.210531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:39:29.489668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196075+0000","last_change":"2024-04-22T13:29:47.335737+0000","last_active":"2024-04-22T13:31:52.196075+0000","last_peered":"2024-04-22T13:31:52.196075+0000","last_clean":"2024-04-22T13:31:52.196075+0000","last_became_active":"2024-04-22T13:29:47.335367+0000","last_became_peered":"2024-04-22T13:29:47.335367+0000","last_unstale":"2024-04-22T13:31:52.196075+0000","last_undegraded":"2024-04-22T13:31:52.196075+0000","last_fullsized":"2024-04-22T13:31:52.196075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:46.803298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111016+0000","last_change":"2024-04-22T13:29:47.337101+0000","last_active":"2024-04-22T13:31:52.111016+0000","last_peered":"2024-04-22T13:31:52.111016+0000","last_clean":"2024-04-22T13:31:52.111016+0000","last_became_active":"2024-04-22T13:29:47.336688+0000","last_became_peered":"2024-04-22T13:29:47.336688+0000","last_unstale":"2024-04-22T13:31:52.111016+0000","last_undegraded":"2024-04-22T13:31:52.111016+0000","last_fullsized":"2024-04-22T13:31:52.111016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:21:23.494041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025552+0000","last_change":"2024-04-22T13:29:47.333224+0000","last_active":"2024-04-22T13:31:52.025552+0000","last_peered":"2024-04-22T13:31:52.025552+0000","last_clean":"2024-04-22T13:31:52.025552+0000","last_became_active":"2024-04-22T13:29:47.332832+0000","last_became_peered":"2024-04-22T13:29:47.332832+0000","last_unstale":"2024-04-22T13:31:52.025552+0000","last_undegraded":"2024-04-22T13:31:52.025552+0000","last_fullsized":"2024-04-22T13:31:52.025552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:22.753719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210462+0000","last_change":"2024-04-22T13:29:47.343145+0000","last_active":"2024-04-22T13:31:52.210462+0000","last_peered":"2024-04-22T13:31:52.210462+0000","last_clean":"2024-04-22T13:31:52.210462+0000","last_became_active":"2024-04-22T13:29:47.342801+0000","last_became_peered":"2024-04-22T13:29:47.342801+0000","last_unstale":"2024-04-22T13:31:52.210462+0000","last_undegraded":"2024-04-22T13:31:52.210462+0000","last_fullsized":"2024-04-22T13:31:52.210462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:37.089882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111257+0000","last_change":"2024-04-22T13:29:47.339256+0000","last_active":"2024-04-22T13:31:52.111257+0000","last_peered":"2024-04-22T13:31:52.111257+0000","last_clean":"2024-04-22T13:31:52.111257+0000","last_became_active":"2024-04-22T13:29:47.337918+0000","last_became_peered":"2024-04-22T13:29:47.337918+0000","last_unstale":"2024-04-22T13:31:52.111257+0000","last_undegraded":"2024-04-22T13:31:52.111257+0000","last_fullsized":"2024-04-22T13:31:52.111257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:25.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210177+0000","last_change":"2024-04-22T13:29:47.339823+0000","last_active":"2024-04-22T13:31:52.210177+0000","last_peered":"2024-04-22T13:31:52.210177+0000","last_clean":"2024-04-22T13:31:52.210177+0000","last_became_active":"2024-04-22T13:29:47.339463+0000","last_became_peered":"2024-04-22T13:29:47.339463+0000","last_unstale":"2024-04-22T13:31:52.210177+0000","last_undegraded":"2024-04-22T13:31:52.210177+0000","last_fullsized":"2024-04-22T13:31:52.210177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:19.794368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025924+0000","last_change":"2024-04-22T13:29:47.336225+0000","last_active":"2024-04-22T13:31:52.025924+0000","last_peered":"2024-04-22T13:31:52.025924+0000","last_clean":"2024-04-22T13:31:52.025924+0000","last_became_active":"2024-04-22T13:29:47.335676+0000","last_became_peered":"2024-04-22T13:29:47.335676+0000","last_unstale":"2024-04-22T13:31:52.025924+0000","last_undegraded":"2024-04-22T13:31:52.025924+0000","last_fullsized":"2024-04-22T13:31:52.025924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:20:22.306869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210839+0000","last_change":"2024-04-22T13:29:47.344475+0000","last_active":"2024-04-22T13:31:52.210839+0000","last_peered":"2024-04-22T13:31:52.210839+0000","last_clean":"2024-04-22T13:31:52.210839+0000","last_became_active":"2024-04-22T13:29:47.344125+0000","last_became_peered":"2024-04-22T13:29:47.344125+0000","last_unstale":"2024-04-22T13:31:52.210839+0000","last_undegraded":"2024-04-22T13:31:52.210839+0000","last_fullsized":"2024-04-22T13:31:52.210839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:35.702696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195979+0000","last_change":"2024-04-22T13:29:47.339273+0000","last_active":"2024-04-22T13:31:52.195979+0000","last_peered":"2024-04-22T13:31:52.195979+0000","last_clean":"2024-04-22T13:31:52.195979+0000","last_became_active":"2024-04-22T13:29:47.338918+0000","last_became_peered":"2024-04-22T13:29:47.338918+0000","last_unstale":"2024-04-22T13:31:52.195979+0000","last_undegraded":"2024-04-22T13:31:52.195979+0000","last_fullsized":"2024-04-22T13:31:52.195979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:30:20.669977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498374+0000","last_change":"2024-04-22T13:29:47.319378+0000","last_active":"2024-04-22T13:31:52.498374+0000","last_peered":"2024-04-22T13:31:52.498374+0000","last_clean":"2024-04-22T13:31:52.498374+0000","last_became_active":"2024-04-22T13:29:47.318997+0000","last_became_peered":"2024-04-22T13:29:47.318997+0000","last_unstale":"2024-04-22T13:31:52.498374+0000","last_undegraded":"2024-04-22T13:31:52.498374+0000","last_fullsized":"2024-04-22T13:31:52.498374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:07.043188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210059+0000","last_change":"2024-04-22T13:29:47.334480+0000","last_active":"2024-04-22T13:31:52.210059+0000","last_peered":"2024-04-22T13:31:52.210059+0000","last_clean":"2024-04-22T13:31:52.210059+0000","last_became_active":"2024-04-22T13:29:47.334120+0000","last_became_peered":"2024-04-22T13:29:47.334120+0000","last_unstale":"2024-04-22T13:31:52.210059+0000","last_undegraded":"2024-04-22T13:31:52.210059+0000","last_fullsized":"2024-04-22T13:31:52.210059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:08:23.761251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209873+0000","last_change":"2024-04-22T13:29:47.339653+0000","last_active":"2024-04-22T13:31:52.209873+0000","last_peered":"2024-04-22T13:31:52.209873+0000","last_clean":"2024-04-22T13:31:52.209873+0000","last_became_active":"2024-04-22T13:29:47.339299+0000","last_became_peered":"2024-04-22T13:29:47.339299+0000","last_unstale":"2024-04-22T13:31:52.209873+0000","last_undegraded":"2024-04-22T13:31:52.209873+0000","last_fullsized":"2024-04-22T13:31:52.209873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:20:16.154430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209837+0000","last_change":"2024-04-22T13:29:47.342835+0000","last_active":"2024-04-22T13:31:52.209837+0000","last_peered":"2024-04-22T13:31:52.209837+0000","last_clean":"2024-04-22T13:31:52.209837+0000","last_became_active":"2024-04-22T13:29:47.341626+0000","last_became_peered":"2024-04-22T13:29:47.341626+0000","last_unstale":"2024-04-22T13:31:52.209837+0000","last_undegraded":"2024-04-22T13:31:52.209837+0000","last_fullsized":"2024-04-22T13:31:52.209837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:35.976913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022392+0000","last_change":"2024-04-22T13:29:47.343604+0000","last_active":"2024-04-22T13:31:52.022392+0000","last_peered":"2024-04-22T13:31:52.022392+0000","last_clean":"2024-04-22T13:31:52.022392+0000","last_became_active":"2024-04-22T13:29:47.343105+0000","last_became_peered":"2024-04-22T13:29:47.343105+0000","last_unstale":"2024-04-22T13:31:52.022392+0000","last_undegraded":"2024-04-22T13:31:52.022392+0000","last_fullsized":"2024-04-22T13:31:52.022392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:31:36.661059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021548+0000","last_change":"2024-04-22T13:29:47.335879+0000","last_active":"2024-04-22T13:31:52.021548+0000","last_peered":"2024-04-22T13:31:52.021548+0000","last_clean":"2024-04-22T13:31:52.021548+0000","last_became_active":"2024-04-22T13:29:47.335556+0000","last_became_peered":"2024-04-22T13:29:47.335556+0000","last_unstale":"2024-04-22T13:31:52.021548+0000","last_undegraded":"2024-04-22T13:31:52.021548+0000","last_fullsized":"2024-04-22T13:31:52.021548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:21:11.728429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210027+0000","last_change":"2024-04-22T13:29:47.332437+0000","last_active":"2024-04-22T13:31:52.210027+0000","last_peered":"2024-04-22T13:31:52.210027+0000","last_clean":"2024-04-22T13:31:52.210027+0000","last_became_active":"2024-04-22T13:29:47.332088+0000","last_became_peered":"2024-04-22T13:29:47.332088+0000","last_unstale":"2024-04-22T13:31:52.210027+0000","last_undegraded":"2024-04-22T13:31:52.210027+0000","last_fullsized":"2024-04-22T13:31:52.210027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:36:29.952047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021571+0000","last_change":"2024-04-22T13:29:46.326645+0000","last_active":"2024-04-22T13:31:52.021571+0000","last_peered":"2024-04-22T13:31:52.021571+0000","last_clean":"2024-04-22T13:31:52.021571+0000","last_became_active":"2024-04-22T13:29:46.326299+0000","last_became_peered":"2024-04-22T13:29:46.326299+0000","last_unstale":"2024-04-22T13:31:52.021571+0000","last_undegraded":"2024-04-22T13:31:52.021571+0000","last_fullsized":"2024-04-22T13:31:52.021571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:43:27.760358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112030+0000","last_change":"2024-04-22T13:29:47.339014+0000","last_active":"2024-04-22T13:31:52.112030+0000","last_peered":"2024-04-22T13:31:52.112030+0000","last_clean":"2024-04-22T13:31:52.112030+0000","last_became_active":"2024-04-22T13:29:47.338671+0000","last_became_peered":"2024-04-22T13:29:47.338671+0000","last_unstale":"2024-04-22T13:31:52.112030+0000","last_undegraded":"2024-04-22T13:31:52.112030+0000","last_fullsized":"2024-04-22T13:31:52.112030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:45:48.959100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112180+0000","last_change":"2024-04-22T13:29:46.336530+0000","last_active":"2024-04-22T13:31:52.112180+0000","last_peered":"2024-04-22T13:31:52.112180+0000","last_clean":"2024-04-22T13:31:52.112180+0000","last_became_active":"2024-04-22T13:29:46.336170+0000","last_became_peered":"2024-04-22T13:29:46.336170+0000","last_unstale":"2024-04-22T13:31:52.112180+0000","last_undegraded":"2024-04-22T13:31:52.112180+0000","last_fullsized":"2024-04-22T13:31:52.112180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:33.339445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.211036+0000","last_change":"2024-04-22T13:29:47.336358+0000","last_active":"2024-04-22T13:31:52.211036+0000","last_peered":"2024-04-22T13:31:52.211036+0000","last_clean":"2024-04-22T13:31:52.211036+0000","last_became_active":"2024-04-22T13:29:47.336056+0000","last_became_peered":"2024-04-22T13:29:47.336056+0000","last_unstale":"2024-04-22T13:31:52.211036+0000","last_undegraded":"2024-04-22T13:31:52.211036+0000","last_fullsized":"2024-04-22T13:31:52.211036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:35:04.246460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022386+0000","last_change":"2024-04-22T13:30:37.903473+0000","last_active":"2024-04-22T13:31:52.022386+0000","last_peered":"2024-04-22T13:31:52.022386+0000","last_clean":"2024-04-22T13:31:52.022386+0000","last_became_active":"2024-04-22T13:30:37.893153+0000","last_became_peered":"2024-04-22T13:30:37.893153+0000","last_unstale":"2024-04-22T13:31:52.022386+0000","last_undegraded":"2024-04-22T13:31:52.022386+0000","last_fullsized":"2024-04-22T13:31:52.022386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:49:35.959905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210046+0000","last_change":"2024-04-22T13:29:47.343013+0000","last_active":"2024-04-22T13:31:52.210046+0000","last_peered":"2024-04-22T13:31:52.210046+0000","last_clean":"2024-04-22T13:31:52.210046+0000","last_became_active":"2024-04-22T13:29:47.342677+0000","last_became_peered":"2024-04-22T13:29:47.342677+0000","last_unstale":"2024-04-22T13:31:52.210046+0000","last_undegraded":"2024-04-22T13:31:52.210046+0000","last_fullsized":"2024-04-22T13:31:52.210046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:43:42.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111206+0000","last_change":"2024-04-22T13:29:46.333637+0000","last_active":"2024-04-22T13:31:52.111206+0000","last_peered":"2024-04-22T13:31:52.111206+0000","last_clean":"2024-04-22T13:31:52.111206+0000","last_became_active":"2024-04-22T13:29:46.333118+0000","last_became_peered":"2024-04-22T13:29:46.333118+0000","last_unstale":"2024-04-22T13:31:52.111206+0000","last_undegraded":"2024-04-22T13:31:52.111206+0000","last_fullsized":"2024-04-22T13:31:52.111206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:37:23.112995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082090+0000","last_change":"2024-04-22T13:29:47.334774+0000","last_active":"2024-04-22T13:31:52.082090+0000","last_peered":"2024-04-22T13:31:52.082090+0000","last_clean":"2024-04-22T13:31:52.082090+0000","last_became_active":"2024-04-22T13:29:47.334426+0000","last_became_peered":"2024-04-22T13:29:47.334426+0000","last_unstale":"2024-04-22T13:31:52.082090+0000","last_undegraded":"2024-04-22T13:31:52.082090+0000","last_fullsized":"2024-04-22T13:31:52.082090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:08:10.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":159,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503560+0000","last_change":"2024-04-22T13:29:46.330446+0000","last_active":"2024-04-22T13:31:56.503560+0000","last_peered":"2024-04-22T13:31:56.503560+0000","last_clean":"2024-04-22T13:31:56.503560+0000","last_became_active":"2024-04-22T13:29:46.330095+0000","last_became_peered":"2024-04-22T13:29:46.330095+0000","last_unstale":"2024-04-22T13:31:56.503560+0000","last_undegraded":"2024-04-22T13:31:56.503560+0000","last_fullsized":"2024-04-22T13:31:56.503560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:33:11.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111039+0000","last_change":"2024-04-22T13:29:47.328886+0000","last_active":"2024-04-22T13:31:52.111039+0000","last_peered":"2024-04-22T13:31:52.111039+0000","last_clean":"2024-04-22T13:31:52.111039+0000","last_became_active":"2024-04-22T13:29:47.328535+0000","last_became_peered":"2024-04-22T13:29:47.328535+0000","last_unstale":"2024-04-22T13:31:52.111039+0000","last_undegraded":"2024-04-22T13:31:52.111039+0000","last_fullsized":"2024-04-22T13:31:52.111039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:29:51.298140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025757+0000","last_change":"2024-04-22T13:29:46.333456+0000","last_active":"2024-04-22T13:31:52.025757+0000","last_peered":"2024-04-22T13:31:52.025757+0000","last_clean":"2024-04-22T13:31:52.025757+0000","last_became_active":"2024-04-22T13:29:46.333094+0000","last_became_peered":"2024-04-22T13:29:46.333094+0000","last_unstale":"2024-04-22T13:31:52.025757+0000","last_undegraded":"2024-04-22T13:31:52.025757+0000","last_fullsized":"2024-04-22T13:31:52.025757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:44:25.229364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082000+0000","last_change":"2024-04-22T13:29:47.336838+0000","last_active":"2024-04-22T13:31:52.082000+0000","last_peered":"2024-04-22T13:31:52.082000+0000","last_clean":"2024-04-22T13:31:52.082000+0000","last_became_active":"2024-04-22T13:29:47.336509+0000","last_became_peered":"2024-04-22T13:29:47.336509+0000","last_unstale":"2024-04-22T13:31:52.082000+0000","last_undegraded":"2024-04-22T13:31:52.082000+0000","last_fullsized":"2024-04-22T13:31:52.082000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:22.468022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498510+0000","last_change":"2024-04-22T13:29:46.323219+0000","last_active":"2024-04-22T13:31:52.498510+0000","last_peered":"2024-04-22T13:31:52.498510+0000","last_clean":"2024-04-22T13:31:52.498510+0000","last_became_active":"2024-04-22T13:29:46.322264+0000","last_became_peered":"2024-04-22T13:29:46.322264+0000","last_unstale":"2024-04-22T13:31:52.498510+0000","last_undegraded":"2024-04-22T13:31:52.498510+0000","last_fullsized":"2024-04-22T13:31:52.498510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:06:12.268447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111356+0000","last_change":"2024-04-22T13:29:47.332130+0000","last_active":"2024-04-22T13:31:52.111356+0000","last_peered":"2024-04-22T13:31:52.111356+0000","last_clean":"2024-04-22T13:31:52.111356+0000","last_became_active":"2024-04-22T13:29:47.331705+0000","last_became_peered":"2024-04-22T13:29:47.331705+0000","last_unstale":"2024-04-22T13:31:52.111356+0000","last_undegraded":"2024-04-22T13:31:52.111356+0000","last_fullsized":"2024-04-22T13:31:52.111356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:28:18.900160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081821+0000","last_change":"2024-04-22T13:29:46.328447+0000","last_active":"2024-04-22T13:31:52.081821+0000","last_peered":"2024-04-22T13:31:52.081821+0000","last_clean":"2024-04-22T13:31:52.081821+0000","last_became_active":"2024-04-22T13:29:46.328072+0000","last_became_peered":"2024-04-22T13:29:46.328072+0000","last_unstale":"2024-04-22T13:31:52.081821+0000","last_undegraded":"2024-04-22T13:31:52.081821+0000","last_fullsized":"2024-04-22T13:31:52.081821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:41.422045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021199+0000","last_change":"2024-04-22T13:29:47.330720+0000","last_active":"2024-04-22T13:31:52.021199+0000","last_peered":"2024-04-22T13:31:52.021199+0000","last_clean":"2024-04-22T13:31:52.021199+0000","last_became_active":"2024-04-22T13:29:47.329744+0000","last_became_peered":"2024-04-22T13:29:47.329744+0000","last_unstale":"2024-04-22T13:31:52.021199+0000","last_undegraded":"2024-04-22T13:31:52.021199+0000","last_fullsized":"2024-04-22T13:31:52.021199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:20:52.060961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503056+0000","last_change":"2024-04-22T13:29:46.324063+0000","last_active":"2024-04-22T13:31:56.503056+0000","last_peered":"2024-04-22T13:31:56.503056+0000","last_clean":"2024-04-22T13:31:56.503056+0000","last_became_active":"2024-04-22T13:29:46.323760+0000","last_became_peered":"2024-04-22T13:29:46.323760+0000","last_unstale":"2024-04-22T13:31:56.503056+0000","last_undegraded":"2024-04-22T13:31:56.503056+0000","last_fullsized":"2024-04-22T13:31:56.503056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:59:18.611798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210254+0000","last_change":"2024-04-22T13:29:47.340227+0000","last_active":"2024-04-22T13:31:52.210254+0000","last_peered":"2024-04-22T13:31:52.210254+0000","last_clean":"2024-04-22T13:31:52.210254+0000","last_became_active":"2024-04-22T13:29:47.339751+0000","last_became_peered":"2024-04-22T13:29:47.339751+0000","last_unstale":"2024-04-22T13:31:52.210254+0000","last_undegraded":"2024-04-22T13:31:52.210254+0000","last_fullsized":"2024-04-22T13:31:52.210254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:02:22.581599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210876+0000","last_change":"2024-04-22T13:29:46.331323+0000","last_active":"2024-04-22T13:31:52.210876+0000","last_peered":"2024-04-22T13:31:52.210876+0000","last_clean":"2024-04-22T13:31:52.210876+0000","last_became_active":"2024-04-22T13:29:46.330892+0000","last_became_peered":"2024-04-22T13:29:46.330892+0000","last_unstale":"2024-04-22T13:31:52.210876+0000","last_undegraded":"2024-04-22T13:31:52.210876+0000","last_fullsized":"2024-04-22T13:31:52.210876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:11:04.285602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210963+0000","last_change":"2024-04-22T13:29:47.340416+0000","last_active":"2024-04-22T13:31:52.210963+0000","last_peered":"2024-04-22T13:31:52.210963+0000","last_clean":"2024-04-22T13:31:52.210963+0000","last_became_active":"2024-04-22T13:29:47.340088+0000","last_became_peered":"2024-04-22T13:29:47.340088+0000","last_unstale":"2024-04-22T13:31:52.210963+0000","last_undegraded":"2024-04-22T13:31:52.210963+0000","last_fullsized":"2024-04-22T13:31:52.210963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:31.609967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082206+0000","last_change":"2024-04-22T13:29:46.330179+0000","last_active":"2024-04-22T13:31:52.082206+0000","last_peered":"2024-04-22T13:31:52.082206+0000","last_clean":"2024-04-22T13:31:52.082206+0000","last_became_active":"2024-04-22T13:29:46.329771+0000","last_became_peered":"2024-04-22T13:29:46.329771+0000","last_unstale":"2024-04-22T13:31:52.082206+0000","last_undegraded":"2024-04-22T13:31:52.082206+0000","last_fullsized":"2024-04-22T13:31:52.082206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:15:04.046964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210160+0000","last_change":"2024-04-22T13:29:47.338341+0000","last_active":"2024-04-22T13:31:52.210160+0000","last_peered":"2024-04-22T13:31:52.210160+0000","last_clean":"2024-04-22T13:31:52.210160+0000","last_became_active":"2024-04-22T13:29:47.337986+0000","last_became_peered":"2024-04-22T13:29:47.337986+0000","last_unstale":"2024-04-22T13:31:52.210160+0000","last_undegraded":"2024-04-22T13:31:52.210160+0000","last_fullsized":"2024-04-22T13:31:52.210160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:38:59.712483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'37","reported_seq":325,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:57.528536+0000","last_change":"2024-04-22T13:29:46.321565+0000","last_active":"2024-04-22T13:31:57.528536+0000","last_peered":"2024-04-22T13:31:57.528536+0000","last_clean":"2024-04-22T13:31:57.528536+0000","last_became_active":"2024-04-22T13:29:46.321181+0000","last_became_peered":"2024-04-22T13:29:46.321181+0000","last_unstale":"2024-04-22T13:31:57.528536+0000","last_undegraded":"2024-04-22T13:31:57.528536+0000","last_fullsized":"2024-04-22T13:31:57.528536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:31:46.116970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68029,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":267,"num_read_kb":200,"num_write":38,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111196+0000","last_change":"2024-04-22T13:29:47.340519+0000","last_active":"2024-04-22T13:31:52.111196+0000","last_peered":"2024-04-22T13:31:52.111196+0000","last_clean":"2024-04-22T13:31:52.111196+0000","last_became_active":"2024-04-22T13:29:47.340147+0000","last_became_peered":"2024-04-22T13:29:47.340147+0000","last_unstale":"2024-04-22T13:31:52.111196+0000","last_undegraded":"2024-04-22T13:31:52.111196+0000","last_fullsized":"2024-04-22T13:31:52.111196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:06.879446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021875+0000","last_change":"2024-04-22T13:30:37.899796+0000","last_active":"2024-04-22T13:31:52.021875+0000","last_peered":"2024-04-22T13:31:52.021875+0000","last_clean":"2024-04-22T13:31:52.021875+0000","last_became_active":"2024-04-22T13:30:37.891788+0000","last_became_peered":"2024-04-22T13:30:37.891788+0000","last_unstale":"2024-04-22T13:31:52.021875+0000","last_undegraded":"2024-04-22T13:31:52.021875+0000","last_fullsized":"2024-04-22T13:31:52.021875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:52:11.715783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498241+0000","last_change":"2024-04-22T13:29:47.332495+0000","last_active":"2024-04-22T13:31:52.498241+0000","last_peered":"2024-04-22T13:31:52.498241+0000","last_clean":"2024-04-22T13:31:52.498241+0000","last_became_active":"2024-04-22T13:29:47.332199+0000","last_became_peered":"2024-04-22T13:29:47.332199+0000","last_unstale":"2024-04-22T13:31:52.498241+0000","last_undegraded":"2024-04-22T13:31:52.498241+0000","last_fullsized":"2024-04-22T13:31:52.498241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:25:29.992668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081960+0000","last_change":"2024-04-22T13:29:46.328322+0000","last_active":"2024-04-22T13:31:52.081960+0000","last_peered":"2024-04-22T13:31:52.081960+0000","last_clean":"2024-04-22T13:31:52.081960+0000","last_became_active":"2024-04-22T13:29:46.327985+0000","last_became_peered":"2024-04-22T13:29:46.327985+0000","last_unstale":"2024-04-22T13:31:52.081960+0000","last_undegraded":"2024-04-22T13:31:52.081960+0000","last_fullsized":"2024-04-22T13:31:52.081960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:13:50.301696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210719+0000","last_change":"2024-04-22T13:30:37.985927+0000","last_active":"2024-04-22T13:31:52.210719+0000","last_peered":"2024-04-22T13:31:52.210719+0000","last_clean":"2024-04-22T13:31:52.210719+0000","last_became_active":"2024-04-22T13:30:37.985543+0000","last_became_peered":"2024-04-22T13:30:37.985543+0000","last_unstale":"2024-04-22T13:31:52.210719+0000","last_undegraded":"2024-04-22T13:31:52.210719+0000","last_fullsized":"2024-04-22T13:31:52.210719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:36.663768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503870+0000","last_change":"2024-04-22T13:29:46.327217+0000","last_active":"2024-04-22T13:31:56.503870+0000","last_peered":"2024-04-22T13:31:56.503870+0000","last_clean":"2024-04-22T13:31:56.503870+0000","last_became_active":"2024-04-22T13:29:46.326879+0000","last_became_peered":"2024-04-22T13:29:46.326879+0000","last_unstale":"2024-04-22T13:31:56.503870+0000","last_undegraded":"2024-04-22T13:31:56.503870+0000","last_fullsized":"2024-04-22T13:31:56.503870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:12:24.612757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210415+0000","last_change":"2024-04-22T13:29:47.342486+0000","last_active":"2024-04-22T13:31:52.210415+0000","last_peered":"2024-04-22T13:31:52.210415+0000","last_clean":"2024-04-22T13:31:52.210415+0000","last_became_active":"2024-04-22T13:29:47.342131+0000","last_became_peered":"2024-04-22T13:29:47.342131+0000","last_unstale":"2024-04-22T13:31:52.210415+0000","last_undegraded":"2024-04-22T13:31:52.210415+0000","last_fullsized":"2024-04-22T13:31:52.210415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:46:11.380305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:46.333463+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:46.333154+0000","last_became_peered":"2024-04-22T13:29:46.333154+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:56:16.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081869+0000","last_change":"2024-04-22T13:29:47.336623+0000","last_active":"2024-04-22T13:31:52.081869+0000","last_peered":"2024-04-22T13:31:52.081869+0000","last_clean":"2024-04-22T13:31:52.081869+0000","last_became_active":"2024-04-22T13:29:47.336240+0000","last_became_peered":"2024-04-22T13:29:47.336240+0000","last_unstale":"2024-04-22T13:31:52.081869+0000","last_undegraded":"2024-04-22T13:31:52.081869+0000","last_fullsized":"2024-04-22T13:31:52.081869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:55:15.693723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'5","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.339239+0000","last_change":"2024-04-22T13:29:46.333951+0000","last_active":"2024-04-22T13:31:52.339239+0000","last_peered":"2024-04-22T13:31:52.339239+0000","last_clean":"2024-04-22T13:31:52.339239+0000","last_became_active":"2024-04-22T13:29:46.333052+0000","last_became_peered":"2024-04-22T13:29:46.333052+0000","last_unstale":"2024-04-22T13:31:52.339239+0000","last_undegraded":"2024-04-22T13:31:52.339239+0000","last_fullsized":"2024-04-22T13:31:52.339239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:46:55.093182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022097+0000","last_change":"2024-04-22T13:30:37.985586+0000","last_active":"2024-04-22T13:31:52.022097+0000","last_peered":"2024-04-22T13:31:52.022097+0000","last_clean":"2024-04-22T13:31:52.022097+0000","last_became_active":"2024-04-22T13:30:37.985261+0000","last_became_peered":"2024-04-22T13:30:37.985261+0000","last_unstale":"2024-04-22T13:31:52.022097+0000","last_undegraded":"2024-04-22T13:31:52.022097+0000","last_fullsized":"2024-04-22T13:31:52.022097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:04:05.063638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021367+0000","last_change":"2024-04-22T13:29:46.318339+0000","last_active":"2024-04-22T13:31:52.021367+0000","last_peered":"2024-04-22T13:31:52.021367+0000","last_clean":"2024-04-22T13:31:52.021367+0000","last_became_active":"2024-04-22T13:29:46.317949+0000","last_became_peered":"2024-04-22T13:29:46.317949+0000","last_unstale":"2024-04-22T13:31:52.021367+0000","last_undegraded":"2024-04-22T13:31:52.021367+0000","last_fullsized":"2024-04-22T13:31:52.021367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:41:28.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025909+0000","last_change":"2024-04-22T13:29:47.332584+0000","last_active":"2024-04-22T13:31:52.025909+0000","last_peered":"2024-04-22T13:31:52.025909+0000","last_clean":"2024-04-22T13:31:52.025909+0000","last_became_active":"2024-04-22T13:29:47.332261+0000","last_became_peered":"2024-04-22T13:29:47.332261+0000","last_unstale":"2024-04-22T13:31:52.025909+0000","last_undegraded":"2024-04-22T13:31:52.025909+0000","last_fullsized":"2024-04-22T13:31:52.025909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:10:44.395910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210597+0000","last_change":"2024-04-22T13:29:46.333986+0000","last_active":"2024-04-22T13:31:52.210597+0000","last_peered":"2024-04-22T13:31:52.210597+0000","last_clean":"2024-04-22T13:31:52.210597+0000","last_became_active":"2024-04-22T13:29:46.333605+0000","last_became_peered":"2024-04-22T13:29:46.333605+0000","last_unstale":"2024-04-22T13:31:52.210597+0000","last_undegraded":"2024-04-22T13:31:52.210597+0000","last_fullsized":"2024-04-22T13:31:52.210597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:54:54.589852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.025701+0000","last_change":"2024-04-22T13:29:47.333185+0000","last_active":"2024-04-22T13:31:52.025701+0000","last_peered":"2024-04-22T13:31:52.025701+0000","last_clean":"2024-04-22T13:31:52.025701+0000","last_became_active":"2024-04-22T13:29:47.332788+0000","last_became_peered":"2024-04-22T13:29:47.332788+0000","last_unstale":"2024-04-22T13:31:52.025701+0000","last_undegraded":"2024-04-22T13:31:52.025701+0000","last_fullsized":"2024-04-22T13:31:52.025701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:33:30.762274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.082089+0000","last_change":"2024-04-22T13:29:47.339465+0000","last_active":"2024-04-22T13:31:52.082089+0000","last_peered":"2024-04-22T13:31:52.082089+0000","last_clean":"2024-04-22T13:31:52.082089+0000","last_became_active":"2024-04-22T13:29:47.339014+0000","last_became_peered":"2024-04-22T13:29:47.339014+0000","last_unstale":"2024-04-22T13:31:52.082089+0000","last_undegraded":"2024-04-22T13:31:52.082089+0000","last_fullsized":"2024-04-22T13:31:52.082089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:49.431931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":154,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:56.503404+0000","last_change":"2024-04-22T13:29:46.331601+0000","last_active":"2024-04-22T13:31:56.503404+0000","last_peered":"2024-04-22T13:31:56.503404+0000","last_clean":"2024-04-22T13:31:56.503404+0000","last_became_active":"2024-04-22T13:29:46.331087+0000","last_became_peered":"2024-04-22T13:29:46.331087+0000","last_unstale":"2024-04-22T13:31:56.503404+0000","last_undegraded":"2024-04-22T13:31:56.503404+0000","last_fullsized":"2024-04-22T13:31:56.503404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:00:15.789871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111808+0000","last_change":"2024-04-22T13:29:47.339270+0000","last_active":"2024-04-22T13:31:52.111808+0000","last_peered":"2024-04-22T13:31:52.111808+0000","last_clean":"2024-04-22T13:31:52.111808+0000","last_became_active":"2024-04-22T13:29:47.338366+0000","last_became_peered":"2024-04-22T13:29:47.338366+0000","last_unstale":"2024-04-22T13:31:52.111808+0000","last_undegraded":"2024-04-22T13:31:52.111808+0000","last_fullsized":"2024-04-22T13:31:52.111808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:34.972816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.195930+0000","last_change":"2024-04-22T13:29:46.324504+0000","last_active":"2024-04-22T13:31:52.195930+0000","last_peered":"2024-04-22T13:31:52.195930+0000","last_clean":"2024-04-22T13:31:52.195930+0000","last_became_active":"2024-04-22T13:29:46.324063+0000","last_became_peered":"2024-04-22T13:29:46.324063+0000","last_unstale":"2024-04-22T13:31:52.195930+0000","last_undegraded":"2024-04-22T13:31:52.195930+0000","last_fullsized":"2024-04-22T13:31:52.195930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:11:49.327662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:54.141174+0000","last_change":"2024-04-22T13:28:30.235182+0000","last_active":"2024-04-22T13:31:54.141174+0000","last_peered":"2024-04-22T13:31:54.141174+0000","last_clean":"2024-04-22T13:31:54.141174+0000","last_became_active":"2024-04-22T13:28:30.234593+0000","last_became_peered":"2024-04-22T13:28:30.234593+0000","last_unstale":"2024-04-22T13:31:54.141174+0000","last_undegraded":"2024-04-22T13:31:54.141174+0000","last_fullsized":"2024-04-22T13:31:54.141174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:25:01.756335+0000","last_clean_scrub_stamp":"2024-04-22T13:25:01.756335+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:45:13.759299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498175+0000","last_change":"2024-04-22T13:29:47.330969+0000","last_active":"2024-04-22T13:31:52.498175+0000","last_peered":"2024-04-22T13:31:52.498175+0000","last_clean":"2024-04-22T13:31:52.498175+0000","last_became_active":"2024-04-22T13:29:47.330596+0000","last_became_peered":"2024-04-22T13:29:47.330596+0000","last_unstale":"2024-04-22T13:31:52.498175+0000","last_undegraded":"2024-04-22T13:31:52.498175+0000","last_fullsized":"2024-04-22T13:31:52.498175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:24:55.194424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210759+0000","last_change":"2024-04-22T13:29:46.330967+0000","last_active":"2024-04-22T13:31:52.210759+0000","last_peered":"2024-04-22T13:31:52.210759+0000","last_clean":"2024-04-22T13:31:52.210759+0000","last_became_active":"2024-04-22T13:29:46.329819+0000","last_became_peered":"2024-04-22T13:29:46.329819+0000","last_unstale":"2024-04-22T13:31:52.210759+0000","last_undegraded":"2024-04-22T13:31:52.210759+0000","last_fullsized":"2024-04-22T13:31:52.210759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:43.198740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111290+0000","last_change":"2024-04-22T13:29:46.330289+0000","last_active":"2024-04-22T13:31:52.111290+0000","last_peered":"2024-04-22T13:31:52.111290+0000","last_clean":"2024-04-22T13:31:52.111290+0000","last_became_active":"2024-04-22T13:29:46.329894+0000","last_became_peered":"2024-04-22T13:29:46.329894+0000","last_unstale":"2024-04-22T13:31:52.111290+0000","last_undegraded":"2024-04-22T13:31:52.111290+0000","last_fullsized":"2024-04-22T13:31:52.111290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:29:18.554347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498409+0000","last_change":"2024-04-22T13:29:47.332047+0000","last_active":"2024-04-22T13:31:52.498409+0000","last_peered":"2024-04-22T13:31:52.498409+0000","last_clean":"2024-04-22T13:31:52.498409+0000","last_became_active":"2024-04-22T13:29:47.331718+0000","last_became_peered":"2024-04-22T13:29:47.331718+0000","last_unstale":"2024-04-22T13:31:52.498409+0000","last_undegraded":"2024-04-22T13:31:52.498409+0000","last_fullsized":"2024-04-22T13:31:52.498409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:26:09.817688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:53.032318+0000","last_change":"2024-04-22T13:29:46.323776+0000","last_active":"2024-04-22T13:31:53.032318+0000","last_peered":"2024-04-22T13:31:53.032318+0000","last_clean":"2024-04-22T13:31:53.032318+0000","last_became_active":"2024-04-22T13:29:46.323426+0000","last_became_peered":"2024-04-22T13:29:46.323426+0000","last_unstale":"2024-04-22T13:31:53.032318+0000","last_undegraded":"2024-04-22T13:31:53.032318+0000","last_fullsized":"2024-04-22T13:31:53.032318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T00:53:42.522204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112338+0000","last_change":"2024-04-22T13:29:47.340896+0000","last_active":"2024-04-22T13:31:52.112338+0000","last_peered":"2024-04-22T13:31:52.112338+0000","last_clean":"2024-04-22T13:31:52.112338+0000","last_became_active":"2024-04-22T13:29:47.340416+0000","last_became_peered":"2024-04-22T13:29:47.340416+0000","last_unstale":"2024-04-22T13:31:52.112338+0000","last_undegraded":"2024-04-22T13:31:52.112338+0000","last_fullsized":"2024-04-22T13:31:52.112338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:05:52.521891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.196097+0000","last_change":"2024-04-22T13:29:46.324912+0000","last_active":"2024-04-22T13:31:52.196097+0000","last_peered":"2024-04-22T13:31:52.196097+0000","last_clean":"2024-04-22T13:31:52.196097+0000","last_became_active":"2024-04-22T13:29:46.324507+0000","last_became_peered":"2024-04-22T13:29:46.324507+0000","last_unstale":"2024-04-22T13:31:52.196097+0000","last_undegraded":"2024-04-22T13:31:52.196097+0000","last_fullsized":"2024-04-22T13:31:52.196097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:07:06.167843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210877+0000","last_change":"2024-04-22T13:29:47.342127+0000","last_active":"2024-04-22T13:31:52.210877+0000","last_peered":"2024-04-22T13:31:52.210877+0000","last_clean":"2024-04-22T13:31:52.210877+0000","last_became_active":"2024-04-22T13:29:47.341807+0000","last_became_peered":"2024-04-22T13:29:47.341807+0000","last_unstale":"2024-04-22T13:31:52.210877+0000","last_undegraded":"2024-04-22T13:31:52.210877+0000","last_fullsized":"2024-04-22T13:31:52.210877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:34:34.607165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209960+0000","last_change":"2024-04-22T13:29:46.325077+0000","last_active":"2024-04-22T13:31:52.209960+0000","last_peered":"2024-04-22T13:31:52.209960+0000","last_clean":"2024-04-22T13:31:52.209960+0000","last_became_active":"2024-04-22T13:29:46.323943+0000","last_became_peered":"2024-04-22T13:29:46.323943+0000","last_unstale":"2024-04-22T13:31:52.209960+0000","last_undegraded":"2024-04-22T13:31:52.209960+0000","last_fullsized":"2024-04-22T13:31:52.209960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:59.676031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.110920+0000","last_change":"2024-04-22T13:29:47.338258+0000","last_active":"2024-04-22T13:31:52.110920+0000","last_peered":"2024-04-22T13:31:52.110920+0000","last_clean":"2024-04-22T13:31:52.110920+0000","last_became_active":"2024-04-22T13:29:47.337842+0000","last_became_peered":"2024-04-22T13:29:47.337842+0000","last_unstale":"2024-04-22T13:31:52.110920+0000","last_undegraded":"2024-04-22T13:31:52.110920+0000","last_fullsized":"2024-04-22T13:31:52.110920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:09:12.146342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111526+0000","last_change":"2024-04-22T13:29:46.322345+0000","last_active":"2024-04-22T13:31:52.111526+0000","last_peered":"2024-04-22T13:31:52.111526+0000","last_clean":"2024-04-22T13:31:52.111526+0000","last_became_active":"2024-04-22T13:29:46.322012+0000","last_became_peered":"2024-04-22T13:29:46.322012+0000","last_unstale":"2024-04-22T13:31:52.111526+0000","last_undegraded":"2024-04-22T13:31:52.111526+0000","last_fullsized":"2024-04-22T13:31:52.111526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:02:05.452169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021921+0000","last_change":"2024-04-22T13:29:47.330223+0000","last_active":"2024-04-22T13:31:52.021921+0000","last_peered":"2024-04-22T13:31:52.021921+0000","last_clean":"2024-04-22T13:31:52.021921+0000","last_became_active":"2024-04-22T13:29:47.329852+0000","last_became_peered":"2024-04-22T13:29:47.329852+0000","last_unstale":"2024-04-22T13:31:52.021921+0000","last_undegraded":"2024-04-22T13:31:52.021921+0000","last_fullsized":"2024-04-22T13:31:52.021921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:14:59.543498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209941+0000","last_change":"2024-04-22T13:29:46.327462+0000","last_active":"2024-04-22T13:31:52.209941+0000","last_peered":"2024-04-22T13:31:52.209941+0000","last_clean":"2024-04-22T13:31:52.209941+0000","last_became_active":"2024-04-22T13:29:46.327121+0000","last_became_peered":"2024-04-22T13:29:46.327121+0000","last_unstale":"2024-04-22T13:31:52.209941+0000","last_undegraded":"2024-04-22T13:31:52.209941+0000","last_fullsized":"2024-04-22T13:31:52.209941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:02:37.445191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111801+0000","last_change":"2024-04-22T13:29:47.349945+0000","last_active":"2024-04-22T13:31:52.111801+0000","last_peered":"2024-04-22T13:31:52.111801+0000","last_clean":"2024-04-22T13:31:52.111801+0000","last_became_active":"2024-04-22T13:29:47.349466+0000","last_became_peered":"2024-04-22T13:29:47.349466+0000","last_unstale":"2024-04-22T13:31:52.111801+0000","last_undegraded":"2024-04-22T13:31:52.111801+0000","last_fullsized":"2024-04-22T13:31:52.111801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:43:50.808175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111015+0000","last_change":"2024-04-22T13:29:46.334795+0000","last_active":"2024-04-22T13:31:52.111015+0000","last_peered":"2024-04-22T13:31:52.111015+0000","last_clean":"2024-04-22T13:31:52.111015+0000","last_became_active":"2024-04-22T13:29:46.334433+0000","last_became_peered":"2024-04-22T13:29:46.334433+0000","last_unstale":"2024-04-22T13:31:52.111015+0000","last_undegraded":"2024-04-22T13:31:52.111015+0000","last_fullsized":"2024-04-22T13:31:52.111015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:49:47.055444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210723+0000","last_change":"2024-04-22T13:29:47.335090+0000","last_active":"2024-04-22T13:31:52.210723+0000","last_peered":"2024-04-22T13:31:52.210723+0000","last_clean":"2024-04-22T13:31:52.210723+0000","last_became_active":"2024-04-22T13:29:47.334697+0000","last_became_peered":"2024-04-22T13:29:47.334697+0000","last_unstale":"2024-04-22T13:31:52.210723+0000","last_undegraded":"2024-04-22T13:31:52.210723+0000","last_fullsized":"2024-04-22T13:31:52.210723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:32:24.247281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210147+0000","last_change":"2024-04-22T13:29:46.320808+0000","last_active":"2024-04-22T13:31:52.210147+0000","last_peered":"2024-04-22T13:31:52.210147+0000","last_clean":"2024-04-22T13:31:52.210147+0000","last_became_active":"2024-04-22T13:29:46.320402+0000","last_became_peered":"2024-04-22T13:29:46.320402+0000","last_unstale":"2024-04-22T13:31:52.210147+0000","last_undegraded":"2024-04-22T13:31:52.210147+0000","last_fullsized":"2024-04-22T13:31:52.210147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:09:03.465791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210861+0000","last_change":"2024-04-22T13:29:47.335025+0000","last_active":"2024-04-22T13:31:52.210861+0000","last_peered":"2024-04-22T13:31:52.210861+0000","last_clean":"2024-04-22T13:31:52.210861+0000","last_became_active":"2024-04-22T13:29:47.334651+0000","last_became_peered":"2024-04-22T13:29:47.334651+0000","last_unstale":"2024-04-22T13:31:52.210861+0000","last_undegraded":"2024-04-22T13:31:52.210861+0000","last_fullsized":"2024-04-22T13:31:52.210861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:55:38.965640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.220503+0000","last_change":"2024-04-22T13:29:46.326569+0000","last_active":"2024-04-22T13:31:52.220503+0000","last_peered":"2024-04-22T13:31:52.220503+0000","last_clean":"2024-04-22T13:31:52.220503+0000","last_became_active":"2024-04-22T13:29:46.326171+0000","last_became_peered":"2024-04-22T13:29:46.326171+0000","last_unstale":"2024-04-22T13:31:52.220503+0000","last_undegraded":"2024-04-22T13:31:52.220503+0000","last_fullsized":"2024-04-22T13:31:52.220503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:32:56.865082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210696+0000","last_change":"2024-04-22T13:30:37.988344+0000","last_active":"2024-04-22T13:31:52.210696+0000","last_peered":"2024-04-22T13:31:52.210696+0000","last_clean":"2024-04-22T13:31:52.210696+0000","last_became_active":"2024-04-22T13:30:37.985475+0000","last_became_peered":"2024-04-22T13:30:37.985475+0000","last_unstale":"2024-04-22T13:31:52.210696+0000","last_undegraded":"2024-04-22T13:31:52.210696+0000","last_fullsized":"2024-04-22T13:31:52.210696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:18:14.087614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498373+0000","last_change":"2024-04-22T13:29:46.321036+0000","last_active":"2024-04-22T13:31:52.498373+0000","last_peered":"2024-04-22T13:31:52.498373+0000","last_clean":"2024-04-22T13:31:52.498373+0000","last_became_active":"2024-04-22T13:29:46.320553+0000","last_became_peered":"2024-04-22T13:29:46.320553+0000","last_unstale":"2024-04-22T13:31:52.498373+0000","last_undegraded":"2024-04-22T13:31:52.498373+0000","last_fullsized":"2024-04-22T13:31:52.498373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:45.303393+0000","last_clean_scrub_stamp":"2024-04-22T13:29:45.303393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:07:00.128836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210997+0000","last_change":"2024-04-22T13:29:47.330482+0000","last_active":"2024-04-22T13:31:52.210997+0000","last_peered":"2024-04-22T13:31:52.210997+0000","last_clean":"2024-04-22T13:31:52.210997+0000","last_became_active":"2024-04-22T13:29:47.330018+0000","last_became_peered":"2024-04-22T13:29:47.330018+0000","last_unstale":"2024-04-22T13:31:52.210997+0000","last_undegraded":"2024-04-22T13:31:52.210997+0000","last_fullsized":"2024-04-22T13:31:52.210997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:12:32.612139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021940+0000","last_change":"2024-04-22T13:29:47.341307+0000","last_active":"2024-04-22T13:31:52.021940+0000","last_peered":"2024-04-22T13:31:52.021940+0000","last_clean":"2024-04-22T13:31:52.021940+0000","last_became_active":"2024-04-22T13:29:47.340969+0000","last_became_peered":"2024-04-22T13:29:47.340969+0000","last_unstale":"2024-04-22T13:31:52.021940+0000","last_undegraded":"2024-04-22T13:31:52.021940+0000","last_fullsized":"2024-04-22T13:31:52.021940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:29:32.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209999+0000","last_change":"2024-04-22T13:29:47.343998+0000","last_active":"2024-04-22T13:31:52.209999+0000","last_peered":"2024-04-22T13:31:52.209999+0000","last_clean":"2024-04-22T13:31:52.209999+0000","last_became_active":"2024-04-22T13:29:47.343670+0000","last_became_peered":"2024-04-22T13:29:47.343670+0000","last_unstale":"2024-04-22T13:31:52.209999+0000","last_undegraded":"2024-04-22T13:31:52.209999+0000","last_fullsized":"2024-04-22T13:31:52.209999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:03.353626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.209923+0000","last_change":"2024-04-22T13:29:47.330994+0000","last_active":"2024-04-22T13:31:52.209923+0000","last_peered":"2024-04-22T13:31:52.209923+0000","last_clean":"2024-04-22T13:31:52.209923+0000","last_became_active":"2024-04-22T13:29:47.330639+0000","last_became_peered":"2024-04-22T13:29:47.330639+0000","last_unstale":"2024-04-22T13:31:52.209923+0000","last_undegraded":"2024-04-22T13:31:52.209923+0000","last_fullsized":"2024-04-22T13:31:52.209923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:01:25.784125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210319+0000","last_change":"2024-04-22T13:29:47.341273+0000","last_active":"2024-04-22T13:31:52.210319+0000","last_peered":"2024-04-22T13:31:52.210319+0000","last_clean":"2024-04-22T13:31:52.210319+0000","last_became_active":"2024-04-22T13:29:47.340951+0000","last_became_peered":"2024-04-22T13:29:47.340951+0000","last_unstale":"2024-04-22T13:31:52.210319+0000","last_undegraded":"2024-04-22T13:31:52.210319+0000","last_fullsized":"2024-04-22T13:31:52.210319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:56:00.653446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498342+0000","last_change":"2024-04-22T13:29:47.331290+0000","last_active":"2024-04-22T13:31:52.498342+0000","last_peered":"2024-04-22T13:31:52.498342+0000","last_clean":"2024-04-22T13:31:52.498342+0000","last_became_active":"2024-04-22T13:29:47.330950+0000","last_became_peered":"2024-04-22T13:29:47.330950+0000","last_unstale":"2024-04-22T13:31:52.498342+0000","last_undegraded":"2024-04-22T13:31:52.498342+0000","last_fullsized":"2024-04-22T13:31:52.498342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:53:55.034811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.022549+0000","last_change":"2024-04-22T13:29:47.340354+0000","last_active":"2024-04-22T13:31:52.022549+0000","last_peered":"2024-04-22T13:31:52.022549+0000","last_clean":"2024-04-22T13:31:52.022549+0000","last_became_active":"2024-04-22T13:29:47.340006+0000","last_became_peered":"2024-04-22T13:29:47.340006+0000","last_unstale":"2024-04-22T13:31:52.022549+0000","last_undegraded":"2024-04-22T13:31:52.022549+0000","last_fullsized":"2024-04-22T13:31:52.022549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:05.153445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210316+0000","last_change":"2024-04-22T13:29:47.336571+0000","last_active":"2024-04-22T13:31:52.210316+0000","last_peered":"2024-04-22T13:31:52.210316+0000","last_clean":"2024-04-22T13:31:52.210316+0000","last_became_active":"2024-04-22T13:29:47.336243+0000","last_became_peered":"2024-04-22T13:29:47.336243+0000","last_unstale":"2024-04-22T13:31:52.210316+0000","last_undegraded":"2024-04-22T13:31:52.210316+0000","last_fullsized":"2024-04-22T13:31:52.210316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:54.459822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.112032+0000","last_change":"2024-04-22T13:30:37.893674+0000","last_active":"2024-04-22T13:31:52.112032+0000","last_peered":"2024-04-22T13:31:52.112032+0000","last_clean":"2024-04-22T13:31:52.112032+0000","last_became_active":"2024-04-22T13:30:37.893356+0000","last_became_peered":"2024-04-22T13:30:37.893356+0000","last_unstale":"2024-04-22T13:31:52.112032+0000","last_undegraded":"2024-04-22T13:31:52.112032+0000","last_fullsized":"2024-04-22T13:31:52.112032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T23:31:29.309250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.081865+0000","last_change":"2024-04-22T13:29:47.337258+0000","last_active":"2024-04-22T13:31:52.081865+0000","last_peered":"2024-04-22T13:31:52.081865+0000","last_clean":"2024-04-22T13:31:52.081865+0000","last_became_active":"2024-04-22T13:29:47.336919+0000","last_became_peered":"2024-04-22T13:29:47.336919+0000","last_unstale":"2024-04-22T13:31:52.081865+0000","last_undegraded":"2024-04-22T13:31:52.081865+0000","last_fullsized":"2024-04-22T13:31:52.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T01:19:53.725310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.111411+0000","last_change":"2024-04-22T13:29:47.335579+0000","last_active":"2024-04-22T13:31:52.111411+0000","last_peered":"2024-04-22T13:31:52.111411+0000","last_clean":"2024-04-22T13:31:52.111411+0000","last_became_active":"2024-04-22T13:29:47.335280+0000","last_became_peered":"2024-04-22T13:29:47.335280+0000","last_unstale":"2024-04-22T13:31:52.111411+0000","last_undegraded":"2024-04-22T13:31:52.111411+0000","last_fullsized":"2024-04-22T13:31:52.111411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:32:41.345951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.210051+0000","last_change":"2024-04-22T13:29:47.342814+0000","last_active":"2024-04-22T13:31:52.210051+0000","last_peered":"2024-04-22T13:31:52.210051+0000","last_clean":"2024-04-22T13:31:52.210051+0000","last_became_active":"2024-04-22T13:29:47.341592+0000","last_became_peered":"2024-04-22T13:29:47.341592+0000","last_unstale":"2024-04-22T13:31:52.210051+0000","last_undegraded":"2024-04-22T13:31:52.210051+0000","last_fullsized":"2024-04-22T13:31:52.210051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T21:30:04.544535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.498443+0000","last_change":"2024-04-22T13:29:47.338699+0000","last_active":"2024-04-22T13:31:52.498443+0000","last_peered":"2024-04-22T13:31:52.498443+0000","last_clean":"2024-04-22T13:31:52.498443+0000","last_became_active":"2024-04-22T13:29:47.338164+0000","last_became_peered":"2024-04-22T13:29:47.338164+0000","last_unstale":"2024-04-22T13:31:52.498443+0000","last_undegraded":"2024-04-22T13:31:52.498443+0000","last_fullsized":"2024-04-22T13:31:52.498443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:29.726317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T13:31:52.021969+0000","last_change":"2024-04-22T13:29:47.330595+0000","last_active":"2024-04-22T13:31:52.021969+0000","last_peered":"2024-04-22T13:31:52.021969+0000","last_clean":"2024-04-22T13:31:52.021969+0000","last_became_active":"2024-04-22T13:29:47.330259+0000","last_became_peered":"2024-04-22T13:29:47.330259+0000","last_unstale":"2024-04-22T13:31:52.021969+0000","last_undegraded":"2024-04-22T13:31:52.021969+0000","last_fullsized":"2024-04-22T13:31:52.021969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T13:29:46.307612+0000","last_clean_scrub_stamp":"2024-04-22T13:29:46.307612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:51:15.686920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107235,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598718,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":890,"num_read_kb":556,"num_write":292,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1796154,"data_compressed":81432,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464910,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257467,"data_compressed":46637,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":858445,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858248,"data_compressed":22040,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1058834,"data_compressed":38639,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464501,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":532323,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":465772,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":534421,"data_compressed":4736,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":989511,"data_compressed":28196,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466419,"data_compressed":2894,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188637,"data_compressed":44795,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68029,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70127,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133858,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202688,"data_compressed":7998,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T13:32:04.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:04.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- 172.21.15.87:0/1264897911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd4341072b0 msgr2=0x7fd43410ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- 172.21.15.87:0/1264897911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd4341072b0 0x7fd43410ee50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009ca0 tx=0x7fd42c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- 172.21.15.87:0/1264897911 shutdown_connections 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- 172.21.15.87:0/1264897911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd4341072b0 0x7fd43410ee50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- 172.21.15.87:0/1264897911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd43406b280 0x7fd434106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- 172.21.15.87:0/1264897911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 0x7fd43406acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- 172.21.15.87:0/1264897911 >> 172.21.15.87:0/1264897911 conn(0x7fd434077c20 msgr2=0x7fd434078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- 172.21.15.87:0/1264897911 shutdown_connections 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- 172.21.15.87:0/1264897911 wait complete. 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 Processor -- start 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- start start 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 0x7fd434101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4341072b0 0x7fd4341017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341121f0 con 0x7fd434101cf0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd434112070 con 0x7fd43406a8d0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.153+0000 7fd4393f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd434112370 con 0x7fd4341072b0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38050/0 (socket says 172.21.15.87:38050) 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 -- 172.21.15.87:0/13969285 learned_addr learned my addr 172.21.15.87:0/13969285 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd42bfff640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4341072b0 0x7fd4341017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 -- 172.21.15.87:0/13969285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4341072b0 msgr2=0x7fd4341017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd432ffd640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 0x7fd434101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4341072b0 0x7fd4341017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 -- 172.21.15.87:0/13969285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 msgr2=0x7fd434101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 0x7fd434101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 -- 172.21.15.87:0/13969285 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4341a7040 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4337fe640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd42c0098d0 tx=0x7fd42c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c039040 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd42c044070 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4341a72d0 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0042a0 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.154+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4341a77c0 con 0x7fd434101cf0 2024-04-22T13:32:04.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.155+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd434069230 con 0x7fd434101cf0 2024-04-22T13:32:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.156+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd42c004440 con 0x7fd434101cf0 2024-04-22T13:32:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.157+0000 7fd430ff9640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 0x7fd4200a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.157+0000 7fd432ffd640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 0x7fd4200a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.157+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd42c0f2f30 con 0x7fd434101cf0 2024-04-22T13:32:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.157+0000 7fd432ffd640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 0x7fd4200a63c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4240058c0 tx=0x7fd424005c60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.161+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd42c0bf510 con 0x7fd434101cf0 2024-04-22T13:32:04.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:03 smithi132 ceph-mon[31531]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T13:32:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:03 smithi132 ceph-mon[31531]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-04-22T13:32:04.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:03 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/329291659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:04.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:03 smithi087 ceph-mon[27999]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T13:32:04.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:03 smithi087 ceph-mon[27999]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-04-22T13:32:04.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:03 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/329291659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:04.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.375+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd43406b280 con 0x7fd434101cf0 2024-04-22T13:32:04.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.376+0000 7fd430ff9640 1 -- 172.21.15.87:0/13969285 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fd42c0bee10 con 0x7fd434101cf0 2024-04-22T13:32:04.372 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:04.372 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:04.372 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 msgr2=0x7fd4200a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 0x7fd4200a63c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4240058c0 tx=0x7fd424005c60 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 msgr2=0x7fd4341a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd42c0098d0 tx=0x7fd42c035d60 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 shutdown_connections 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd4200a3f00 0x7fd4200a63c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd434101cf0 0x7fd4341a6940 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4341072b0 0x7fd4341017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 --2- 172.21.15.87:0/13969285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd43406a8d0 0x7fd434101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 >> 172.21.15.87:0/13969285 conn(0x7fd434077c20 msgr2=0x7fd4340ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 shutdown_connections 2024-04-22T13:32:04.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.378+0000 7fd4393f4640 1 -- 172.21.15.87:0/13969285 wait complete. 2024-04-22T13:32:04.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:32:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:03 smithi120 ceph-mon[31101]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T13:32:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:03 smithi120 ceph-mon[31101]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-04-22T13:32:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:03 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/329291659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 -- 172.21.15.87:0/2200030277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 msgr2=0x7f0b0410eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2200030277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0410eec0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009ca0 tx=0x7f0afc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 -- 172.21.15.87:0/2200030277 shutdown_connections 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2200030277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0410eec0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2200030277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406b260 0x7f0b04106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2200030277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b0406a8b0 0x7f0b0406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 -- 172.21.15.87:0/2200030277 >> 172.21.15.87:0/2200030277 conn(0x7f0b04077e30 msgr2=0x7f0b04078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 -- 172.21.15.87:0/2200030277 shutdown_connections 2024-04-22T13:32:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.487+0000 7f0b0923a640 1 -- 172.21.15.87:0/2200030277 wait complete. 2024-04-22T13:32:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 Processor -- start 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 -- start start 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406a8b0 0x7f0b0419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 0x7f0b04198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0b04112260 con 0x7f0b04107340 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b041120e0 con 0x7f0b04198640 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0923a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0b041123e0 con 0x7f0b0406a8b0 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38072/0 (socket says 172.21.15.87:38072) 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b0259c640 1 -- 172.21.15.87:0/2104007106 learned_addr learned my addr 172.21.15.87:0/2104007106 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:04.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.488+0000 7f0b02d9d640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406a8b0 0x7f0b0419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0359e640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 0x7f0b04198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 -- 172.21.15.87:0/2104007106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406a8b0 msgr2=0x7f0b0419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406a8b0 0x7f0b0419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 -- 172.21.15.87:0/2104007106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 msgr2=0x7f0b04198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 0x7f0b04198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 -- 172.21.15.87:0/2104007106 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b04199160 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0359e640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 0x7f0b04198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0259c640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c970 tx=0x7f0af800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004220 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af80043c0 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b04199450 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004a00 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.489+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b041a55b0 con 0x7f0b04107340 2024-04-22T13:32:04.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.490+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b04069230 con 0x7f0b04107340 2024-04-22T13:32:04.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.491+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0af80079b0 con 0x7f0b04107340 2024-04-22T13:32:04.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.492+0000 7f0aebfff640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 0x7f0aec0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.492+0000 7f0b02d9d640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 0x7f0aec0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.492+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f0af80d0750 con 0x7f0b04107340 2024-04-22T13:32:04.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.492+0000 7f0b02d9d640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 0x7f0aec0a6390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0af4005d00 tx=0x7f0af400b410 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.496+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0af8014030 con 0x7f0b04107340 2024-04-22T13:32:04.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.675+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b04199ad0 con 0x7f0b04107340 2024-04-22T13:32:04.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.677+0000 7f0aebfff640 1 -- 172.21.15.87:0/2104007106 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7f0af809cc20 con 0x7f0b04107340 2024-04-22T13:32:04.672 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:04.672 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:32:04.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.678+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 msgr2=0x7f0aec0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.678+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 0x7f0aec0a6390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0af4005d00 tx=0x7f0af400b410 comp rx=0 tx=0).stop 2024-04-22T13:32:04.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 msgr2=0x7f0b0419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c970 tx=0x7f0af800ce30 comp rx=0 tx=0).stop 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 shutdown_connections 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0aec0a3ed0 0x7f0aec0a6390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b04198640 0x7f0b04198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b04107340 0x7f0b0419e930 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 --2- 172.21.15.87:0/2104007106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b0406a8b0 0x7f0b0419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 >> 172.21.15.87:0/2104007106 conn(0x7f0b04077e30 msgr2=0x7f0b040ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 shutdown_connections 2024-04-22T13:32:04.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.679+0000 7f0b0923a640 1 -- 172.21.15.87:0/2104007106 wait complete. 2024-04-22T13:32:04.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:32:04.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3215069715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec103ad0 msgr2=0x7f5fec10bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3215069715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec103ad0 0x7f5fec10bfb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009900 tx=0x7f5fdc02bcd0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3215069715 shutdown_connections 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3215069715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec10c580 0x7f5fec10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3215069715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec103ad0 0x7f5fec10bfb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3215069715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3215069715 >> 172.21.15.87:0/3215069715 conn(0x7f5fec0771d0 msgr2=0x7f5fec0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3215069715 shutdown_connections 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.792+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3215069715 wait complete. 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 Processor -- start 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 -- start start 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 0x7f5fec198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec1121a0 con 0x7f5fec10c580 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec112020 con 0x7f5fec103ad0 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff3c11640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec112320 con 0x7f5fec1031b0 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38086/0 (socket says 172.21.15.87:38086) 2024-04-22T13:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 -- 172.21.15.87:0/3849114261 learned_addr learned my addr 172.21.15.87:0/3849114261 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff19ad640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff11ac640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 0x7f5fec198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 -- 172.21.15.87:0/3849114261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 msgr2=0x7f5fec198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 -- 172.21.15.87:0/3849114261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 msgr2=0x7f5fec198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 0x7f5fec198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff21ae640 1 -- 172.21.15.87:0/3849114261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec19d730 con 0x7f5fec10c580 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff19ad640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.793+0000 7f5ff11ac640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 0x7f5fec198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5ff21ae640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400e9d0 tx=0x7f5fe400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400cd80 con 0x7f5fec10c580 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fe4005020 con 0x7f5fec10c580 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec19da20 con 0x7f5fec10c580 2024-04-22T13:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40116d0 con 0x7f5fec10c580 2024-04-22T13:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.794+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fec1a52b0 con 0x7f5fec10c580 2024-04-22T13:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.795+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fec06ad70 con 0x7f5fec10c580 2024-04-22T13:32:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.796+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f5fe40051d0 con 0x7f5fec10c580 2024-04-22T13:32:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.796+0000 7f5fe2ffd640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 0x7f5fd80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.797+0000 7f5ff19ad640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 0x7f5fd80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.797+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5fe40d0cb0 con 0x7f5fec10c580 2024-04-22T13:32:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.797+0000 7f5ff19ad640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 0x7f5fd80a6390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80058c0 tx=0x7f5fe8005ec0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:04.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.801+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fe4016030 con 0x7f5fec10c580 2024-04-22T13:32:04.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.980+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fec199a90 con 0x7f5fec10c580 2024-04-22T13:32:04.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.980+0000 7f5fe2ffd640 1 -- 172.21.15.87:0/3849114261 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5fe409d180 con 0x7f5fec10c580 2024-04-22T13:32:04.976 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:32:04.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 msgr2=0x7f5fd80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 0x7f5fd80a6390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80058c0 tx=0x7f5fe8005ec0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 msgr2=0x7f5fec19d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:04.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400e9d0 tx=0x7f5fe400ee90 comp rx=0 tx=0).stop 2024-04-22T13:32:04.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 shutdown_connections 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5fd80a3ed0 0x7f5fd80a6390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fec10c580 0x7f5fec19d000 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fec103ad0 0x7f5fec198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 --2- 172.21.15.87:0/3849114261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fec1031b0 0x7f5fec198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 >> 172.21.15.87:0/3849114261 conn(0x7f5fec0771d0 msgr2=0x7f5fec0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 shutdown_connections 2024-04-22T13:32:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:04.982+0000 7f5ff3c11640 1 -- 172.21.15.87:0/3849114261 wait complete. 2024-04-22T13:32:04.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:32:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 -- 172.21.15.87:0/1829929546 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c101a40 msgr2=0x7f163c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 --2- 172.21.15.87:0/1829929546 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c101a40 0x7f163c105a80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f162c009900 tx=0x7f162c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 -- 172.21.15.87:0/1829929546 shutdown_connections 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 --2- 172.21.15.87:0/1829929546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 0x7f163c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 --2- 172.21.15.87:0/1829929546 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c101a40 0x7f163c105a80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 --2- 172.21.15.87:0/1829929546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 0x7f163c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 -- 172.21.15.87:0/1829929546 >> 172.21.15.87:0/1829929546 conn(0x7f163c0771d0 msgr2=0x7f163c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 -- 172.21.15.87:0/1829929546 shutdown_connections 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.094+0000 7f16437f4640 1 -- 172.21.15.87:0/1829929546 wait complete. 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 Processor -- start 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 -- start start 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 0x7f163c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 0x7f163c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f163c1110a0 con 0x7f163c19a0e0 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f163c110f20 con 0x7f163c106160 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f16437f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f163c111220 con 0x7f163c101090 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38098/0 (socket says 172.21.15.87:38098) 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 -- 172.21.15.87:0/1478177929 learned_addr learned my addr 172.21.15.87:0/1478177929 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 -- 172.21.15.87:0/1478177929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 msgr2=0x7f163c199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641590640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 0x7f163c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1640d8f640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 0x7f163c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 0x7f163c199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 -- 172.21.15.87:0/1478177929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 msgr2=0x7f163c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 0x7f163c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.095+0000 7f1641d91640 1 -- 172.21.15.87:0/1478177929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c1a4e30 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f1641d91640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f163400eaa0 tx=0x7f163400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163400cdb0 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1634005020 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c1a5120 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16340116d0 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.096+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c1a5670 con 0x7f163c19a0e0 2024-04-22T13:32:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.097+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c102560 con 0x7f163c19a0e0 2024-04-22T13:32:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.098+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f16340051d0 con 0x7f163c19a0e0 2024-04-22T13:32:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.098+0000 7f16327fc640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 0x7f16280a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.099+0000 7f1641590640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 0x7f16280a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.099+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f16340d10d0 con 0x7f163c19a0e0 2024-04-22T13:32:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.099+0000 7f1641590640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 0x7f16280a6440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16380058c0 tx=0x7f1638005c60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.102+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1634016030 con 0x7f163c19a0e0 2024-04-22T13:32:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:04 smithi120 ceph-mon[31101]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:04 smithi120 ceph-mon[31101]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:04 smithi120 ceph-mon[31101]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:04 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/13969285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:04 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2104007106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:04 smithi132 ceph-mon[31531]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:04 smithi132 ceph-mon[31531]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:04 smithi132 ceph-mon[31531]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:04 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/13969285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:04 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2104007106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:04 smithi087 ceph-mon[27999]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:04 smithi087 ceph-mon[27999]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:04 smithi087 ceph-mon[27999]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T13:32:05.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:04 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/13969285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:04 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2104007106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:05.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.280+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f163c101a40 con 0x7f163c19a0e0 2024-04-22T13:32:05.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.280+0000 7f16327fc640 1 -- 172.21.15.87:0/1478177929 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f163409d540 con 0x7f163c19a0e0 2024-04-22T13:32:05.276 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:32:05.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 msgr2=0x7f16280a6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 0x7f16280a6440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16380058c0 tx=0x7f1638005c60 comp rx=0 tx=0).stop 2024-04-22T13:32:05.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 msgr2=0x7f163c1a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f163400eaa0 tx=0x7f163400ef60 comp rx=0 tx=0).stop 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 shutdown_connections 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16280a3f80 0x7f16280a6440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f163c19a0e0 0x7f163c1a46e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f163c106160 0x7f163c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 --2- 172.21.15.87:0/1478177929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f163c101090 0x7f163c199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.282+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 >> 172.21.15.87:0/1478177929 conn(0x7f163c0771d0 msgr2=0x7f163c0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.283+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 shutdown_connections 2024-04-22T13:32:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.283+0000 7f16437f4640 1 -- 172.21.15.87:0/1478177929 wait complete. 2024-04-22T13:32:05.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 -- 172.21.15.87:0/187433437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698107030 msgr2=0x7ff698107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 --2- 172.21.15.87:0/187433437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698107030 0x7ff698107410 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff688009930 tx=0x7ff68802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 -- 172.21.15.87:0/187433437 shutdown_connections 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 --2- 172.21.15.87:0/187433437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff698101b40 0x7ff69810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 --2- 172.21.15.87:0/187433437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6981011a0 0x7ff698101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 --2- 172.21.15.87:0/187433437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698107030 0x7ff698107410 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 -- 172.21.15.87:0/187433437 >> 172.21.15.87:0/187433437 conn(0x7ff698076c50 msgr2=0x7ff698077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 -- 172.21.15.87:0/187433437 shutdown_connections 2024-04-22T13:32:05.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.395+0000 7ff69e46d640 1 -- 172.21.15.87:0/187433437 wait complete. 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 Processor -- start 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 -- start start 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6981011a0 0x7ff69819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698107030 0x7ff6981985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff698112260 con 0x7ff698101b40 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981120e0 con 0x7ff6981011a0 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69e46d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981123e0 con 0x7ff698107030 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38110/0 (socket says 172.21.15.87:38110) 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 -- 172.21.15.87:0/1571181234 learned_addr learned my addr 172.21.15.87:0/1571181234 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:05.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff697fff640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6981011a0 0x7ff69819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff69ca0a640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698107030 0x7ff6981985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 -- 172.21.15.87:0/1571181234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698107030 msgr2=0x7ff6981985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698107030 0x7ff6981985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 -- 172.21.15.87:0/1571181234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6981011a0 msgr2=0x7ff69819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6981011a0 0x7ff69819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.396+0000 7ff6977fe640 1 -- 172.21.15.87:0/1571181234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff698198d60 con 0x7ff698101b40 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff6977fe640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00d770 tx=0x7ff68c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c017070 con 0x7ff698101b40 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff698199050 con 0x7ff698101b40 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff68c0054d0 con 0x7ff698101b40 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c004c30 con 0x7ff698101b40 2024-04-22T13:32:05.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.397+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6981a52f0 con 0x7ff698101b40 2024-04-22T13:32:05.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.398+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff698102590 con 0x7ff698101b40 2024-04-22T13:32:05.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.399+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7ff68c004e10 con 0x7ff698101b40 2024-04-22T13:32:05.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.399+0000 7ff6957fa640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 0x7ff6800a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.400+0000 7ff697fff640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 0x7ff6800a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.400+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff68c0d0540 con 0x7ff698101b40 2024-04-22T13:32:05.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.400+0000 7ff697fff640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 0x7ff6800a6390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff688005ab0 tx=0x7ff688005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.404+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff68c09ca10 con 0x7ff698101b40 2024-04-22T13:32:05.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.581+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff698109320 con 0x7ff698101b40 2024-04-22T13:32:05.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.582+0000 7ff6957fa640 1 -- 172.21.15.87:0/1571181234 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff68c09c310 con 0x7ff698101b40 2024-04-22T13:32:05.577 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:32:05.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 msgr2=0x7ff6800a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 0x7ff6800a6390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff688005ab0 tx=0x7ff688005fd0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 msgr2=0x7ff69819e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00d770 tx=0x7ff68c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 shutdown_connections 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6800a3ed0 0x7ff6800a6390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698107030 0x7ff6981985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff698101b40 0x7ff69819e970 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 --2- 172.21.15.87:0/1571181234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6981011a0 0x7ff69819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 >> 172.21.15.87:0/1571181234 conn(0x7ff698076c50 msgr2=0x7ff6980fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 shutdown_connections 2024-04-22T13:32:05.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.584+0000 7ff69e46d640 1 -- 172.21.15.87:0/1571181234 wait complete. 2024-04-22T13:32:05.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:05.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 -- 172.21.15.87:0/17470782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06b280 msgr2=0x7f440c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 --2- 172.21.15.87:0/17470782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06b280 0x7f440c106b80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009900 tx=0x7f43fc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 -- 172.21.15.87:0/17470782 shutdown_connections 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 --2- 172.21.15.87:0/17470782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c107260 0x7f440c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 --2- 172.21.15.87:0/17470782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06b280 0x7f440c106b80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 --2- 172.21.15.87:0/17470782 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c06a8d0 0x7f440c06acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 -- 172.21.15.87:0/17470782 >> 172.21.15.87:0/17470782 conn(0x7f440c077c20 msgr2=0x7f440c078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 -- 172.21.15.87:0/17470782 shutdown_connections 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.692+0000 7f441390d640 1 -- 172.21.15.87:0/17470782 wait complete. 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 Processor -- start 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 -- start start 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 0x7f440c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 0x7f440c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f440c1121f0 con 0x7f440c06a8d0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f440c112070 con 0x7f440c19cfb0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f441390d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f440c112370 con 0x7f440c107260 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f44116a9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f44116a9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38126/0 (socket says 172.21.15.87:38126) 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f44116a9640 1 -- 172.21.15.87:0/2376198966 learned_addr learned my addr 172.21.15.87:0/2376198966 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f4410ea8640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 0x7f440c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.693+0000 7f4411eaa640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 0x7f440c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 -- 172.21.15.87:0/2376198966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 msgr2=0x7f440c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 0x7f440c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 -- 172.21.15.87:0/2376198966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 msgr2=0x7f440c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 0x7f440c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 -- 172.21.15.87:0/2376198966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c19daa0 con 0x7f440c06a8d0 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f4411eaa640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 0x7f440c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:05.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f4410ea8640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 0x7f440c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44116a9640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f440800d770 tx=0x7f440800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408017070 con 0x7f440c06a8d0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c19dd90 con 0x7f440c06a8d0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44080054d0 con 0x7f440c06a8d0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408004c30 con 0x7f440c06a8d0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.694+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c1a5500 con 0x7f440c06a8d0 2024-04-22T13:32:05.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.695+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c069230 con 0x7f440c06a8d0 2024-04-22T13:32:05.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.696+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4408004e10 con 0x7f440c06a8d0 2024-04-22T13:32:05.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.696+0000 7f44027fc640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 0x7f43f80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:05.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.697+0000 7f4410ea8640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 0x7f43f80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:05.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.697+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f44080d05f0 con 0x7f440c06a8d0 2024-04-22T13:32:05.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.697+0000 7f4410ea8640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 0x7f43f80a6390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43fc02c240 tx=0x7f43fc005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:05.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.701+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f440809cac0 con 0x7f440c06a8d0 2024-04-22T13:32:05.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.914+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f440c06b280 con 0x7f440c06a8d0 2024-04-22T13:32:05.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.914+0000 7f44027fc640 1 -- 172.21.15.87:0/2376198966 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f440809c3c0 con 0x7f440c06a8d0 2024-04-22T13:32:05.910 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:05.910 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:05.911 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:05.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 msgr2=0x7f43f80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 0x7f43f80a6390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43fc02c240 tx=0x7f43fc005fd0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 msgr2=0x7f440c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f440800d770 tx=0x7f440800dc30 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 shutdown_connections 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f43f80a3ed0 0x7f43f80a6390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f440c19cfb0 0x7f440c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f440c107260 0x7f440c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 --2- 172.21.15.87:0/2376198966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f440c06a8d0 0x7f440c1986f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.916+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 >> 172.21.15.87:0/2376198966 conn(0x7f440c077c20 msgr2=0x7f440c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.917+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 shutdown_connections 2024-04-22T13:32:05.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:05.917+0000 7f441390d640 1 -- 172.21.15.87:0/2376198966 wait complete. 2024-04-22T13:32:05.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:32:06.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 -- 172.21.15.87:0/2380880407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84102b50 msgr2=0x7f0b84106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 --2- 172.21.15.87:0/2380880407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84102b50 0x7f0b84106b90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009900 tx=0x7f0b7802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:06.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 -- 172.21.15.87:0/2380880407 shutdown_connections 2024-04-22T13:32:06.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 --2- 172.21.15.87:0/2380880407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b84107270 0x7f0b8410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 --2- 172.21.15.87:0/2380880407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84102b50 0x7f0b84106b90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 --2- 172.21.15.87:0/2380880407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 0x7f0b84102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 -- 172.21.15.87:0/2380880407 >> 172.21.15.87:0/2380880407 conn(0x7f0b840793a0 msgr2=0x7f0b840797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 -- 172.21.15.87:0/2380880407 shutdown_connections 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 -- 172.21.15.87:0/2380880407 wait complete. 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.026+0000 7f0b89fca640 1 Processor -- start 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 -- start start 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 0x7f0b84198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 0x7f0b8419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0b841121b0 con 0x7f0b84107270 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b84112030 con 0x7f0b8419cff0 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b89fca640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0b84112330 con 0x7f0b841021a0 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38142/0 (socket says 172.21.15.87:38142) 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 -- 172.21.15.87:0/1085434690 learned_addr learned my addr 172.21.15.87:0/1085434690 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b837fe640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 0x7f0b84198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b83fff640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 0x7f0b8419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 -- 172.21.15.87:0/1085434690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 msgr2=0x7f0b84198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 0x7f0b84198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 -- 172.21.15.87:0/1085434690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 msgr2=0x7f0b8419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 0x7f0b8419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 -- 172.21.15.87:0/1085434690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8419dae0 con 0x7f0b84107270 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b83fff640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 0x7f0b8419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:06.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.027+0000 7f0b82ffd640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b78036040 tx=0x7f0b7802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.028+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78043020 con 0x7f0b84107270 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.028+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b7802cbe0 con 0x7f0b84107270 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.028+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8419dd70 con 0x7f0b84107270 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.028+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78003bf0 con 0x7f0b84107270 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.028+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b841a5590 con 0x7f0b84107270 2024-04-22T13:32:06.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.029+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b84069230 con 0x7f0b84107270 2024-04-22T13:32:06.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.030+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0b78046020 con 0x7f0b84107270 2024-04-22T13:32:06.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.030+0000 7f0b80ff9640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 0x7f0b6c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.030+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f0b780f31e0 con 0x7f0b84107270 2024-04-22T13:32:06.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.030+0000 7f0b837fe640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 0x7f0b6c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.031+0000 7f0b837fe640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 0x7f0b6c0a6390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0b74005fd0 tx=0x7f0b7400b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.034+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b7804d020 con 0x7f0b84107270 2024-04-22T13:32:06.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.212+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b84199a80 con 0x7f0b84107270 2024-04-22T13:32:06.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.214+0000 7f0b80ff9640 1 -- 172.21.15.87:0/1085434690 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22954 (secure 0 0 0) 0x7f0b84199a80 con 0x7f0b84107270 2024-04-22T13:32:06.210 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:06.210 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":88,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:31:52.016468+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:32:06.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 msgr2=0x7f0b6c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 0x7f0b6c0a6390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0b74005fd0 tx=0x7f0b7400b390 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 msgr2=0x7f0b84198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b78036040 tx=0x7f0b7802c7e0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 shutdown_connections 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b6c0a3ed0 0x7f0b6c0a6390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8419cff0 0x7f0b8419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b84107270 0x7f0b84198c70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 --2- 172.21.15.87:0/1085434690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b841021a0 0x7f0b84198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 >> 172.21.15.87:0/1085434690 conn(0x7f0b840793a0 msgr2=0x7f0b840fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 shutdown_connections 2024-04-22T13:32:06.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.216+0000 7f0b89fca640 1 -- 172.21.15.87:0/1085434690 wait complete. 2024-04-22T13:32:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:05 smithi132 ceph-mon[31531]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-22T13:32:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3849114261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1478177929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1571181234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2376198966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:06.221 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:05 smithi087 ceph-mon[27999]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-22T13:32:06.222 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3849114261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.222 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1478177929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.222 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1571181234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.222 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2376198966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:06.223 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 832672 2024-04-22T13:32:06.224 DEBUG:tasks.quiescer:Yielding 2024-04-22T13:32:06.224 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T13:32:06.251 INFO:tasks.workunit:Pulling workunits from ref d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:32:06.251 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T13:32:06.252 INFO:tasks.workunit:timeout=3h 2024-04-22T13:32:06.252 INFO:tasks.workunit:cleanup=True 2024-04-22T13:32:06.252 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T13:32:06.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:06.267 INFO:teuthology.orchestra.run.smithi087.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T13:32:06.267 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-04-22 13:29:52.689770935 +0000 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-04-22 13:29:52.689770935 +0000 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-04-22 13:31:50.141869088 +0000 2024-04-22T13:32:06.268 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-04-22 13:29:52.689770935 +0000 2024-04-22T13:32:06.268 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T13:32:06.269 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T13:32:06.338 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- 172.21.15.87:0/2309194013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25681011a0 msgr2=0x7f2568101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 --2- 172.21.15.87:0/2309194013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25681011a0 0x7f2568101620 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2558009930 tx=0x7f255802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- 172.21.15.87:0/2309194013 shutdown_connections 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 --2- 172.21.15.87:0/2309194013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2568101b60 0x7f256810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 --2- 172.21.15.87:0/2309194013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25681011a0 0x7f2568101620 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 --2- 172.21.15.87:0/2309194013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568105e30 0x7f2568106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- 172.21.15.87:0/2309194013 >> 172.21.15.87:0/2309194013 conn(0x7f2568077e30 msgr2=0x7f2568078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- 172.21.15.87:0/2309194013 shutdown_connections 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- 172.21.15.87:0/2309194013 wait complete. 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 Processor -- start 2024-04-22T13:32:06.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.359+0000 7f257029a640 1 -- start start 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 0x7f25681975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 0x7f2568197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2568111130 con 0x7f2568105e30 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2568110fb0 con 0x7f2568101b60 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f25681112b0 con 0x7f25681011a0 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38146/0 (socket says 172.21.15.87:38146) 2024-04-22T13:32:06.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 -- 172.21.15.87:0/1963808284 learned_addr learned my addr 172.21.15.87:0/1963808284 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e036640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 0x7f25681975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256d835640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 0x7f2568197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 -- 172.21.15.87:0/1963808284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 msgr2=0x7f25681975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 0x7f25681975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 -- 172.21.15.87:0/1963808284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 msgr2=0x7f2568197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 0x7f2568197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 -- 172.21.15.87:0/1963808284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256819c570 con 0x7f2568105e30 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e036640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 0x7f25681975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256d835640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 0x7f2568197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f256e837640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f256000e970 tx=0x7f256000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256000ccc0 con 0x7f2568105e30 2024-04-22T13:32:06.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256819c860 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f256000ce60 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560010430 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.360+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25681a40f0 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.361+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2568068de0 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.362+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f2560004f90 con 0x7f2568105e30 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.363+0000 7f255f7fe640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 0x7f25540a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.363+0000 7f256e036640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 0x7f25540a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.363+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f25600d0f70 con 0x7f2568105e30 2024-04-22T13:32:06.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.363+0000 7f256e036640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 0x7f25540a6390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2564007e70 tx=0x7f2564007850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.368+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f256009d440 con 0x7f2568105e30 2024-04-22T13:32:06.370 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T13:32:06.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:05 smithi120 ceph-mon[31101]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-22T13:32:06.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3849114261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1478177929' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1571181234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:32:06.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2376198966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:06.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.590+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2568108dc0 con 0x7f2568105e30 2024-04-22T13:32:06.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.590+0000 7f255f7fe640 1 -- 172.21.15.87:0/1963808284 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f256009cd40 con 0x7f2568105e30 2024-04-22T13:32:06.586 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:06.586 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:06.587 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 msgr2=0x7f25540a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 0x7f25540a6390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2564007e70 tx=0x7f2564007850 comp rx=0 tx=0).stop 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 msgr2=0x7f256819be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f256000e970 tx=0x7f256000ed70 comp rx=0 tx=0).stop 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 shutdown_connections 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f25540a3ed0 0x7f25540a6390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2568105e30 0x7f256819be40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101b60 0x7f2568197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 --2- 172.21.15.87:0/1963808284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25681011a0 0x7f25681975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 >> 172.21.15.87:0/1963808284 conn(0x7f2568077e30 msgr2=0x7f25680feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 shutdown_connections 2024-04-22T13:32:06.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.592+0000 7f257029a640 1 -- 172.21.15.87:0/1963808284 wait complete. 2024-04-22T13:32:06.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T13:29:46.666137+0000', 'modified': '2024-04-22T13:31:50.833203+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24482, 'mds_1': 24475, 'mds_2': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24407': {'gid': 24407, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1829745193', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1829745193}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1829745193}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6837/3391616456', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 3391616456}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 3391616456}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6835/1119314155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1119314155}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1119314155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.132:6837/453938062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 453938062}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 453938062}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.132:6839/2904643536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2904643536}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2904643536}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6839/2692231328', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 2692231328}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 2692231328}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24482, 'qdb_cluster': [24460, 24475, 24482]} 2024-04-22T13:32:06.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T13:32:06.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T13:32:06.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- 172.21.15.87:0/1474800300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0106160 msgr2=0x7f17d010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 --2- 172.21.15.87:0/1474800300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0106160 0x7f17d010dd30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009ca0 tx=0x7f17c802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- 172.21.15.87:0/1474800300 shutdown_connections 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 --2- 172.21.15.87:0/1474800300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0106160 0x7f17d010dd30 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 --2- 172.21.15.87:0/1474800300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d0101a40 0x7f17d0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 --2- 172.21.15.87:0/1474800300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0101090 0x7f17d0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- 172.21.15.87:0/1474800300 >> 172.21.15.87:0/1474800300 conn(0x7f17d0077f50 msgr2=0x7f17d0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- 172.21.15.87:0/1474800300 shutdown_connections 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- 172.21.15.87:0/1474800300 wait complete. 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 Processor -- start 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 -- start start 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.750+0000 7f17d8643640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 0x7f17d0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d019cfa0 0x7f17d019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f17d01121e0 con 0x7f17d0101090 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0112060 con 0x7f17d0106160 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0112360 con 0x7f17d019cfa0 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38174/0 (socket says 172.21.15.87:38174) 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 -- 172.21.15.87:0/2885404166 learned_addr learned my addr 172.21.15.87:0/2885404166 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d6be0640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d019cfa0 0x7f17d019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d5bde640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 0x7f17d0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 -- 172.21.15.87:0/2885404166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d019cfa0 msgr2=0x7f17d019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d019cfa0 0x7f17d019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 -- 172.21.15.87:0/2885404166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 msgr2=0x7f17d0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 0x7f17d0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 -- 172.21.15.87:0/2885404166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d019da90 con 0x7f17d0101090 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d5bde640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 0x7f17d0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:06.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d63df640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00d950 tx=0x7f17cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17c77fe640 1 -- 172.21.15.87:0/2885404166 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc017030 con 0x7f17d0101090 2024-04-22T13:32:06.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d019dd80 con 0x7f17d0101090 2024-04-22T13:32:06.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17c77fe640 1 -- 172.21.15.87:0/2885404166 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17cc004020 con 0x7f17d0101090 2024-04-22T13:32:06.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17c77fe640 1 -- 172.21.15.87:0/2885404166 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0047a0 con 0x7f17d0101090 2024-04-22T13:32:06.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.751+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d01a5570 con 0x7f17d0101090 2024-04-22T13:32:06.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.754+0000 7f17c77fe640 1 -- 172.21.15.87:0/2885404166 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f17cc009a30 con 0x7f17d0101090 2024-04-22T13:32:06.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.754+0000 7f17db235740 1 Processor -- start 2024-04-22T13:32:06.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.754+0000 7f17db235740 1 -- start start 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177264b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x56517726d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177270130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5651771c8b10 con 0x5651772647b0 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565177166240 con 0x565177265160 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17db235740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5651771c4ad0 con 0x56517726dce0 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17c77fe640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 0x7f17bc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177270130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d6be0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177264b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d6be0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177264b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38178/0 (socket says 172.21.15.87:38178) 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d6be0640 1 -- 172.21.15.87:0/2868144557 learned_addr learned my addr 172.21.15.87:0/2868144557 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d63df640 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x56517726d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d6be0640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 0x7f17bc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17c77fe640 1 -- 172.21.15.87:0/2885404166 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f17cc0d0590 con 0x7f17d0101090 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 -- 172.21.15.87:0/2868144557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 msgr2=0x56517726d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x56517726d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 -- 172.21.15.87:0/2868144557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 msgr2=0x565177264b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177264b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 -- 172.21.15.87:0/2868144557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565176edb2d0 con 0x56517726dce0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d6be0640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 0x7f17bc0a6390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009a60 tx=0x7f17c80095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.755+0000 7f17d5bde640 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177270130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17c00058c0 tx=0x7f17c000bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc57f98a7623d683 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17c4bfe640 1 -- 172.21.15.87:0/2868144557 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c000de20 con 0x56517726dce0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17c4bfe640 1 -- 172.21.15.87:0/2868144557 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c00178d0 con 0x56517726dce0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17c4bfe640 1 -- 172.21.15.87:0/2868144557 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c000f8f0 con 0x56517726dce0 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 -- 172.21.15.87:0/2868144557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 msgr2=0x565177270130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177270130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17c00058c0 tx=0x7f17c000bec0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 -- 172.21.15.87:0/2868144557 shutdown_connections 2024-04-22T13:32:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177270130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x56517726d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 --2- 172.21.15.87:0/2868144557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177264b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 -- 172.21.15.87:0/2868144557 >> 172.21.15.87:0/2868144557 conn(0x56517725ea60 msgr2=0x56517725ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 -- 172.21.15.87:0/2868144557 shutdown_connections 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.756+0000 7f17db235740 1 -- 172.21.15.87:0/2868144557 wait complete. 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 -- start start 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 Processor -- start 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x56517726b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x56517726b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x565177270c50 con 0x5651772647b0 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565177151190 con 0x56517726dce0 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565177270dd0 con 0x565177265160 2024-04-22T13:32:06.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17db235740 10 client.0 _ll_register_callbacks cb 0x56517714c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38180/0 (socket says 172.21.15.87:38180) 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 -- 172.21.15.87:0/1726507159 learned_addr learned my addr 172.21.15.87:0/1726507159 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d5bde640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x56517726b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d6be0640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x56517726b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 -- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 msgr2=0x56517726b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x56517726b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 -- 172.21.15.87:0/1726507159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 msgr2=0x56517726b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x56517726b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.757+0000 7f17d63df640 1 -- 172.21.15.87:0/1726507159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565176fa56b0 con 0x5651772647b0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17d63df640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17cc098d30 tx=0x7f17cc005100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c5017112b3bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:06.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09ca60 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f17cc0041d0 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c00178d0 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f17cc09c360 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 10 client.15681 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56517727cf30 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c00178d0 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f17cc0989d0 con 0x5651772647b0 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 10 client.15681 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 10 client.15681 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 20 client.15681 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56517727f570 0x5651772809e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 4 client.15681 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:06.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.758+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651771d6530 con 0x56517727f570 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.759+0000 7f17d5bde640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56517727f570 0x5651772809e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.759+0000 7f17b9ffb640 10 client.15681 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.759+0000 7f17d5bde640 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56517727f570 0x5651772809e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e51a5412b4a88 server_cookie=70c43d6e042ac7b6 in_seq=0 out_seq=0 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17b9ffb640 1 -- 172.21.15.87:0/1726507159 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651771d6530 con 0x56517727f570 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17b9ffb640 10 client.15681 handle_command_reply: tid=0 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 client.15681 shutdown 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 msgr2=0x56517726ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f17cc098d30 tx=0x7f17cc005100 comp rx=0 tx=0).stop 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 shutdown_connections 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56517727f570 0x5651772809e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x56517726b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x56517726b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17b9ffb640 0 client.15681 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 --2- 172.21.15.87:0/1726507159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726ac70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 >> 172.21.15.87:0/1726507159 conn(0x56517725ea60 msgr2=0x565177260bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 shutdown_connections 2024-04-22T13:32:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 1 -- 172.21.15.87:0/1726507159 wait complete. 2024-04-22T13:32:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.760+0000 7f17db235740 20 client.15681 trim_cache size 0 max 16384 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 Processor -- start 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 -- start start 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x565177269560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x565177269aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56517727ed20 con 0x5651772647b0 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56517727f3f0 con 0x56517726dce0 2024-04-22T13:32:06.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.783+0000 7f17db235740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56517727fac0 con 0x565177265160 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38196/0 (socket says 172.21.15.87:38196) 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 -- 172.21.15.87:0/3943999584 learned_addr learned my addr 172.21.15.87:0/3943999584 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d6be0640 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x565177269aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d5bde640 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x565177269560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 -- 172.21.15.87:0/3943999584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 msgr2=0x565177269560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x565177269560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 -- 172.21.15.87:0/3943999584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 msgr2=0x565177269aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x565177269aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 -- 172.21.15.87:0/3943999584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651772691e0 con 0x5651772647b0 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17d63df640 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cdc00 tx=0x7f17cc09a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f688529505f994 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17bbfff640 1 -- 172.21.15.87:0/3943999584 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09c130 con 0x5651772647b0 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17bbfff640 1 -- 172.21.15.87:0/3943999584 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17cc09ca60 con 0x5651772647b0 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.784+0000 7f17bbfff640 1 -- 172.21.15.87:0/3943999584 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09bcc0 con 0x5651772647b0 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- 172.21.15.87:0/3943999584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 msgr2=0x56517726d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cdc00 tx=0x7f17cc09a870 comp rx=0 tx=0).stop 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- 172.21.15.87:0/3943999584 shutdown_connections 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517726dce0 0x565177269aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565177265160 0x565177269560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 --2- 172.21.15.87:0/3943999584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x56517726d3a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- 172.21.15.87:0/3943999584 >> 172.21.15.87:0/3943999584 conn(0x56517725ea60 msgr2=0x565177259800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- 172.21.15.87:0/3943999584 shutdown_connections 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- 172.21.15.87:0/3943999584 wait complete. 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 -- start start 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.785+0000 7f17db235740 1 Processor -- start 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x565177289070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177283ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x565177280860 con 0x5651772647b0 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565177280190 con 0x565177265160 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565177280f30 con 0x56517726dce0 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 10 client.0 _ll_register_callbacks cb 0x7f17cc0044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38208/0 (socket says 172.21.15.87:38208) 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 -- 172.21.15.87:0/677986069 learned_addr learned my addr 172.21.15.87:0/677986069 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d5bde640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x565177289070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d6be0640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177283ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 -- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 msgr2=0x565177283ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177283ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 -- 172.21.15.87:0/677986069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 msgr2=0x565177289070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x565177289070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 -- 172.21.15.87:0/677986069 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565177284020 con 0x5651772647b0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d6be0640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177283ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17d63df640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f17cc004770 tx=0x7f17cc004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e2652a5937bce6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09bcc0 con 0x5651772647b0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f17cc018d60 con 0x5651772647b0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.786+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17cc09ca60 con 0x5651772647b0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f17cc09c130 con 0x5651772647b0 2024-04-22T13:32:06.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 10 client.15687 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5651772e5d00 con 0x5651772647b0 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17cc09ca60 con 0x5651772647b0 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f17cc0980b0 con 0x5651772647b0 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 10 client.15687 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 10 client.15687 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 20 client.15687 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5651772ea260 0x5651772e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 4 client.15687 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "status"}) -- 0x5651771d03a0 con 0x5651772ea260 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.787+0000 7f17d5bde640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5651772ea260 0x5651772e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17b9ffb640 10 client.15687 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17d5bde640 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5651772ea260 0x5651772e8f70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19a1dca23cb14a6b server_cookie=6040294527abe7eb in_seq=0 out_seq=0 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17b9ffb640 1 -- 172.21.15.87:0/677986069 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5651771d03a0 con 0x5651772ea260 2024-04-22T13:32:06.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17b9ffb640 10 client.15687 handle_command_reply: tid=0 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 client.15687 shutdown 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 msgr2=0x565177288b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f17cc004770 tx=0x7f17cc004580 comp rx=0 tx=0).stop 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 shutdown_connections 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5651772ea260 0x5651772e8f70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56517726dce0 0x565177283ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17b9ffb640 0 client.15687 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565177265160 0x565177289070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 --2- 172.21.15.87:0/677986069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5651772647b0 0x565177288b30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 >> 172.21.15.87:0/677986069 conn(0x56517725ea60 msgr2=0x5651772693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 shutdown_connections 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 1 -- 172.21.15.87:0/677986069 wait complete. 2024-04-22T13:32:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.788+0000 7f17db235740 20 client.15687 trim_cache size 0 max 16384 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 2, 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24460, 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 136.46889809300001, 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 21, 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 88, 2024-04-22T13:32:06.792 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T13:32:06.793 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 160.17177521799999 2024-04-22T13:32:06.793 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 msgr2=0x7f17bc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 0x7f17bc0a6390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009a60 tx=0x7f17c80095c0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 msgr2=0x7f17d01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00d950 tx=0x7f17cc00de10 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 shutdown_connections 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17bc0a3ed0 0x7f17bc0a6390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17d019cfa0 0x7f17d019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d0106160 0x7f17d0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 --2- 172.21.15.87:0/2885404166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17d0101090 0x7f17d01986e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 >> 172.21.15.87:0/2885404166 conn(0x7f17d0077f50 msgr2=0x7f17d00fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 shutdown_connections 2024-04-22T13:32:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.789+0000 7f17d8643640 1 -- 172.21.15.87:0/2885404166 wait complete. 2024-04-22T13:32:06.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 -- 172.21.15.87:0/2703815062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c101a40 msgr2=0x7feb1c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 --2- 172.21.15.87:0/2703815062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c101a40 0x7feb1c105a80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7feb04009900 tx=0x7feb0402bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 -- 172.21.15.87:0/2703815062 shutdown_connections 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 --2- 172.21.15.87:0/2703815062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c106160 0x7feb1c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 --2- 172.21.15.87:0/2703815062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c101a40 0x7feb1c105a80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 --2- 172.21.15.87:0/2703815062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 0x7feb1c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 -- 172.21.15.87:0/2703815062 >> 172.21.15.87:0/2703815062 conn(0x7feb1c077f50 msgr2=0x7feb1c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 -- 172.21.15.87:0/2703815062 shutdown_connections 2024-04-22T13:32:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.904+0000 7feb20cc8640 1 -- 172.21.15.87:0/2703815062 wait complete. 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 Processor -- start 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 -- start start 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 0x7feb1c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c1985b0 0x7feb1c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c1110a0 con 0x7feb1c106160 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c110f20 con 0x7feb1c1985b0 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb20cc8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c111220 con 0x7feb1c101090 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38226/0 (socket says 172.21.15.87:38226) 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 -- 172.21.15.87:0/3890118914 learned_addr learned my addr 172.21.15.87:0/3890118914 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb1a59c640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 0x7feb1c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c1985b0 0x7feb1c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 -- 172.21.15.87:0/3890118914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 msgr2=0x7feb1c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 0x7feb1c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 -- 172.21.15.87:0/3890118914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c1985b0 msgr2=0x7feb1c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c1985b0 0x7feb1c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 -- 172.21.15.87:0/3890118914 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c1990d0 con 0x7feb1c106160 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.905+0000 7feb19d9b640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb04036040 tx=0x7feb0402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.906+0000 7feb137fe640 1 -- 172.21.15.87:0/3890118914 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04043020 con 0x7feb1c106160 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.906+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1c199360 con 0x7feb1c106160 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.906+0000 7feb137fe640 1 -- 172.21.15.87:0/3890118914 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb0402cbe0 con 0x7feb1c106160 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.906+0000 7feb137fe640 1 -- 172.21.15.87:0/3890118914 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb04003bf0 con 0x7feb1c106160 2024-04-22T13:32:06.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.906+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1c1a55b0 con 0x7feb1c106160 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb137fe640 1 -- 172.21.15.87:0/3890118914 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7feb04046020 con 0x7feb1c106160 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 Processor -- start 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 -- start start 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x557696790a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 0x55769679a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5576966f4b10 con 0x557696791040 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557696692240 con 0x557696790690 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.908+0000 7feb2399a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5576966f0ad0 con 0x55769679a140 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38236/0 (socket says 172.21.15.87:38236) 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 -- 172.21.15.87:0/598279510 learned_addr learned my addr 172.21.15.87:0/598279510 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb137fe640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 0x7feb080a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x557696790a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb19d9b640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 0x55769679a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 0x7feb080a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 -- 172.21.15.87:0/598279510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 msgr2=0x55769679a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 0x55769679a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 -- 172.21.15.87:0/598279510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 msgr2=0x557696790a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb137fe640 1 -- 172.21.15.87:0/3890118914 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7feb040f2b50 con 0x7feb1c106160 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x557696790a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 -- 172.21.15.87:0/598279510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576964072d0 con 0x557696791040 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x557696790a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb19d9b640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 0x55769679a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1a59c640 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7feb140103f0 tx=0x7feb140108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03bbd95234b61be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 0x7feb080a6390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005e00 tx=0x7feb0c005cc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb10ff9640 1 -- 172.21.15.87:0/598279510 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14016070 con 0x557696791040 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb10ff9640 1 -- 172.21.15.87:0/598279510 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb14004d20 con 0x557696791040 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.909+0000 7feb10ff9640 1 -- 172.21.15.87:0/598279510 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb140053e0 con 0x557696791040 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- 172.21.15.87:0/598279510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 msgr2=0x557696799b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7feb140103f0 tx=0x7feb140108b0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- 172.21.15.87:0/598279510 shutdown_connections 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55769679a140 0x55769679a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557696791040 0x557696799b70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 --2- 172.21.15.87:0/598279510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x557696790a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- 172.21.15.87:0/598279510 >> 172.21.15.87:0/598279510 conn(0x55769678aa60 msgr2=0x55769678ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- 172.21.15.87:0/598279510 shutdown_connections 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- 172.21.15.87:0/598279510 wait complete. 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 -- start start 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.910+0000 7feb2399a740 1 Processor -- start 2024-04-22T13:32:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x55769679db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x55769679e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5576967a1080 con 0x55769679a140 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55769667d190 con 0x557696791040 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5576967a1200 con 0x557696790690 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 10 client.0 _ll_register_callbacks cb 0x5576966788b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38250/0 (socket says 172.21.15.87:38250) 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 -- 172.21.15.87:0/248654349 learned_addr learned my addr 172.21.15.87:0/248654349 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb19d9b640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x55769679e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1a59c640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x55769679db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 -- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 msgr2=0x55769679db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x55769679db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 -- 172.21.15.87:0/248654349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 msgr2=0x55769679e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x55769679e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 -- 172.21.15.87:0/248654349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576964d16b0 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7feb0c01c7b0 tx=0x7feb0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca570234034a9f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c023070 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7feb0c00fb50 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.911+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb14004d20 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb0c00fd90 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 10 client.15699 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5576967ad1e0 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb14004d20 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7feb0c004bb0 con 0x55769679a140 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 10 client.15699 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 10 client.15699 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 20 client.15699 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5576967af8a0 0x5576967b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 4 client.15699 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557696702530 con 0x5576967af8a0 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.912+0000 7feb1a59c640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5576967af8a0 0x5576967b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.913+0000 7feb01ffb640 10 client.15699 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.913+0000 7feb1a59c640 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5576967af8a0 0x5576967b0d10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=abc3c560549a2cc6 server_cookie=79d50cff3d3e4592 in_seq=0 out_seq=0 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.913+0000 7feb01ffb640 1 -- 172.21.15.87:0/248654349 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557696702530 con 0x5576967af8a0 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.913+0000 7feb01ffb640 10 client.15699 handle_command_reply: tid=0 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 client.15699 shutdown 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 msgr2=0x55769679e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7feb0c01c7b0 tx=0x7feb0c0046c0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 shutdown_connections 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5576967af8a0 0x5576967b0d10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679e580 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x55769679e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 --2- 172.21.15.87:0/248654349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x55769679db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb01ffb640 0 client.15699 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 >> 172.21.15.87:0/248654349 conn(0x55769678aa60 msgr2=0x55769678cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 shutdown_connections 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 1 -- 172.21.15.87:0/248654349 wait complete. 2024-04-22T13:32:06.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.914+0000 7feb2399a740 20 client.15699 trim_cache size 0 max 16384 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.931+0000 7feb2399a740 1 Processor -- start 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.931+0000 7feb2399a740 1 -- start start 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x5576967a0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x557696799890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5576967af050 con 0x55769679a140 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576967af720 con 0x557696791040 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb2399a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5576967afdf0 con 0x557696790690 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38252/0 (socket says 172.21.15.87:38252) 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1579980862 learned_addr learned my addr 172.21.15.87:0/1579980862 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb19d9b640 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x557696799890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1579980862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 msgr2=0x5576967a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x5576967a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1579980862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 msgr2=0x557696799890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x557696799890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1579980862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576967902c0 con 0x55769679a140 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb19d9b640 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x557696799890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7feb0c014010 tx=0x7feb0c00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc1ba7b7c70f0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb03fff640 1 -- 172.21.15.87:0/1579980862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c034070 con 0x55769679a140 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb03fff640 1 -- 172.21.15.87:0/1579980862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb0c027880 con 0x55769679a140 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.932+0000 7feb03fff640 1 -- 172.21.15.87:0/1579980862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c023040 con 0x55769679a140 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- 172.21.15.87:0/1579980862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 msgr2=0x55769679b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7feb0c014010 tx=0x7feb0c00bbb0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- 172.21.15.87:0/1579980862 shutdown_connections 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679b530 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696791040 0x557696799890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- 172.21.15.87:0/1579980862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696790690 0x5576967a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- 172.21.15.87:0/1579980862 >> 172.21.15.87:0/1579980862 conn(0x55769678aa60 msgr2=0x557696785800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- 172.21.15.87:0/1579980862 shutdown_connections 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- 172.21.15.87:0/1579980862 wait complete. 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- start start 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 Processor -- start 2024-04-22T13:32:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x5576967b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696791040 0x55769679a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5576967b0b90 con 0x55769679a140 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576967b04c0 con 0x557696790690 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5576967b1260 con 0x557696791040 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.933+0000 7feb2399a740 10 client.0 _ll_register_callbacks cb 0x7feb0c00bdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38258/0 (socket says 172.21.15.87:38258) 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1535497667 learned_addr learned my addr 172.21.15.87:0/1535497667 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1a59c640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x5576967b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696791040 msgr2=0x55769679a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb19d9b640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696791040 0x55769679a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696791040 0x55769679a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1535497667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 msgr2=0x5576967b91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x5576967b91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 -- 172.21.15.87:0/1535497667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55769679b050 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb1ad9d640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00b910 tx=0x7feb0c005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac1866063b5ef25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c03d030 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7feb0c027c40 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c027880 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb0c04a050 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb2399a740 10 client.15705 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5576967b8ad0 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb0c027880 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.934+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7feb0c003860 con 0x55769679a140 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 10 client.15705 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 10 client.15705 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T13:32:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 20 client.15705 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557696819d30 0x557696818a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 4 client.15705 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "status"}) -- 0x5576966fc3a0 con 0x557696819d30 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb1a59c640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557696819d30 0x557696818a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb1a59c640 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557696819d30 0x557696818a40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee32a60c459fe9ef server_cookie=12d1fb90e46e3fdf in_seq=0 out_seq=0 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb01ffb640 10 client.15705 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb01ffb640 1 -- 172.21.15.87:0/1535497667 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5576966fc3a0 con 0x557696819d30 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb01ffb640 10 client.15705 handle_command_reply: tid=0 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.935+0000 7feb2399a740 1 client.15705 shutdown 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 msgr2=0x55769679aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00b910 tx=0x7feb0c005f40 comp rx=0 tx=0).stop 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 shutdown_connections 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557696819d30 0x557696818a40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55769679a140 0x55769679aa60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb01ffb640 0 client.15705 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557696791040 0x55769679a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 --2- 172.21.15.87:0/1535497667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557696790690 0x5576967b91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 >> 172.21.15.87:0/1535497667 conn(0x55769678aa60 msgr2=0x557696797ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 shutdown_connections 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 1 -- 172.21.15.87:0/1535497667 wait complete. 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb2399a740 20 client.15705 trim_cache size 0 max 16384 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:32:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 1, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24475, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 137.62950624600001, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 21, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 88, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 154.68030736700001 2024-04-22T13:32:06.940 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 msgr2=0x7feb080a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 0x7feb080a6390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005e00 tx=0x7feb0c005cc0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 msgr2=0x7feb1c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb04036040 tx=0x7feb0402c7e0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 shutdown_connections 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb080a3ed0 0x7feb080a6390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb1c1985b0 0x7feb1c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c106160 0x7feb1c19e940 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 --2- 172.21.15.87:0/3890118914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb1c101090 0x7feb1c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:06.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 >> 172.21.15.87:0/3890118914 conn(0x7feb1c077f50 msgr2=0x7feb1c069500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:06.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 shutdown_connections 2024-04-22T13:32:06.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:06.936+0000 7feb20cc8640 1 -- 172.21.15.87:0/3890118914 wait complete. 2024-04-22T13:32:06.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1844593802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 msgr2=0x7f7fdc10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1844593802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc10ee80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009ca0 tx=0x7f7fd402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1844593802 shutdown_connections 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1844593802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc10ee80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1844593802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc102b90 0x7f7fdc106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1844593802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1021e0 0x7f7fdc1025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1844593802 >> 172.21.15.87:0/1844593802 conn(0x7f7fdc0fda60 msgr2=0x7f7fdc0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1844593802 shutdown_connections 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1844593802 wait complete. 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 Processor -- start 2024-04-22T13:32:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.047+0000 7f7fe41e2640 1 -- start start 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc1021e0 0x7f7fdc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 0x7f7fdc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc112220 con 0x7f7fdc1072b0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc1120a0 con 0x7f7fdc1021e0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc1123a0 con 0x7f7fdc1985a0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38274/0 (socket says 172.21.15.87:38274) 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 -- 172.21.15.87:0/1219686278 learned_addr learned my addr 172.21.15.87:0/1219686278 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe277f640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 0x7f7fdc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc1021e0 0x7f7fdc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 -- 172.21.15.87:0/1219686278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 msgr2=0x7f7fdc198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 0x7f7fdc198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 -- 172.21.15.87:0/1219686278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc1021e0 msgr2=0x7f7fdc19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc1021e0 0x7f7fdc19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 -- 172.21.15.87:0/1219686278 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc1990c0 con 0x7f7fdc1072b0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe277f640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 0x7f7fdc198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe177d640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c970 tx=0x7f7fd800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fceffd640 1 -- 172.21.15.87:0/1219686278 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8007bb0 con 0x7f7fdc1072b0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fceffd640 1 -- 172.21.15.87:0/1219686278 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd8007d50 con 0x7f7fdc1072b0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fceffd640 1 -- 172.21.15.87:0/1219686278 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800f450 con 0x7f7fdc1072b0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc1993b0 con 0x7f7fdc1072b0 2024-04-22T13:32:07.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.048+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc1a5600 con 0x7f7fdc1072b0 2024-04-22T13:32:07.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fceffd640 1 -- 172.21.15.87:0/1219686278 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f7fd8004f20 con 0x7f7fdc1072b0 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 Processor -- start 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 -- start start 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcd3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b5fc2eb10 con 0x558b5fcca7b0 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b5fbcc240 con 0x558b5fcd3ce0 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.051+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b5fc2aad0 con 0x558b5fccb160 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fceffd640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 0x7f7fc80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38278/0 (socket says 172.21.15.87:38278) 2024-04-22T13:32:07.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 -- 172.21.15.87:0/2454675970 learned_addr learned my addr 172.21.15.87:0/2454675970 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcd3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe177d640 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 0x7f7fc80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fceffd640 1 -- 172.21.15.87:0/1219686278 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7fd80d0bd0 con 0x7f7fdc1072b0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 -- 172.21.15.87:0/2454675970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 msgr2=0x558b5fcd3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcd3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 -- 172.21.15.87:0/2454675970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 msgr2=0x558b5fcd6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 -- 172.21.15.87:0/2454675970 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b5f9412d0 con 0x558b5fcca7b0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4032540 tx=0x7f7fd4032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2ea5972438d344 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fc7fff640 1 -- 172.21.15.87:0/2454675970 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4004040 con 0x558b5fcca7b0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fe277f640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 0x7f7fc80a6390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fd404c000 tx=0x7f7fd4048000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fc7fff640 1 -- 172.21.15.87:0/2454675970 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd4004690 con 0x558b5fcca7b0 2024-04-22T13:32:07.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.052+0000 7f7fc7fff640 1 -- 172.21.15.87:0/2454675970 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4039070 con 0x558b5fcca7b0 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/2454675970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 msgr2=0x558b5fccab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4032540 tx=0x7f7fd4032570 comp rx=0 tx=0).stop 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/2454675970 shutdown_connections 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcd3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/2454675970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fccab90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/2454675970 >> 172.21.15.87:0/2454675970 conn(0x558b5fcc4a60 msgr2=0x558b5fcc4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/2454675970 shutdown_connections 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/2454675970 wait complete. 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 -- start start 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.053+0000 7f7fe6dcb740 1 Processor -- start 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcca7b0 0x558b5fcd0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b5fcd6c50 con 0x558b5fccb160 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b5fbb7190 con 0x558b5fcd3ce0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b5fcd6dd0 con 0x558b5fcca7b0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe6dcb740 10 client.0 _ll_register_callbacks cb 0x558b5fbb28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38292/0 (socket says 172.21.15.87:38292) 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 -- 172.21.15.87:0/3474158698 learned_addr learned my addr 172.21.15.87:0/3474158698 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe277f640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcca7b0 0x558b5fcd0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 -- 172.21.15.87:0/3474158698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcca7b0 msgr2=0x558b5fcd0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcca7b0 0x558b5fcd0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 -- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 msgr2=0x558b5fcd1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 -- 172.21.15.87:0/3474158698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b5fa0b6b0 con 0x558b5fccb160 2024-04-22T13:32:07.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8000c00 tx=0x7f7fd801ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5aaeac6f9b8052 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.054+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd809c8c0 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7fd801ecf0 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd4004690 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fd80a0c70 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 10 client.15717 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b5fce2fe0 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd4004690 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7fd809cc20 con 0x558b5fccb160 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 10 client.15717 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 10 client.15717 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 20 client.15717 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fce5580 0x558b5fce69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 4 client.15717 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:07.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b5fc3c530 con 0x558b5fce5580 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.055+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fce5580 0x558b5fce69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.056+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fce5580 0x558b5fce69f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=187700eb77df2152 server_cookie=81a9e2df6a360201 in_seq=0 out_seq=0 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.056+0000 7f7fc57fa640 10 client.15717 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.056+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3474158698 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b5fc3c530 con 0x558b5fce5580 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.056+0000 7f7fc57fa640 10 client.15717 handle_command_reply: tid=0 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 client.15717 shutdown 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 msgr2=0x558b5fcd1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8000c00 tx=0x7f7fd801ec80 comp rx=0 tx=0).stop 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 shutdown_connections 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fce5580 0x558b5fce69f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fcd1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fccb160 0x558b5fcd1220 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3474158698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcca7b0 0x558b5fcd0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fc57fa640 0 client.15717 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:07.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 >> 172.21.15.87:0/3474158698 conn(0x558b5fcc4a60 msgr2=0x558b5fcc6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 shutdown_connections 2024-04-22T13:32:07.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3474158698 wait complete. 2024-04-22T13:32:07.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.057+0000 7f7fe6dcb740 20 client.15717 trim_cache size 0 max 16384 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 Processor -- start 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 -- start start 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcbf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fccf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce4d30 con 0x558b5fcca7b0 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce5400 con 0x558b5fcd3ce0 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce5ad0 con 0x558b5fccb160 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38294/0 (socket says 172.21.15.87:38294) 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/53445092 learned_addr learned my addr 172.21.15.87:0/53445092 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/53445092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 msgr2=0x558b5fcbf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcbf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/53445092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 msgr2=0x558b5fccf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fccf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/53445092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b5fccc8c0 con 0x558b5fcca7b0 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000bad0 tx=0x7f7fd000b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea507f22ec88668 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fc77fe640 1 -- 172.21.15.87:0/53445092 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd001a380 con 0x558b5fcca7b0 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fc77fe640 1 -- 172.21.15.87:0/53445092 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd0020070 con 0x558b5fcca7b0 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.075+0000 7f7fc77fe640 1 -- 172.21.15.87:0/53445092 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00116a0 con 0x558b5fcca7b0 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/53445092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 msgr2=0x558b5fcd3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000bad0 tx=0x7f7fd000b340 comp rx=0 tx=0).stop 2024-04-22T13:32:07.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/53445092 shutdown_connections 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fcd3ce0 0x558b5fccf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fccb160 0x558b5fcbf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/53445092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fcd3410 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/53445092 >> 172.21.15.87:0/53445092 conn(0x558b5fcc4a60 msgr2=0x558b5fcc4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/53445092 shutdown_connections 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/53445092 wait complete. 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 -- start start 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.076+0000 7f7fe6dcb740 1 Processor -- start 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 0x558b5fce9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcd3ce0 0x558b5fced280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce6870 con 0x558b5fcca7b0 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce61a0 con 0x558b5fccb160 2024-04-22T13:32:07.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b5fce6f40 con 0x558b5fcd3ce0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 10 client.0 _ll_register_callbacks cb 0x7f7fd80058a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38308/0 (socket says 172.21.15.87:38308) 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/3630121073 learned_addr learned my addr 172.21.15.87:0/3630121073 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 0x558b5fce9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe277f640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcd3ce0 0x558b5fced280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/3630121073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcd3ce0 msgr2=0x558b5fced280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcd3ce0 0x558b5fced280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 msgr2=0x558b5fce9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 0x558b5fce9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 -- 172.21.15.87:0/3630121073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b5fced7c0 con 0x558b5fcca7b0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 0x558b5fce9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe1f7e640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0005830 tx=0x7f7fd000bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb413ac981f2e803 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:07.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0004c80 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7fd001a380 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.077+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd0020070 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fd001a580 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 10 client.15723 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b5fd4bc30 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd0020070 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7fd000f450 con 0x558b5fcca7b0 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 10 client.15723 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 10 client.15723 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 20 client.15723 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fd50550 0x558b5fd4f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 4 client.15723 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "status"}) -- 0x558b5fc363a0 con 0x558b5fd50550 2024-04-22T13:32:07.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.078+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fd50550 0x558b5fd4f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe177d640 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fd50550 0x558b5fd4f260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee20faa5b5d8a96 server_cookie=cf547a1d28eb888a in_seq=0 out_seq=0 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fc57fa640 10 client.15723 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fc57fa640 1 -- 172.21.15.87:0/3630121073 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558b5fc363a0 con 0x558b5fd50550 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fc57fa640 10 client.15723 handle_command_reply: tid=0 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 client.15723 shutdown 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 msgr2=0x558b5fce8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0005830 tx=0x7f7fd000bcd0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 shutdown_connections 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b5fd50550 0x558b5fd4f260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b5fcd3ce0 0x558b5fced280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fc57fa640 0 client.15723 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b5fccb160 0x558b5fce9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 --2- 172.21.15.87:0/3630121073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b5fcca7b0 0x558b5fce8ac0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 >> 172.21.15.87:0/3630121073 conn(0x558b5fcc4a60 msgr2=0x558b5fcbf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 shutdown_connections 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.079+0000 7f7fe6dcb740 1 -- 172.21.15.87:0/3630121073 wait complete. 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe6dcb740 20 client.15723 trim_cache size 0 max 16384 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_fsid": "176003e8-00ab-11ef-bc93-c7b262605968", 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "whoami": 0, 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "id": 24482, 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "want_state": "up:active", 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "state": "up:active", 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "fs_name": "cephfs", 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "rank_uptime": 139.75318311000001, 2024-04-22T13:32:07.082 INFO:teuthology.orchestra.run.smithi087.stdout: "mdsmap_epoch": 21, 2024-04-22T13:32:07.083 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch": 88, 2024-04-22T13:32:07.083 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T13:32:07.083 INFO:teuthology.orchestra.run.smithi087.stdout: "uptime": 146.202662852 2024-04-22T13:32:07.083 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 msgr2=0x7f7fc80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 0x7f7fc80a6390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fd404c000 tx=0x7f7fd4048000 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 msgr2=0x7f7fdc19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c970 tx=0x7f7fd800ce30 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 shutdown_connections 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7fc80a3ed0 0x7f7fc80a6390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fdc1985a0 0x7f7fdc198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7fdc1072b0 0x7f7fdc19e930 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 --2- 172.21.15.87:0/1219686278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fdc1021e0 0x7f7fdc19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 >> 172.21.15.87:0/1219686278 conn(0x7f7fdc0fda60 msgr2=0x7f7fdc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 shutdown_connections 2024-04-22T13:32:07.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:07.080+0000 7f7fe41e2640 1 -- 172.21.15.87:0/1219686278 wait complete. 2024-04-22T13:32:07.086 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-22T13:32:07.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:06 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1085434690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:06 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1963808284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:07.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1085434690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:06 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1963808284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:07.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:06 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1085434690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:32:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:06 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1963808284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:08.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:07 smithi132 ceph-mon[31531]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:08.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:07 smithi087 ceph-mon[27999]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:08.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:07 smithi120 ceph-mon[31101]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:10.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:09 smithi132 ceph-mon[31531]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:10.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:09 smithi087 ceph-mon[27999]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:10.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:09 smithi120 ceph-mon[31101]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-22T13:32:12.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:11 smithi087 ceph-mon[27999]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-22T13:32:12.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:11 smithi120 ceph-mon[31101]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-22T13:32:12.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:11 smithi132 ceph-mon[31531]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-04-22T13:32:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2627519072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 msgr2=0x7f8b9c1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2627519072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c1035c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009900 tx=0x7f8b8c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:32:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2627519072 shutdown_connections 2024-04-22T13:32:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2627519072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c10c520 0x7f8b9c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2627519072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c103b00 0x7f8b9c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2627519072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c1035c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2627519072 >> 172.21.15.87:0/2627519072 conn(0x7f8b9c077fd0 msgr2=0x7f8b9c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2627519072 shutdown_connections 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2627519072 wait complete. 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 Processor -- start 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.719+0000 7f8ba25e2640 1 -- start start 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 0x7f8b9c101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c10c520 0x7f8b9c1018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1121b0 con 0x7f8b9c1031e0 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c112030 con 0x7f8b9c10c520 2024-04-22T13:32:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba25e2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c112330 con 0x7f8b9c103b00 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58602/0 (socket says 172.21.15.87:58602) 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 -- 172.21.15.87:0/2167176996 learned_addr learned my addr 172.21.15.87:0/2167176996 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 0x7f8b9c101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c10c520 0x7f8b9c1018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 -- 172.21.15.87:0/2167176996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 msgr2=0x7f8b9c101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 0x7f8b9c101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 -- 172.21.15.87:0/2167176996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c10c520 msgr2=0x7f8b9c1018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c10c520 0x7f8b9c1018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9bfff640 1 -- 172.21.15.87:0/2167176996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c1a6bb0 con 0x7f8b9c1031e0 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.720+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 0x7f8b9c101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8b9bfff640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005940 tx=0x7f8b8c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8b997fa640 1 -- 172.21.15.87:0/2167176996 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c043020 con 0x7f8b9c1031e0 2024-04-22T13:32:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c1a6e40 con 0x7f8b9c1031e0 2024-04-22T13:32:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8b997fa640 1 -- 172.21.15.87:0/2167176996 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b8c02dc30 con 0x7f8b9c1031e0 2024-04-22T13:32:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8b997fa640 1 -- 172.21.15.87:0/2167176996 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c003a10 con 0x7f8b9c1031e0 2024-04-22T13:32:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.721+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c1a7350 con 0x7f8b9c1031e0 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8b997fa640 1 -- 172.21.15.87:0/2167176996 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f8b8c046020 con 0x7f8b9c1031e0 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8ba51cc740 1 Processor -- start 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8ba51cc740 1 -- start start 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8b997fa640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 0x7f8b840a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfd6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.723+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcf3b8d0 con 0x55cbdcfe00e0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbdced8e80 con 0x55cbdcfd78d0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcf37890 con 0x55cbdcfd6b60 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b9bfff640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 0x7f8b840a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58616/0 (socket says 172.21.15.87:58616) 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/2457374603 learned_addr learned my addr 172.21.15.87:0/2457374603 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b997fa640 1 -- 172.21.15.87:0/2167176996 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8b8c0f2b90 con 0x7f8b9c1031e0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfd6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/2457374603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 msgr2=0x55cbdcfd6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfd6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/2457374603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd78d0 msgr2=0x55cbdcfd6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/2457374603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbdcefa320 con 0x55cbdcfe00e0 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8b94012820 tx=0x7f8b94009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73479c52f7dbaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b9bfff640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 0x7f8b840a6430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0f0280 tx=0x7f8b8c0a9040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b827fc640 1 -- 172.21.15.87:0/2457374603 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94012e60 con 0x55cbdcfe00e0 2024-04-22T13:32:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b827fc640 1 -- 172.21.15.87:0/2457374603 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94010d80 con 0x55cbdcfe00e0 2024-04-22T13:32:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.724+0000 7f8b827fc640 1 -- 172.21.15.87:0/2457374603 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94004500 con 0x55cbdcfe00e0 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2457374603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 msgr2=0x55cbdcfe24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8b94012820 tx=0x7f8b94009ad0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2457374603 shutdown_connections 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe24d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfd6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2457374603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2457374603 >> 172.21.15.87:0/2457374603 conn(0x55cbdcf9dc90 msgr2=0x55cbdcfd1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2457374603 shutdown_connections 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2457374603 wait complete. 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.725+0000 7f8ba51cc740 1 -- start start 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 Processor -- start 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfdfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcfe2e10 con 0x55cbdcfd78d0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcec3eb0 con 0x55cbdcfe00e0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcfe2f90 con 0x55cbdcfd6b60 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba51cc740 10 client.0 _ll_register_callbacks cb 0x55cbdcef0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58632/0 (socket says 172.21.15.87:58632) 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2309043912 learned_addr learned my addr 172.21.15.87:0/2309043912 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfdfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2309043912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 msgr2=0x55cbdcfdfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfdfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 msgr2=0x55cbdcfdd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2309043912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbdcc4e210 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.726+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfdfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000bf50 tx=0x7f8b900128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71455f4e3dd474a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b900035f0 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8b90003790 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b94010d80 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8b90003990 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 10 client.15735 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cbdcd1a620 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94010d80 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8b90023020 con 0x55cbdcfd78d0 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 10 client.15735 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 20 client.15735 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdcff0210 0x55cbdcff2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 4 client.15735 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.727+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbdcf4f880 con 0x55cbdcff0210 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.728+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdcff0210 0x55cbdcff2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.728+0000 7f8b7b7fe640 10 client.15735 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.728+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdcff0210 0x55cbdcff2600 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a54918782eb1663 server_cookie=aba13d271ee95f83 in_seq=0 out_seq=0 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2309043912 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbdcf4f880 con 0x55cbdcff0210 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8b7b7fe640 10 client.15735 handle_command_reply: tid=0 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 client.15735 shutdown 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 msgr2=0x55cbdcfdce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000bf50 tx=0x7f8b900128e0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 shutdown_connections 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdcff0210 0x55cbdcff2600 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfdce50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2309043912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfdfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8b7b7fe640 0 client.15735 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 >> 172.21.15.87:0/2309043912 conn(0x55cbdcf9dc90 msgr2=0x55cbdcfd35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 shutdown_connections 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.729+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2309043912 wait complete. 2024-04-22T13:32:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.730+0000 7f8ba51cc740 20 client.15735 trim_cache size 0 max 16384 2024-04-22T13:32:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 Processor -- start 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 -- start start 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfebd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1b2400 con 0x55cbdcfe00e0 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1afcd0 con 0x55cbdcfd6b60 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.744+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1adb40 con 0x55cbdcfd78d0 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58634/0 (socket says 172.21.15.87:58634) 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/1096860435 learned_addr learned my addr 172.21.15.87:0/1096860435 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfebd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/1096860435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 msgr2=0x55cbdcfd85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/1096860435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd6b60 msgr2=0x55cbdcfebd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfebd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 -- 172.21.15.87:0/1096860435 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbdcfd1a60 con 0x55cbdcfe00e0 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8b940107a0 tx=0x7f8b94009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdd424a6ee4bf27 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8b81ffb640 1 -- 172.21.15.87:0/1096860435 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94033070 con 0x55cbdcfe00e0 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8b81ffb640 1 -- 172.21.15.87:0/1096860435 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94009480 con 0x55cbdcfe00e0 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.745+0000 7f8b81ffb640 1 -- 172.21.15.87:0/1096860435 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94005530 con 0x55cbdcfe00e0 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- 172.21.15.87:0/1096860435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 msgr2=0x55cbdcfdf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8b940107a0 tx=0x7f8b94009ad0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- 172.21.15.87:0/1096860435 shutdown_connections 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfdf580 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfd85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/1096860435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfebd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- 172.21.15.87:0/1096860435 >> 172.21.15.87:0/1096860435 conn(0x55cbdcf9dc90 msgr2=0x55cbdcfd7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- 172.21.15.87:0/1096860435 shutdown_connections 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- 172.21.15.87:0/1096860435 wait complete. 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 -- start start 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.746+0000 7f8ba51cc740 1 Processor -- start 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfe04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1ae190 con 0x55cbdcfd78d0 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1af600 con 0x55cbdcfe00e0 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c1ae860 con 0x55cbdcfd6b60 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 10 client.0 _ll_register_callbacks cb 0x7f8b9000b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58646/0 (socket says 172.21.15.87:58646) 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2366359586 learned_addr learned my addr 172.21.15.87:0/2366359586 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfe04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2366359586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 msgr2=0x55cbdcfe04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfe04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 msgr2=0x55cbdcfe0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 -- 172.21.15.87:0/2366359586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbdcfdd7d0 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfe04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b9b7fe640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8b90025e70 tx=0x7f8b90010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8210aef9225e4fef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90003b30 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8b90003cd0 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b94009480 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8b90005040 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 10 client.15741 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.747+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cbdcfe8400 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94009480 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8b90037050 con 0x55cbdcfd78d0 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 10 client.15741 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 20 client.15741 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdd056780 0x55cbdd058b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 4 client.15741 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbdcedb3d0 con 0x55cbdd056780 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.748+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdd056780 0x55cbdd058b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.749+0000 7f8ba0b7f640 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdd056780 0x55cbdd058b70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b164dbd9e3ccb7d server_cookie=2c31f39bd619efba in_seq=0 out_seq=0 2024-04-22T13:32:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.749+0000 7f8b7b7fe640 10 client.15741 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8b7b7fe640 1 -- 172.21.15.87:0/2366359586 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbdcedb3d0 con 0x55cbdd056780 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8b7b7fe640 10 client.15741 handle_command_reply: tid=0 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 client.15741 shutdown 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 msgr2=0x55cbdcfe0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8b90025e70 tx=0x7f8b90010490 comp rx=0 tx=0).stop 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 shutdown_connections 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cbdd056780 0x55cbdd058b70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbdcfe00e0 0x55cbdcfe0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cbdcfd78d0 0x55cbdcfe0a00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8b7b7fe640 0 client.15741 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 --2- 172.21.15.87:0/2366359586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbdcfd6b60 0x55cbdcfe04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 >> 172.21.15.87:0/2366359586 conn(0x55cbdcf9dc90 msgr2=0x55cbdcfdb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 shutdown_connections 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 1 -- 172.21.15.87:0/2366359586 wait complete. 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.760+0000 7f8ba51cc740 20 client.15741 trim_cache size 0 max 16384 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:32:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:32:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 msgr2=0x7f8b840a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 0x7f8b840a6430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0f0280 tx=0x7f8b8c0a9040 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 msgr2=0x7f8b9c100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005940 tx=0x7f8b8c0045e0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 shutdown_connections 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b840a3f70 0x7f8b840a6430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b9c10c520 0x7f8b9c1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b9c103b00 0x7f8b9c101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 --2- 172.21.15.87:0/2167176996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b9c1031e0 0x7f8b9c100e30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 >> 172.21.15.87:0/2167176996 conn(0x7f8b9c077fd0 msgr2=0x7f8b9c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 shutdown_connections 2024-04-22T13:32:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:13.761+0000 7f8ba25e2640 1 -- 172.21.15.87:0/2167176996 wait complete. 2024-04-22T13:32:14.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:13 smithi087 ceph-mon[27999]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 99 KiB/s wr, 52 op/s 2024-04-22T13:32:14.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:13 smithi120 ceph-mon[31101]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 99 KiB/s wr, 52 op/s 2024-04-22T13:32:14.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:13 smithi132 ceph-mon[31531]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 99 KiB/s wr, 52 op/s 2024-04-22T13:32:16.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:15 smithi087 ceph-mon[27999]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 99 KiB/s wr, 55 op/s 2024-04-22T13:32:16.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:15 smithi120 ceph-mon[31101]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 99 KiB/s wr, 55 op/s 2024-04-22T13:32:16.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:15 smithi132 ceph-mon[31531]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 99 KiB/s wr, 55 op/s 2024-04-22T13:32:18.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:17 smithi087 ceph-mon[27999]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T13:32:18.265 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:17 smithi132 ceph-mon[31531]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T13:32:18.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:17 smithi120 ceph-mon[31101]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T13:32:20.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:19 smithi087 ceph-mon[27999]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-22T13:32:20.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:19 smithi120 ceph-mon[31101]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-22T13:32:20.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:19 smithi132 ceph-mon[31531]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-22T13:32:22.087 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.2 2024-04-22T13:32:22.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:22.114 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:21 smithi087 ceph-mon[27999]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 -- 172.21.15.87:0/2917999215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40101a60 msgr2=0x7f7e40105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 --2- 172.21.15.87:0/2917999215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40101a60 0x7f7e40105aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009900 tx=0x7f7e3402bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 -- 172.21.15.87:0/2917999215 shutdown_connections 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 --2- 172.21.15.87:0/2917999215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e40106180 0x7f7e4010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 --2- 172.21.15.87:0/2917999215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40101a60 0x7f7e40105aa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 --2- 172.21.15.87:0/2917999215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 0x7f7e40101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 -- 172.21.15.87:0/2917999215 >> 172.21.15.87:0/2917999215 conn(0x7f7e400771d0 msgr2=0x7f7e400775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 -- 172.21.15.87:0/2917999215 shutdown_connections 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 -- 172.21.15.87:0/2917999215 wait complete. 2024-04-22T13:32:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.205+0000 7f7e45d96640 1 Processor -- start 2024-04-22T13:32:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 -- start start 2024-04-22T13:32:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 0x7f7e40198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e4019cfc0 0x7f7e4019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401110c0 con 0x7f7e40106180 2024-04-22T13:32:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40110f40 con 0x7f7e4019cfc0 2024-04-22T13:32:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e45d96640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40111240 con 0x7f7e401010b0 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46564/0 (socket says 172.21.15.87:46564) 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 -- 172.21.15.87:0/3609596176 learned_addr learned my addr 172.21.15.87:0/3609596176 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3ffff640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e4019cfc0 0x7f7e4019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3f7fe640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 0x7f7e40198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 -- 172.21.15.87:0/3609596176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 msgr2=0x7f7e40198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 0x7f7e40198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 -- 172.21.15.87:0/3609596176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e4019cfc0 msgr2=0x7f7e4019d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e4019cfc0 0x7f7e4019d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 -- 172.21.15.87:0/3609596176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4019dab0 con 0x7f7e40106180 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.206+0000 7f7e3effd640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7e34036040 tx=0x7f7e3402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.207+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34043020 con 0x7f7e40106180 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.207+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e3402cbe0 con 0x7f7e40106180 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.207+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34003bf0 con 0x7f7e40106180 2024-04-22T13:32:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.207+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4019dd40 con 0x7f7e40106180 2024-04-22T13:32:22.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.207+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e401a5510 con 0x7f7e40106180 2024-04-22T13:32:22.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.208+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e40069230 con 0x7f7e40106180 2024-04-22T13:32:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.209+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f7e34046020 con 0x7f7e40106180 2024-04-22T13:32:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.209+0000 7f7e3cff9640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 0x7f7e280a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.210+0000 7f7e3f7fe640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 0x7f7e280a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.210+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7e340f32a0 con 0x7f7e40106180 2024-04-22T13:32:22.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.210+0000 7f7e3f7fe640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 0x7f7e280a6390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e30005fd0 tx=0x7f7e30005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.213+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e3404d020 con 0x7f7e40106180 2024-04-22T13:32:22.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:21 smithi120 ceph-mon[31101]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-22T13:32:22.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.433+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e40101a60 con 0x7f7e40106180 2024-04-22T13:32:22.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.434+0000 7f7e3cff9640 1 -- 172.21.15.87:0/3609596176 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f7e340f2090 con 0x7f7e40106180 2024-04-22T13:32:22.430 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:22.430 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:22.430 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 msgr2=0x7f7e280a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 0x7f7e280a6390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e30005fd0 tx=0x7f7e30005850 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 msgr2=0x7f7e40198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7e34036040 tx=0x7f7e3402c7e0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 shutdown_connections 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7e280a3ed0 0x7f7e280a6390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e4019cfc0 0x7f7e4019d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e40106180 0x7f7e40198c40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 --2- 172.21.15.87:0/3609596176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e401010b0 0x7f7e40198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 >> 172.21.15.87:0/3609596176 conn(0x7f7e400771d0 msgr2=0x7f7e4006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 shutdown_connections 2024-04-22T13:32:22.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.436+0000 7f7e45d96640 1 -- 172.21.15.87:0/3609596176 wait complete. 2024-04-22T13:32:22.443 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 74.2 --await' 2024-04-22T13:32:22.443 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 74.2 --await 2024-04-22T13:32:22.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:21 smithi132 ceph-mon[31531]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-04-22T13:32:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 -- 172.21.15.87:0/1644088126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 msgr2=0x7f276c1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 --2- 172.21.15.87:0/1644088126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c1035c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2768009900 tx=0x7f276802bd60 comp rx=0 tx=0).stop 2024-04-22T13:32:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 -- 172.21.15.87:0/1644088126 shutdown_connections 2024-04-22T13:32:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 --2- 172.21.15.87:0/1644088126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 --2- 172.21.15.87:0/1644088126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 0x7f276c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 --2- 172.21.15.87:0/1644088126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c1035c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.594+0000 7f2773996640 1 -- 172.21.15.87:0/1644088126 >> 172.21.15.87:0/1644088126 conn(0x7f276c077f50 msgr2=0x7f276c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- 172.21.15.87:0/1644088126 shutdown_connections 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- 172.21.15.87:0/1644088126 wait complete. 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 Processor -- start 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- start start 2024-04-22T13:32:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 0x7f276c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1121e0 con 0x7f276c1031e0 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f276c112060 con 0x7f276c103b00 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2773996640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f276c112360 con 0x7f276c10c520 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2771732640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2771732640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46584/0 (socket says 172.21.15.87:46584) 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.595+0000 7f2771732640 1 -- 172.21.15.87:0/1400261870 learned_addr learned my addr 172.21.15.87:0/1400261870 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771f33640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2770f31640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 0x7f276c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 -- 172.21.15.87:0/1400261870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 msgr2=0x7f276c19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 -- 172.21.15.87:0/1400261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 msgr2=0x7f276c198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 0x7f276c198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 -- 172.21.15.87:0/1400261870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c19d720 con 0x7f276c1031e0 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771f33640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2771732640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2768000c00 tx=0x7f276802d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f27627fc640 1 -- 172.21.15.87:0/1400261870 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768043020 con 0x7f276c1031e0 2024-04-22T13:32:22.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f27627fc640 1 -- 172.21.15.87:0/1400261870 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f276802dde0 con 0x7f276c1031e0 2024-04-22T13:32:22.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276c19d9b0 con 0x7f276c1031e0 2024-04-22T13:32:22.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f27627fc640 1 -- 172.21.15.87:0/1400261870 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768003b80 con 0x7f276c1031e0 2024-04-22T13:32:22.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.596+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276c1a52a0 con 0x7f276c1031e0 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f27627fc640 1 -- 172.21.15.87:0/1400261870 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f2768046020 con 0x7f276c1031e0 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 Processor -- start 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 -- start start 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e13da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 0x558ba8e1f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d740b0 con 0x558ba8e142e0 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d25680 con 0x558ba8e139c0 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2776583740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d31c60 con 0x558ba8e1cec0 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2771732640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2771732640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46594/0 (socket says 172.21.15.87:46594) 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.599+0000 7f2771732640 1 -- 172.21.15.87:0/300351864 learned_addr learned my addr 172.21.15.87:0/300351864 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f27627fc640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 0x7f27580a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2770f31640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 0x558ba8e1f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771f33640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e13da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771f33640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 0x7f27580a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 -- 172.21.15.87:0/300351864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 msgr2=0x558ba8e1f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 0x558ba8e1f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 -- 172.21.15.87:0/300351864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 msgr2=0x558ba8e13da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e13da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 -- 172.21.15.87:0/300351864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba8b56c90 con 0x558ba8e142e0 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2770f31640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 0x558ba8e1f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771f33640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e13da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f27627fc640 1 -- 172.21.15.87:0/1400261870 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f27680f2b60 con 0x7f276c1031e0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771732640 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f27680f0330 tx=0x7f276807e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48db86a9655efad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f2771f33640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 0x7f27580a6390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2764005c40 tx=0x7f2764009280 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f27577fe640 1 -- 172.21.15.87:0/300351864 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27680be930 con 0x558ba8e142e0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f27577fe640 1 -- 172.21.15.87:0/300351864 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27680f6160 con 0x558ba8e142e0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.600+0000 7f27577fe640 1 -- 172.21.15.87:0/300351864 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27680be4c0 con 0x558ba8e142e0 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- 172.21.15.87:0/300351864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 msgr2=0x558ba8e1c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f27680f0330 tx=0x7f276807e720 comp rx=0 tx=0).stop 2024-04-22T13:32:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- 172.21.15.87:0/300351864 shutdown_connections 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e1cec0 0x558ba8e1f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e142e0 0x558ba8e1c980 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 --2- 172.21.15.87:0/300351864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e13da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- 172.21.15.87:0/300351864 >> 172.21.15.87:0/300351864 conn(0x558ba8e0e0f0 msgr2=0x558ba8e0e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- 172.21.15.87:0/300351864 shutdown_connections 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- 172.21.15.87:0/300351864 wait complete. 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 -- start start 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.601+0000 7f2776583740 1 Processor -- start 2024-04-22T13:32:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e19ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e1a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d77400 con 0x558ba8e1cec0 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d33840 con 0x558ba8e142e0 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558ba8d7c720 con 0x558ba8e139c0 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2776583740 10 client.0 _ll_register_callbacks cb 0x558ba8d84ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46598/0 (socket says 172.21.15.87:46598) 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 -- 172.21.15.87:0/2462583101 learned_addr learned my addr 172.21.15.87:0/2462583101 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2770f31640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e1a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771732640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e19ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 -- 172.21.15.87:0/2462583101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 msgr2=0x558ba8e19ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e19ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 -- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 msgr2=0x558ba8e1a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e1a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 -- 172.21.15.87:0/2462583101 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba8bb7420 con 0x558ba8e1cec0 2024-04-22T13:32:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771732640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e19ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2771f33640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f27640109d0 tx=0x7f2764004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca59aecc627d79c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.602+0000 7f2754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276400dc30 con 0x558ba8e1cec0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f276400ddd0 con 0x558ba8e1cec0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ba8d85b10 con 0x558ba8e1cec0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2764028470 con 0x558ba8e1cec0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 10 client.15759 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558ba8a4a580 con 0x558ba8e1cec0 2024-04-22T13:32:22.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558ba8d85b10 con 0x558ba8e1cec0 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2764030050 con 0x558ba8e1cec0 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 10 client.15759 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 10 client.15759 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 20 client.15759 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e2c850 0x558ba8e2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 4 client.15759 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ba8d7f980 con 0x558ba8e2c850 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.603+0000 7f2771732640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e2c850 0x558ba8e2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.604+0000 7f2754ff9640 10 client.15759 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.604+0000 7f2771732640 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e2c850 0x558ba8e2ecf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3612973328069b server_cookie=eca4dddc45337a77 in_seq=0 out_seq=0 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.604+0000 7f2754ff9640 1 -- 172.21.15.87:0/2462583101 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ba8d7f980 con 0x558ba8e2c850 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.604+0000 7f2754ff9640 10 client.15759 handle_command_reply: tid=0 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 client.15759 shutdown 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 msgr2=0x558ba8e1a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f27640109d0 tx=0x7f2764004a40 comp rx=0 tx=0).stop 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 shutdown_connections 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e2c850 0x558ba8e2ecf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a920 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e1a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 --2- 172.21.15.87:0/2462583101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e19ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2754ff9640 0 client.15759 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:22.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 >> 172.21.15.87:0/2462583101 conn(0x558ba8e0e0f0 msgr2=0x558ba8e0fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 shutdown_connections 2024-04-22T13:32:22.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 1 -- 172.21.15.87:0/2462583101 wait complete. 2024-04-22T13:32:22.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.605+0000 7f2776583740 20 client.15759 trim_cache size 0 max 16384 2024-04-22T13:32:22.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 Processor -- start 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 -- start start 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e142e0 0x558ba8e19650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1adc80 con 0x558ba8e1cec0 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1abad0 con 0x558ba8e139c0 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2776583740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ad5b0 con 0x558ba8e142e0 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46610/0 (socket says 172.21.15.87:46610) 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 -- 172.21.15.87:0/2770233553 learned_addr learned my addr 172.21.15.87:0/2770233553 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 -- 172.21.15.87:0/2770233553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e142e0 msgr2=0x558ba8e19650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e142e0 0x558ba8e19650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 -- 172.21.15.87:0/2770233553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 msgr2=0x558ba8e283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.622+0000 7f2771f33640 1 -- 172.21.15.87:0/2770233553 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba8e0df20 con 0x558ba8e1cec0 2024-04-22T13:32:22.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2771f33640 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2764010fd0 tx=0x7f276400d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09df335c1206680 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2756ffd640 1 -- 172.21.15.87:0/2770233553 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764010520 con 0x558ba8e1cec0 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2756ffd640 1 -- 172.21.15.87:0/2770233553 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2764036070 con 0x558ba8e1cec0 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2756ffd640 1 -- 172.21.15.87:0/2770233553 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764028610 con 0x558ba8e1cec0 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 -- 172.21.15.87:0/2770233553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 msgr2=0x558ba8e1a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2764010fd0 tx=0x7f276400d340 comp rx=0 tx=0).stop 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 -- 172.21.15.87:0/2770233553 shutdown_connections 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e1a6f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e142e0 0x558ba8e19650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 --2- 172.21.15.87:0/2770233553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e139c0 0x558ba8e283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 -- 172.21.15.87:0/2770233553 >> 172.21.15.87:0/2770233553 conn(0x558ba8e0e0f0 msgr2=0x558ba8e1be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 -- 172.21.15.87:0/2770233553 shutdown_connections 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.623+0000 7f2776583740 1 -- 172.21.15.87:0/2770233553 wait complete. 2024-04-22T13:32:22.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 -- start start 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 Processor -- start 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e24e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e201b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ac810 con 0x558ba8e1cec0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ac140 con 0x558ba8e142e0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1acee0 con 0x558ba8e139c0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2776583740 10 client.0 _ll_register_callbacks cb 0x7f2764009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46622/0 (socket says 172.21.15.87:46622) 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 -- 172.21.15.87:0/3943692065 learned_addr learned my addr 172.21.15.87:0/3943692065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771732640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e24e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2770f31640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 -- 172.21.15.87:0/3943692065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 msgr2=0x558ba8e24e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e24e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 -- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 msgr2=0x558ba8e201b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e201b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2771f33640 1 -- 172.21.15.87:0/3943692065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba8e91e30 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.624+0000 7f2770f31640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e201b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2771f33640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f27640097a0 tx=0x7f2764031c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d516d899c974d595 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276400dc30 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f276400ddd0 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2764036070 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2764048050 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 10 client.15765 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558ba8e92fd0 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2764036070 con 0x558ba8e1cec0 2024-04-22T13:32:22.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2764030040 con 0x558ba8e1cec0 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 10 client.15765 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 10 client.15765 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 20 client.15765 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e93620 0x558ba8e95a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 4 client.15765 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.2, "await": true}] 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.625+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.2, "await": true}) -- 0x558ba8cfbd30 con 0x558ba8e93620 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.626+0000 7f2771732640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e93620 0x558ba8e95a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.626+0000 7f2771732640 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e93620 0x558ba8e95a40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27625862b5aaee9 server_cookie=7bd54fd073828b09 in_seq=0 out_seq=0 2024-04-22T13:32:22.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.626+0000 7f2754ff9640 10 client.15765 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2754ff9640 1 -- 172.21.15.87:0/3943692065 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x558ba8cfbd30 con 0x558ba8e93620 2024-04-22T13:32:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2754ff9640 10 client.15765 handle_command_reply: tid=0 2024-04-22T13:32:22.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 client.15765 shutdown 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 msgr2=0x558ba8e208f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f27640097a0 tx=0x7f2764031c30 comp rx=0 tx=0).stop 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 shutdown_connections 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ba8e93620 0x558ba8e95a40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ba8e1cec0 0x558ba8e208f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ba8e142e0 0x558ba8e201b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2754ff9640 0 client.15765 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 --2- 172.21.15.87:0/3943692065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ba8e139c0 0x558ba8e24e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 >> 172.21.15.87:0/3943692065 conn(0x558ba8e0e0f0 msgr2=0x558ba8e0d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 shutdown_connections 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.638+0000 7f2776583740 1 -- 172.21.15.87:0/3943692065 wait complete. 2024-04-22T13:32:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2776583740 20 client.15765 trim_cache size 0 max 16384 2024-04-22T13:32:22.636 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:22.636 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:32:22.636 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:32:22.636 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 2, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "3ca5eb30": { 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 2, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 74.2, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:32:22.637 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:22.638 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 msgr2=0x7f27580a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 0x7f27580a6390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2764005c40 tx=0x7f2764009280 comp rx=0 tx=0).stop 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 msgr2=0x7f276c198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2768000c00 tx=0x7f276802d920 comp rx=0 tx=0).stop 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 shutdown_connections 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f27580a3ed0 0x7f27580a6390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f276c10c520 0x7f276c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f276c103b00 0x7f276c198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 --2- 172.21.15.87:0/1400261870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f276c1031e0 0x7f276c198780 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:22.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 >> 172.21.15.87:0/1400261870 conn(0x7f276c077f50 msgr2=0x7f276c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:22.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 shutdown_connections 2024-04-22T13:32:22.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:22.639+0000 7f2773996640 1 -- 172.21.15.87:0/1400261870 wait complete. 2024-04-22T13:32:22.646 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 3ca5eb30, quiesce took 0.6 seconds. Will release after: 14.2 2024-04-22T13:32:23.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3609596176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:23.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3609596176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:23.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3609596176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:24.337 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:23 smithi120 ceph-mon[31101]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-22T13:32:24.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:23 smithi132 ceph-mon[31531]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-22T13:32:24.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:23 smithi087 ceph-mon[27999]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-22T13:32:26.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:25 smithi120 ceph-mon[31101]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-04-22T13:32:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:25 smithi132 ceph-mon[31531]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-04-22T13:32:26.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:25 smithi087 ceph-mon[27999]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-04-22T13:32:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 -- 172.21.15.87:0/3116664084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 msgr2=0x7f8328107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 --2- 172.21.15.87:0/3116664084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328107350 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8318009930 tx=0x7f831802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 -- 172.21.15.87:0/3116664084 shutdown_connections 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 --2- 172.21.15.87:0/3116664084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101af0 0x7f832810dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 --2- 172.21.15.87:0/3116664084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101130 0x7f83281015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 --2- 172.21.15.87:0/3116664084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328107350 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 -- 172.21.15.87:0/3116664084 >> 172.21.15.87:0/3116664084 conn(0x7f8328077ba0 msgr2=0x7f8328077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 -- 172.21.15.87:0/3116664084 shutdown_connections 2024-04-22T13:32:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.856+0000 7f832ccb3640 1 -- 172.21.15.87:0/3116664084 wait complete. 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.857+0000 7f832ccb3640 1 Processor -- start 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.857+0000 7f832ccb3640 1 -- start start 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 0x7f8328102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 0x7f8328103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f83281110a0 con 0x7f8328106f70 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8328110f20 con 0x7f8328101af0 2024-04-22T13:32:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8328111220 con 0x7f8328101130 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 0x7f8328102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46646/0 (socket says 172.21.15.87:46646) 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 -- 172.21.15.87:0/2260977219 learned_addr learned my addr 172.21.15.87:0/2260977219 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f831ffff640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 0x7f8328103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 -- 172.21.15.87:0/2260977219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 msgr2=0x7f8328102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 0x7f8328102b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 -- 172.21.15.87:0/2260977219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 msgr2=0x7f8328103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 0x7f8328103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 -- 172.21.15.87:0/2260977219 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83281a6a40 con 0x7f8328106f70 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f831ffff640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 0x7f8328103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832659c640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 0x7f8328102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f8326d9d640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f832000eaa0 tx=0x7f832000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.856 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f831f7fe640 1 -- 172.21.15.87:0/2260977219 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832000cdb0 con 0x7f8328106f70 2024-04-22T13:32:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83281a6d30 con 0x7f8328106f70 2024-04-22T13:32:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f831f7fe640 1 -- 172.21.15.87:0/2260977219 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8320005020 con 0x7f8328106f70 2024-04-22T13:32:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.858+0000 7f831f7fe640 1 -- 172.21.15.87:0/2260977219 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200116d0 con 0x7f8328106f70 2024-04-22T13:32:26.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.859+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a7280 con 0x7f8328106f70 2024-04-22T13:32:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.861+0000 7f831f7fe640 1 -- 172.21.15.87:0/2260977219 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f83200054a0 con 0x7f8328106f70 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.861+0000 7f832f980740 1 Processor -- start 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.861+0000 7f832f980740 1 -- start start 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda0cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda184d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d3cd9718d0 con 0x55d3cda0d8d0 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d3cd90ee80 con 0x55d3cda160e0 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831f7fe640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 0x7f83140a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832f980740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d3cd96d890 con 0x55d3cda0cb60 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46650/0 (socket says 172.21.15.87:46650) 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 -- 172.21.15.87:0/2204088570 learned_addr learned my addr 172.21.15.87:0/2204088570 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f8326d9d640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 0x7f83140a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f832659c640 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda0cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda184d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831f7fe640 1 -- 172.21.15.87:0/2260977219 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f83200d0b40 con 0x7f8328106f70 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 -- 172.21.15.87:0/2204088570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 msgr2=0x55d3cda0cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda0cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 -- 172.21.15.87:0/2204088570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 msgr2=0x55d3cda184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda184d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 -- 172.21.15.87:0/2204088570 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3cd930320 con 0x55d3cda0d8d0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831ffff640 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8310012900 tx=0x7f8310008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9a71c998e2610e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f8326d9d640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 0x7f83140a6470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f83200ce120 tx=0x7f83200bd040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.862+0000 7f831cff9640 1 -- 172.21.15.87:0/2204088570 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310010b80 con 0x55d3cda0d8d0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f831cff9640 1 -- 172.21.15.87:0/2204088570 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310010d20 con 0x55d3cda0d8d0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f831cff9640 1 -- 172.21.15.87:0/2204088570 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310005240 con 0x55d3cda0d8d0 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 -- 172.21.15.87:0/2204088570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 msgr2=0x55d3cda0c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8310012900 tx=0x7f8310008f90 comp rx=0 tx=0).stop 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 -- 172.21.15.87:0/2204088570 shutdown_connections 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda184d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda0cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 --2- 172.21.15.87:0/2204088570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0d8d0 0x55d3cda0c620 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 -- 172.21.15.87:0/2204088570 >> 172.21.15.87:0/2204088570 conn(0x55d3cd9d3c90 msgr2=0x55d3cda07850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 -- 172.21.15.87:0/2204088570 shutdown_connections 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.863+0000 7f832f980740 1 -- 172.21.15.87:0/2204088570 wait complete. 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 -- start start 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 Processor -- start 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda15a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d3cda18e10 con 0x55d3cda160e0 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d3cd8f9eb0 con 0x55d3cda0d8d0 2024-04-22T13:32:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d3cda18f90 con 0x55d3cda0cb60 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832f980740 10 client.0 _ll_register_callbacks cb 0x55d3cd926b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46662/0 (socket says 172.21.15.87:46662) 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 -- 172.21.15.87:0/1079680777 learned_addr learned my addr 172.21.15.87:0/1079680777 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f832659c640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda15a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 -- 172.21.15.87:0/1079680777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 msgr2=0x55d3cda15a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda15a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 -- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0d8d0 msgr2=0x55d3cda12d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.864+0000 7f831ffff640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f831ffff640 1 -- 172.21.15.87:0/1079680777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3cd684210 con 0x55d3cda160e0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f831ffff640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f83100126a0 tx=0x7f8310008600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac664bf02e8a264 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310004a50 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8310004bf0 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8310010d20 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8310036050 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 10 client.15777 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d3cd750620 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310010d20 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f831001e810 con 0x55d3cda160e0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 10 client.15777 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 10 client.15777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 20 client.15777 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.865+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda25f50 0x55d3cda28340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.866+0000 7f832f980740 4 client.15777 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.866+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3cd985880 con 0x55d3cda25f50 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.866+0000 7f832659c640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda25f50 0x55d3cda28340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.866+0000 7f832659c640 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda25f50 0x55d3cda28340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1f0845a235abbd server_cookie=96d91be5b40b28c3 in_seq=0 out_seq=0 2024-04-22T13:32:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.866+0000 7f830dffb640 10 client.15777 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f830dffb640 1 -- 172.21.15.87:0/1079680777 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3cd985880 con 0x55d3cda25f50 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f830dffb640 10 client.15777 handle_command_reply: tid=0 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 client.15777 shutdown 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 msgr2=0x55d3cda132d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f83100126a0 tx=0x7f8310008600 comp rx=0 tx=0).stop 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 shutdown_connections 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda25f50 0x55d3cda28340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda132d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f830dffb640 0 client.15777 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 --2- 172.21.15.87:0/1079680777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0cb60 0x55d3cda15a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 >> 172.21.15.87:0/1079680777 conn(0x55d3cd9d3c90 msgr2=0x55d3cda095a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 shutdown_connections 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 1 -- 172.21.15.87:0/1079680777 wait complete. 2024-04-22T13:32:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.867+0000 7f832f980740 20 client.15777 trim_cache size 0 max 16384 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 Processor -- start 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 -- start start 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda15860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f83281b23f0 con 0x55d3cda0cb60 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83281afcc0 con 0x55d3cda160e0 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832f980740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83281adb30 con 0x55d3cda0d8d0 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46668/0 (socket says 172.21.15.87:46668) 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 -- 172.21.15.87:0/3267701239 learned_addr learned my addr 172.21.15.87:0/3267701239 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f832659c640 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda15860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 -- 172.21.15.87:0/3267701239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 msgr2=0x55d3cda12670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 -- 172.21.15.87:0/3267701239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 msgr2=0x55d3cda15860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda15860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.882+0000 7f831ffff640 1 -- 172.21.15.87:0/3267701239 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3cda0be30 con 0x55d3cda0cb60 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f831ffff640 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8310034040 tx=0x7f8310008920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3677575ad31618 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f830ffff640 1 -- 172.21.15.87:0/3267701239 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310008c10 con 0x55d3cda0cb60 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f830ffff640 1 -- 172.21.15.87:0/3267701239 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310010c90 con 0x55d3cda0cb60 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f830ffff640 1 -- 172.21.15.87:0/3267701239 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831001e970 con 0x55d3cda0cb60 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 -- 172.21.15.87:0/3267701239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 msgr2=0x55d3cda21d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8310034040 tx=0x7f8310008920 comp rx=0 tx=0).stop 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 -- 172.21.15.87:0/3267701239 shutdown_connections 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda160e0 0x55d3cda15860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda12670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 --2- 172.21.15.87:0/3267701239 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda0cb60 0x55d3cda21d10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 -- 172.21.15.87:0/3267701239 >> 172.21.15.87:0/3267701239 conn(0x55d3cd9d3c90 msgr2=0x55d3cda12d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 -- 172.21.15.87:0/3267701239 shutdown_connections 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.883+0000 7f832f980740 1 -- 172.21.15.87:0/3267701239 wait complete. 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 -- start start 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 Processor -- start 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0cb60 0x55d3cda191e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda19720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f83281ae180 con 0x55d3cda160e0 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83281af5f0 con 0x55d3cda0cb60 2024-04-22T13:32:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83281ae850 con 0x55d3cda0d8d0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832f980740 10 client.0 _ll_register_callbacks cb 0x7f8310008df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46678/0 (socket says 172.21.15.87:46678) 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 -- 172.21.15.87:0/2571878917 learned_addr learned my addr 172.21.15.87:0/2571878917 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0cb60 0x55d3cda191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f832659c640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda19720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 -- 172.21.15.87:0/2571878917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 msgr2=0x55d3cda19720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda19720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 -- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0cb60 msgr2=0x55d3cda191e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0cb60 0x55d3cda191e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.884+0000 7f831ffff640 1 -- 172.21.15.87:0/2571878917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3cda8a7d0 con 0x55d3cda160e0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f831ffff640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8310010ea0 tx=0x7f831001e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f1042120c964ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310004a50 con 0x55d3cda160e0 2024-04-22T13:32:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8310004bf0 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8310010c90 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8310004020 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 10 client.15783 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d3cda8b920 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310010c90 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8310050050 con 0x55d3cda160e0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 10 client.15783 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 20 client.15783 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda8bda0 0x55d3cda8e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 4 client.15783 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.885+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d3cd9113d0 con 0x55d3cda8bda0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.886+0000 7f832659c640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda8bda0 0x55d3cda8e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.886+0000 7f832659c640 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda8bda0 0x55d3cda8e190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6fea220529499db server_cookie=7b4385442eb859fc in_seq=0 out_seq=0 2024-04-22T13:32:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.886+0000 7f830dffb640 10 client.15783 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f830dffb640 1 -- 172.21.15.87:0/2571878917 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d3cd9113d0 con 0x55d3cda8bda0 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f830dffb640 10 client.15783 handle_command_reply: tid=0 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 client.15783 shutdown 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 msgr2=0x55d3cda8a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8310010ea0 tx=0x7f831001e550 comp rx=0 tx=0).stop 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 shutdown_connections 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d3cda8bda0 0x55d3cda8e190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d3cda160e0 0x55d3cda8a290 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d3cda0d8d0 0x55d3cda19720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 --2- 172.21.15.87:0/2571878917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d3cda0cb60 0x55d3cda191e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f830dffb640 0 client.15783 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 >> 172.21.15.87:0/2571878917 conn(0x55d3cd9d3c90 msgr2=0x55d3cda06040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 shutdown_connections 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 1 -- 172.21.15.87:0/2571878917 wait complete. 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.893+0000 7f832f980740 20 client.15783 trim_cache size 0 max 16384 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:32:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:32:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 msgr2=0x7f83140a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 0x7f83140a6470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f83200ce120 tx=0x7f83200bd040 comp rx=0 tx=0).stop 2024-04-22T13:32:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 msgr2=0x7f8328103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f832000eaa0 tx=0x7f832000ef60 comp rx=0 tx=0).stop 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 shutdown_connections 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f83140a3fb0 0x7f83140a6470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8328106f70 0x7f8328103580 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8328101af0 0x7f8328103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 --2- 172.21.15.87:0/2260977219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8328101130 0x7f8328102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 >> 172.21.15.87:0/2260977219 conn(0x7f8328077ba0 msgr2=0x7f8328100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 shutdown_connections 2024-04-22T13:32:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:26.894+0000 7f832ccb3640 1 -- 172.21.15.87:0/2260977219 wait complete. 2024-04-22T13:32:28.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:28 smithi120 ceph-mon[31101]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-22T13:32:28.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:28 smithi132 ceph-mon[31531]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-22T13:32:28.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:28 smithi087 ceph-mon[27999]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-22T13:32:29.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:32:29.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:32:29.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- 172.21.15.87:0/2265355010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101a30 msgr2=0x7f45dc105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 --2- 172.21.15.87:0/2265355010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101a30 0x7f45dc105a70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- 172.21.15.87:0/2265355010 shutdown_connections 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 --2- 172.21.15.87:0/2265355010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc106150 0x7f45dc10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 --2- 172.21.15.87:0/2265355010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101a30 0x7f45dc105a70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 --2- 172.21.15.87:0/2265355010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc101080 0x7f45dc101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- 172.21.15.87:0/2265355010 >> 172.21.15.87:0/2265355010 conn(0x7f45dc078000 msgr2=0x7f45dc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- 172.21.15.87:0/2265355010 shutdown_connections 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- 172.21.15.87:0/2265355010 wait complete. 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 Processor -- start 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.442+0000 7f45e444e640 1 -- start start 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc106150 0x7f45dc199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc19a080 0x7f45dc1a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111090 con 0x7f45dc101080 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc110f10 con 0x7f45dc19a080 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111210 con 0x7f45dc106150 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46694/0 (socket says 172.21.15.87:46694) 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 -- 172.21.15.87:0/1516063498 learned_addr learned my addr 172.21.15.87:0/1516063498 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e19e9640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc106150 0x7f45dc199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e29eb640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc19a080 0x7f45dc1a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 -- 172.21.15.87:0/1516063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc106150 msgr2=0x7f45dc199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc106150 0x7f45dc199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 -- 172.21.15.87:0/1516063498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc19a080 msgr2=0x7f45dc1a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc19a080 0x7f45dc1a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 -- 172.21.15.87:0/1516063498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc1a4e00 con 0x7f45dc101080 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e21ea640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f45d800d770 tx=0x7f45d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45d37fe640 1 -- 172.21.15.87:0/1516063498 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8017070 con 0x7f45dc101080 2024-04-22T13:32:29.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc1a50f0 con 0x7f45dc101080 2024-04-22T13:32:29.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.443+0000 7f45d37fe640 1 -- 172.21.15.87:0/1516063498 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45d80054d0 con 0x7f45dc101080 2024-04-22T13:32:29.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.444+0000 7f45d37fe640 1 -- 172.21.15.87:0/1516063498 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8004c30 con 0x7f45dc101080 2024-04-22T13:32:29.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.444+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a55c0 con 0x7f45dc101080 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.446+0000 7f45d37fe640 1 -- 172.21.15.87:0/1516063498 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f45d8004e10 con 0x7f45dc101080 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.446+0000 7f45e7044740 1 Processor -- start 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.446+0000 7f45d37fe640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 0x7f45c80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e29eb640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 0x7f45c80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 -- start start 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45d37fe640 1 -- 172.21.15.87:0/1516063498 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45d80d0540 con 0x7f45dc101080 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467aee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467af9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559467a531e0 con 0x559467aeeea0 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5594679f0390 con 0x559467af7a80 2024-04-22T13:32:29.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e7044740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559467a4f1a0 con 0x559467aee580 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46708/0 (socket says 172.21.15.87:46708) 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 -- 172.21.15.87:0/1851114613 learned_addr learned my addr 172.21.15.87:0/1851114613 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e29eb640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 0x7f45c80a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009370 tx=0x7f45d40092c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e29eb640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467af9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e21ea640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467aee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 -- 172.21.15.87:0/1851114613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 msgr2=0x559467aee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467aee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 -- 172.21.15.87:0/1851114613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 msgr2=0x559467af9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467af9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 -- 172.21.15.87:0/1851114613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55946782f7d0 con 0x559467aeeea0 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e29eb640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467af9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e21ea640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467aee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.447+0000 7f45e19e9640 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b34ba4c58935d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45d0bfe640 1 -- 172.21.15.87:0/1851114613 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc045020 con 0x559467aeeea0 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45d0bfe640 1 -- 172.21.15.87:0/1851114613 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc02c7e0 con 0x559467aeeea0 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45d0bfe640 1 -- 172.21.15.87:0/1851114613 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0046c0 con 0x559467aeeea0 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 -- 172.21.15.87:0/1851114613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 msgr2=0x559467af7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02c610 comp rx=0 tx=0).stop 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 -- 172.21.15.87:0/1851114613 shutdown_connections 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467af9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467af7540 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 --2- 172.21.15.87:0/1851114613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467aee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.448+0000 7f45e7044740 1 -- 172.21.15.87:0/1851114613 >> 172.21.15.87:0/1851114613 conn(0x559467ae8cb0 msgr2=0x559467ae9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- 172.21.15.87:0/1851114613 shutdown_connections 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- 172.21.15.87:0/1851114613 wait complete. 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- start start 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 Processor -- start 2024-04-22T13:32:29.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467af4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467af5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559467afa810 con 0x559467af7a80 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559467a0c2f0 con 0x559467aee580 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559467afa990 con 0x559467aeeea0 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e7044740 10 client.0 _ll_register_callbacks cb 0x559467a19580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e29eb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e29eb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46716/0 (socket says 172.21.15.87:46716) 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.449+0000 7f45e29eb640 1 -- 172.21.15.87:0/935813833 learned_addr learned my addr 172.21.15.87:0/935813833 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e19e9640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467af5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e21ea640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467af4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 -- 172.21.15.87:0/935813833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 msgr2=0x559467af5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467af5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 -- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 msgr2=0x559467af4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467af4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 -- 172.21.15.87:0/935813833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594677653f0 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e21ea640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467af4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e19e9640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467af5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e29eb640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f45d4011810 tx=0x7f45d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c75d03de39f40f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4010b50 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45d4010cf0 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc02c7e0 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f45d401f430 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e7044740 10 client.15795 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5594677250f0 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.450+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc02c7e0 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f45d402f020 con 0x559467af7a80 2024-04-22T13:32:29.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 10 client.15795 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 20 client.15795 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b07b40 0x559467b09f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 4 client.15795 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559467a60c00 con 0x559467b07b40 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e21ea640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b07b40 0x559467b09f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45c5ffb640 10 client.15795 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.451+0000 7f45e21ea640 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b07b40 0x559467b09f80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be0707da070fa320 server_cookie=1cac3220676d7df5 in_seq=0 out_seq=0 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45c5ffb640 1 -- 172.21.15.87:0/935813833 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559467a60c00 con 0x559467b07b40 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45c5ffb640 10 client.15795 handle_command_reply: tid=0 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 client.15795 shutdown 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 msgr2=0x559467af5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f45d4011810 tx=0x7f45d4004a40 comp rx=0 tx=0).stop 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 shutdown_connections 2024-04-22T13:32:29.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b07b40 0x559467b09f80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467af5560 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45c5ffb640 0 client.15795 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467af5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 --2- 172.21.15.87:0/935813833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467af4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.452+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 >> 172.21.15.87:0/935813833 conn(0x559467ae8cb0 msgr2=0x559467aeaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.453+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 shutdown_connections 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.453+0000 7f45e7044740 1 -- 172.21.15.87:0/935813833 wait complete. 2024-04-22T13:32:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.453+0000 7f45e7044740 20 client.15795 trim_cache size 0 max 16384 2024-04-22T13:32:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 Processor -- start 2024-04-22T13:32:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 -- start start 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467af7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467aea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1b01e0 con 0x559467aeeea0 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1adab0 con 0x559467af7a80 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e7044740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ab9e0 con 0x559467aee580 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46724/0 (socket says 172.21.15.87:46724) 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 -- 172.21.15.87:0/617022248 learned_addr learned my addr 172.21.15.87:0/617022248 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e21ea640 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467af7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 -- 172.21.15.87:0/617022248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 msgr2=0x559467af7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467af7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 -- 172.21.15.87:0/617022248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 msgr2=0x559467aea7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e29eb640 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467aea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467aea7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.470+0000 7f45e19e9640 1 -- 172.21.15.87:0/617022248 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559467b06690 con 0x559467aeeea0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e19e9640 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79a1bda6ed0acc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45c7fff640 1 -- 172.21.15.87:0/617022248 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc04e070 con 0x559467aeeea0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45c7fff640 1 -- 172.21.15.87:0/617022248 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc02cc90 con 0x559467aeeea0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45c7fff640 1 -- 172.21.15.87:0/617022248 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc045030 con 0x559467aeeea0 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 -- 172.21.15.87:0/617022248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 msgr2=0x559467b036d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009900 tx=0x7f45cc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 -- 172.21.15.87:0/617022248 shutdown_connections 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467af7a80 0x559467aea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467aeeea0 0x559467b036d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 --2- 172.21.15.87:0/617022248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aee580 0x559467af7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 -- 172.21.15.87:0/617022248 >> 172.21.15.87:0/617022248 conn(0x559467ae8cb0 msgr2=0x559467aea0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 -- 172.21.15.87:0/617022248 shutdown_connections 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.471+0000 7f45e7044740 1 -- 172.21.15.87:0/617022248 wait complete. 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 -- start start 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 Processor -- start 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467affbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467b00120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ac030 con 0x559467af7a80 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ad3e0 con 0x559467aee580 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1ac680 con 0x559467aeeea0 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e7044740 10 client.0 _ll_register_callbacks cb 0x7f45d4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46728/0 (socket says 172.21.15.87:46728) 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 -- 172.21.15.87:0/2126724044 learned_addr learned my addr 172.21.15.87:0/2126724044 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e19e9640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467b00120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e21ea640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467affbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 -- 172.21.15.87:0/2126724044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 msgr2=0x559467b00120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467b00120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 -- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 msgr2=0x559467affbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467affbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 -- 172.21.15.87:0/2126724044 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559467afd120 con 0x559467af7a80 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e19e9640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467b00120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45e29eb640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f45d4031c50 tx=0x7f45d4031c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419f0fc1dd0bdc57 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.472+0000 7f45c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d403f060 con 0x559467af7a80 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45d4004cb0 con 0x559467af7a80 2024-04-22T13:32:29.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc02cc90 con 0x559467af7a80 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f45d4036070 con 0x559467af7a80 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 10 client.15801 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559467b6e940 con 0x559467af7a80 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc02cc90 con 0x559467af7a80 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f45d4042020 con 0x559467af7a80 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 10 client.15801 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 20 client.15801 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b6f060 0x559467b71480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 4 client.15801 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559467a5aa70 con 0x559467b6f060 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.473+0000 7f45e21ea640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b6f060 0x559467b71480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45c5ffb640 10 client.15801 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e21ea640 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b6f060 0x559467b71480 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c86c7bcf72c3504 server_cookie=7716eae726d61fd2 in_seq=0 out_seq=0 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45c5ffb640 1 -- 172.21.15.87:0/2126724044 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559467a5aa70 con 0x559467b6f060 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45c5ffb640 10 client.15801 handle_command_reply: tid=0 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 client.15801 shutdown 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 msgr2=0x559467afcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f45d4031c50 tx=0x7f45d4031c80 comp rx=0 tx=0).stop 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 shutdown_connections 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559467b6f060 0x559467b71480 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559467af7a80 0x559467afcbb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45c5ffb640 0 client.15801 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:29.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559467aeeea0 0x559467b00120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 --2- 172.21.15.87:0/2126724044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559467aee580 0x559467affbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 >> 172.21.15.87:0/2126724044 conn(0x559467ae8cb0 msgr2=0x559467ae7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 shutdown_connections 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.474+0000 7f45e7044740 1 -- 172.21.15.87:0/2126724044 wait complete. 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e7044740 20 client.15801 trim_cache size 0 max 16384 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:32:29.479 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 msgr2=0x7f45c80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 0x7f45c80a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009370 tx=0x7f45d40092c0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 msgr2=0x7f45dc1995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f45d800d770 tx=0x7f45d800dc30 comp rx=0 tx=0).stop 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 shutdown_connections 2024-04-22T13:32:29.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c80a3ed0 0x7f45c80a6390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45dc19a080 0x7f45dc1a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45dc106150 0x7f45dc199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 --2- 172.21.15.87:0/1516063498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45dc101080 0x7f45dc1995b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 >> 172.21.15.87:0/1516063498 conn(0x7f45dc078000 msgr2=0x7f45dc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 shutdown_connections 2024-04-22T13:32:29.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.475+0000 7f45e444e640 1 -- 172.21.15.87:0/1516063498 wait complete. 2024-04-22T13:32:29.483 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:32:29.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:29.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.585+0000 7fc00405c640 1 -- 172.21.15.87:0/273724276 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc102190 msgr2=0x7fbffc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.585+0000 7fc00405c640 1 --2- 172.21.15.87:0/273724276 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc102190 0x7fbffc102570 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009860 tx=0x7fbff802bb70 comp rx=0 tx=0).stop 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- 172.21.15.87:0/273724276 shutdown_connections 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- 172.21.15.87:0/273724276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc107260 0x7fbffc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- 172.21.15.87:0/273724276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc102b40 0x7fbffc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- 172.21.15.87:0/273724276 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc102190 0x7fbffc102570 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- 172.21.15.87:0/273724276 >> 172.21.15.87:0/273724276 conn(0x7fbffc0793e0 msgr2=0x7fbffc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- 172.21.15.87:0/273724276 shutdown_connections 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- 172.21.15.87:0/273724276 wait complete. 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 Processor -- start 2024-04-22T13:32:29.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- start start 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc102190 0x7fbffc19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 0x7fbffc198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc112170 con 0x7fbffc107260 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc111ff0 con 0x7fbffc102190 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc00405c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc1122f0 con 0x7fbffc198590 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc0015f7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.586+0000 7fc0015f7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35832/0 (socket says 172.21.15.87:35832) 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 -- 172.21.15.87:0/2059725230 learned_addr learned my addr 172.21.15.87:0/2059725230 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc001df8640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc102190 0x7fbffc19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0025f9640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 0x7fbffc198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 -- 172.21.15.87:0/2059725230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 msgr2=0x7fbffc198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 0x7fbffc198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 -- 172.21.15.87:0/2059725230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc102190 msgr2=0x7fbffc19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc102190 0x7fbffc19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 -- 172.21.15.87:0/2059725230 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc1990b0 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0025f9640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 0x7fbffc198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc0015f7640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00e9f0 tx=0x7fbfec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec00ccc0 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfec0054d0 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc1993a0 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec010430 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.587+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc1a55a0 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.588+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc069230 con 0x7fbffc107260 2024-04-22T13:32:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.589+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fbfec0046c0 con 0x7fbffc107260 2024-04-22T13:32:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.590+0000 7fbff2ffd640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 0x7fbfe80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.590+0000 7fc001df8640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 0x7fbfe80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.590+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fbfec0d0e70 con 0x7fbffc107260 2024-04-22T13:32:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.590+0000 7fc001df8640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 0x7fbfe80a6390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbff802c180 tx=0x7fbff80058d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.594+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfec09d340 con 0x7fbffc107260 2024-04-22T13:32:29.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.813+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbffc199ac0 con 0x7fbffc107260 2024-04-22T13:32:29.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.813+0000 7fbff2ffd640 1 -- 172.21.15.87:0/2059725230 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fbffc199ac0 con 0x7fbffc107260 2024-04-22T13:32:29.810 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:29.810 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:29.810 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 msgr2=0x7fbfe80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 0x7fbfe80a6390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbff802c180 tx=0x7fbff80058d0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 msgr2=0x7fbffc19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00e9f0 tx=0x7fbfec00eeb0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 shutdown_connections 2024-04-22T13:32:29.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbfe80a3ed0 0x7fbfe80a6390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbffc198590 0x7fbffc198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbffc107260 0x7fbffc19e920 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 --2- 172.21.15.87:0/2059725230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbffc102190 0x7fbffc19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.815+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 >> 172.21.15.87:0/2059725230 conn(0x7fbffc0793e0 msgr2=0x7fbffc1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.816+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 shutdown_connections 2024-04-22T13:32:29.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.816+0000 7fc00405c640 1 -- 172.21.15.87:0/2059725230 wait complete. 2024-04-22T13:32:29.823 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.927+0000 7f4028352640 1 -- 172.21.15.87:0/2618991569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4020101090 msgr2=0x7f4020101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.927+0000 7f4028352640 1 --2- 172.21.15.87:0/2618991569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4020101090 0x7f4020101470 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f401c009ca0 tx=0x7f401c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- 172.21.15.87:0/2618991569 shutdown_connections 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- 172.21.15.87:0/2618991569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020106160 0x7f402010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- 172.21.15.87:0/2618991569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020101a40 0x7f4020105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- 172.21.15.87:0/2618991569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4020101090 0x7f4020101470 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- 172.21.15.87:0/2618991569 >> 172.21.15.87:0/2618991569 conn(0x7f4020077f50 msgr2=0x7f4020078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- 172.21.15.87:0/2618991569 shutdown_connections 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- 172.21.15.87:0/2618991569 wait complete. 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 Processor -- start 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- start start 2024-04-22T13:32:29.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020101090 0x7f4020197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 0x7f4020197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f40201110a0 con 0x7f402019be40 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4020110f20 con 0x7f4020106160 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f4028352640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4020111220 con 0x7f4020101090 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f40268ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f40268ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35850/0 (socket says 172.21.15.87:35850) 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.928+0000 7f40268ef640 1 -- 172.21.15.87:0/591584545 learned_addr learned my addr 172.21.15.87:0/591584545 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40258ed640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 0x7f4020197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40260ee640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020101090 0x7f4020197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 -- 172.21.15.87:0/591584545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020101090 msgr2=0x7f4020197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020101090 0x7f4020197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 -- 172.21.15.87:0/591584545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 msgr2=0x7f4020197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 0x7f4020197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 -- 172.21.15.87:0/591584545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402019c930 con 0x7f402019be40 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40258ed640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 0x7f4020197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40268ef640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f401800d750 tx=0x7f401800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40177fe640 1 -- 172.21.15.87:0/591584545 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018017070 con 0x7f402019be40 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40177fe640 1 -- 172.21.15.87:0/591584545 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40180054d0 con 0x7f402019be40 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f40177fe640 1 -- 172.21.15.87:0/591584545 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018004c60 con 0x7f402019be40 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402019cc20 con 0x7f402019be40 2024-04-22T13:32:29.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.929+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40201a43c0 con 0x7f402019be40 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.931+0000 7f40177fe640 1 -- 172.21.15.87:0/591584545 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4018009bd0 con 0x7f402019be40 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 Processor -- start 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 -- start start 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7e340 0x557d38c7e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7ecf0 0x557d38c85620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557d38be31f0 con 0x557d38c85bf0 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557d38b804b0 con 0x557d38c7ecf0 2024-04-22T13:32:29.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f402b026740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557d38bdf1b0 con 0x557d38c7e340 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f40177fe640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 0x7f400c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35864/0 (socket says 172.21.15.87:35864) 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.932+0000 7f40258ed640 1 -- 172.21.15.87:0/3121500620 learned_addr learned my addr 172.21.15.87:0/3121500620 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40268ef640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 0x7f400c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7e340 0x557d38c7e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40260ee640 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7ecf0 0x557d38c85620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40177fe640 1 -- 172.21.15.87:0/591584545 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f40180d0480 con 0x7f402019be40 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 -- 172.21.15.87:0/3121500620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7e340 msgr2=0x557d38c7e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7e340 0x557d38c7e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 -- 172.21.15.87:0/3121500620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7ecf0 msgr2=0x557d38c85620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7ecf0 0x557d38c85620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 -- 172.21.15.87:0/3121500620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d38b721b0 con 0x557d38c85bf0 2024-04-22T13:32:29.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40268ef640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 0x7f400c0a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f40180cda60 tx=0x7f401806dc90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f40258ed640 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4010013970 tx=0x7f401000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef125b4d802f3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f4014bfe640 1 -- 172.21.15.87:0/3121500620 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010004030 con 0x557d38c85bf0 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f4014bfe640 1 -- 172.21.15.87:0/3121500620 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f401000e960 con 0x557d38c85bf0 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.933+0000 7f4014bfe640 1 -- 172.21.15.87:0/3121500620 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010003910 con 0x557d38c85bf0 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- 172.21.15.87:0/3121500620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 msgr2=0x557d38c86070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4010013970 tx=0x7f401000b4c0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- 172.21.15.87:0/3121500620 shutdown_connections 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c86070 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7ecf0 0x557d38c85620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 --2- 172.21.15.87:0/3121500620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7e340 0x557d38c7e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- 172.21.15.87:0/3121500620 >> 172.21.15.87:0/3121500620 conn(0x557d38c78ba0 msgr2=0x557d38c78f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- 172.21.15.87:0/3121500620 shutdown_connections 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- 172.21.15.87:0/3121500620 wait complete. 2024-04-22T13:32:29.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 -- start start 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.934+0000 7f402b026740 1 Processor -- start 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c8bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c89370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557d38c8cb90 con 0x557d38c85bf0 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557d38b9c500 con 0x557d38c7e340 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557d38c8cd10 con 0x557d38c7ecf0 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 10 client.0 _ll_register_callbacks cb 0x557d38b6ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35878/0 (socket says 172.21.15.87:35878) 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 -- 172.21.15.87:0/2948529733 learned_addr learned my addr 172.21.15.87:0/2948529733 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40260ee640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c89370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c8bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 -- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 msgr2=0x557d38c89370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c89370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 -- 172.21.15.87:0/2948529733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 msgr2=0x557d38c8bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c8bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 -- 172.21.15.87:0/2948529733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d388f53f0 con 0x557d38c85bf0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f40258ed640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4010013700 tx=0x7f4010013370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a4d856045e5d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f4009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401000eda0 con 0x557d38c85bf0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4010011440 con 0x557d38c85bf0 2024-04-22T13:32:29.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f401000e960 con 0x557d38c85bf0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4010011640 con 0x557d38c85bf0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 10 client.15819 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.935+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557d389bf7d0 con 0x557d38c85bf0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f401000e960 con 0x557d38c85bf0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4010010740 con 0x557d38c85bf0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 10 client.15819 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 10 client.15819 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 20 client.15819 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38c99d80 0x557d38c9c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 4 client.15819 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d38b82930 con 0x557d38c99d80 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.936+0000 7f40260ee640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38c99d80 0x557d38c9c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f40260ee640 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38c99d80 0x557d38c9c170 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac4c0c01b822fadb server_cookie=ccf5c43f827a9b95 in_seq=0 out_seq=0 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f4009ffb640 10 client.15819 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f4009ffb640 1 -- 172.21.15.87:0/2948529733 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d38b82930 con 0x557d38c99d80 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f4009ffb640 10 client.15819 handle_command_reply: tid=0 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 client.15819 shutdown 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 msgr2=0x557d38c898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4010013700 tx=0x7f4010013370 comp rx=0 tx=0).stop 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 shutdown_connections 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38c99d80 0x557d38c9c170 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c898b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f4009ffb640 0 client.15819 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c89370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 --2- 172.21.15.87:0/2948529733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c8bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 >> 172.21.15.87:0/2948529733 conn(0x557d38c78ba0 msgr2=0x557d38c7a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.937+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 shutdown_connections 2024-04-22T13:32:29.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.938+0000 7f402b026740 1 -- 172.21.15.87:0/2948529733 wait complete. 2024-04-22T13:32:29.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.938+0000 7f402b026740 20 client.15819 trim_cache size 0 max 16384 2024-04-22T13:32:29.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 Processor -- start 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 -- start start 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c84af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c8baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f40201adfd0 con 0x557d38c85bf0 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f40201ab8a0 con 0x557d38c7e340 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f402b026740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f40201a96f0 con 0x557d38c7ecf0 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35884/0 (socket says 172.21.15.87:35884) 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 -- 172.21.15.87:0/2509324916 learned_addr learned my addr 172.21.15.87:0/2509324916 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40260ee640 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c8baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 -- 172.21.15.87:0/2509324916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 msgr2=0x557d38c8baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c8baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 -- 172.21.15.87:0/2509324916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 msgr2=0x557d38c84af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c84af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 -- 172.21.15.87:0/2509324916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d38c74470 con 0x557d38c85bf0 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40260ee640 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c8baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f40258ed640 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4010013fd0 tx=0x7f4010005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d5603a2bed3dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.951+0000 7f400bfff640 1 -- 172.21.15.87:0/2509324916 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401000bea0 con 0x557d38c85bf0 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f400bfff640 1 -- 172.21.15.87:0/2509324916 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4010011440 con 0x557d38c85bf0 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f400bfff640 1 -- 172.21.15.87:0/2509324916 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010010430 con 0x557d38c85bf0 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- 172.21.15.87:0/2509324916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 msgr2=0x557d38c8bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4010013fd0 tx=0x7f4010005630 comp rx=0 tx=0).stop 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- 172.21.15.87:0/2509324916 shutdown_connections 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c85bf0 0x557d38c8bfe0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c7ecf0 0x557d38c8baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 --2- 172.21.15.87:0/2509324916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c84af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- 172.21.15.87:0/2509324916 >> 172.21.15.87:0/2509324916 conn(0x557d38c78ba0 msgr2=0x557d38c95830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- 172.21.15.87:0/2509324916 shutdown_connections 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- 172.21.15.87:0/2509324916 wait complete. 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 -- start start 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.952+0000 7f402b026740 1 Processor -- start 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c9f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c85bf0 0x557d38c9ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f40201a9d60 con 0x557d38c7ecf0 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f40201ab1d0 con 0x557d38c7e340 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f40201aa430 con 0x557d38c85bf0 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 10 client.0 _ll_register_callbacks cb 0x7f4010013870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35886/0 (socket says 172.21.15.87:35886) 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 -- 172.21.15.87:0/3487048301 learned_addr learned my addr 172.21.15.87:0/3487048301 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:29.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40258ed640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c9f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40258ed640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c85bf0 0x557d38c9ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 -- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c85bf0 msgr2=0x557d38c9ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c85bf0 0x557d38c9ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 -- 172.21.15.87:0/3487048301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 msgr2=0x557d38c9f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c9f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 -- 172.21.15.87:0/3487048301 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d38c9b3a0 con 0x557d38c7ecf0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40258ed640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c9f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f40260ee640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f401c000f80 tx=0x7f401c02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26212e0e00292475 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f4009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c039040 con 0x557d38c7ecf0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f401c0324c0 con 0x557d38c7ecf0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4010011440 con 0x557d38c7ecf0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f401c059050 con 0x557d38c7ecf0 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 10 client.15825 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:29.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.953+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557d38cffd80 con 0x557d38c7ecf0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4010011440 con 0x557d38c7ecf0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f401c0041d0 con 0x557d38c7ecf0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 10 client.15825 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 10 client.15825 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 20 client.15825 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38d00530 0x557d38d02950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 4 client.15825 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d38c89c40 con 0x557d38d00530 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f40258ed640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38d00530 0x557d38d02950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f40258ed640 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38d00530 0x557d38d02950 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df4cf8ea6fa368d3 server_cookie=74075365c9cd0cbf in_seq=0 out_seq=0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.954+0000 7f4009ffb640 10 client.15825 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4009ffb640 1 -- 172.21.15.87:0/3487048301 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d38c89c40 con 0x557d38d00530 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4009ffb640 10 client.15825 handle_command_reply: tid=0 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 client.15825 shutdown 2024-04-22T13:32:29.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 msgr2=0x557d38c9a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f401c000f80 tx=0x7f401c02c800 comp rx=0 tx=0).stop 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 shutdown_connections 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557d38d00530 0x557d38d02950 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d38c85bf0 0x557d38c9ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4009ffb640 0 client.15825 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d38c7ecf0 0x557d38c9a7c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 --2- 172.21.15.87:0/3487048301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d38c7e340 0x557d38c9f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 >> 172.21.15.87:0/3487048301 conn(0x557d38c78ba0 msgr2=0x557d38c77370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 shutdown_connections 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 1 -- 172.21.15.87:0/3487048301 wait complete. 2024-04-22T13:32:29.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f402b026740 20 client.15825 trim_cache size 0 max 16384 2024-04-22T13:32:29.959 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:32:29.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 msgr2=0x7f400c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 0x7f400c0a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f40180cda60 tx=0x7f401806dc90 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 msgr2=0x7f402019c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f401800d750 tx=0x7f401800dc10 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 shutdown_connections 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f400c0a3ed0 0x7f400c0a6390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f402019be40 0x7f402019c2f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4020106160 0x7f4020197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 --2- 172.21.15.87:0/591584545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4020101090 0x7f4020197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.955+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 >> 172.21.15.87:0/591584545 conn(0x7f4020077f50 msgr2=0x7f40200fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.956+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 shutdown_connections 2024-04-22T13:32:29.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:29.956+0000 7f4028352640 1 -- 172.21.15.87:0/591584545 wait complete. 2024-04-22T13:32:29.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 -- 172.21.15.87:0/60703714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 msgr2=0x7f73c8102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/60703714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c8102570 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009900 tx=0x7f73bc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 -- 172.21.15.87:0/60703714 shutdown_connections 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/60703714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/60703714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c8102b40 0x7f73c8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/60703714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c8102570 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 -- 172.21.15.87:0/60703714 >> 172.21.15.87:0/60703714 conn(0x7f73c80793a0 msgr2=0x7f73c80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 -- 172.21.15.87:0/60703714 shutdown_connections 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.067+0000 7f73cc8a0640 1 -- 172.21.15.87:0/60703714 wait complete. 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 Processor -- start 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 -- start start 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c81986a0 0x7f73c8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f73c8112220 con 0x7f73c8102190 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81120a0 con 0x7f73c8107260 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73cc8a0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81123a0 con 0x7f73c81986a0 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73c659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73c659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35902/0 (socket says 172.21.15.87:35902) 2024-04-22T13:32:30.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.068+0000 7f73c659c640 1 -- 172.21.15.87:0/1234711300 learned_addr learned my addr 172.21.15.87:0/1234711300 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c81986a0 0x7f73c8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 -- 172.21.15.87:0/1234711300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c81986a0 msgr2=0x7f73c8198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c81986a0 0x7f73c8198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 -- 172.21.15.87:0/1234711300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 msgr2=0x7f73c819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 -- 172.21.15.87:0/1234711300 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c81992a0 con 0x7f73c8102190 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73c659c640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f73bc005940 tx=0x7f73bc02c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73b77fe640 1 -- 172.21.15.87:0/1234711300 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc043020 con 0x7f73c8102190 2024-04-22T13:32:30.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73b77fe640 1 -- 172.21.15.87:0/1234711300 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc02cc60 con 0x7f73c8102190 2024-04-22T13:32:30.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c8199530 con 0x7f73c8102190 2024-04-22T13:32:30.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73b77fe640 1 -- 172.21.15.87:0/1234711300 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc003c30 con 0x7f73c8102190 2024-04-22T13:32:30.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.069+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c81a5680 con 0x7f73c8102190 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.071+0000 7f73b77fe640 1 -- 172.21.15.87:0/1234711300 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f73bc046020 con 0x7f73c8102190 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.071+0000 7f73cf568740 1 Processor -- start 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 -- start start 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 0x55b2709e3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 0x55b2709e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b27093f1f0 con 0x55b2709da4d0 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2708dc4b0 con 0x55b2709dae80 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73cf568740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b27093b1b0 con 0x55b2709e3a00 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35916/0 (socket says 172.21.15.87:35916) 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 -- 172.21.15.87:0/2871822519 learned_addr learned my addr 172.21.15.87:0/2871822519 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73b77fe640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 0x7f73b00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c659c640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 0x55b2709e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 0x55b2709e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 0x7f73b00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73b77fe640 1 -- 172.21.15.87:0/1234711300 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f73bc0f2e20 con 0x7f73c8102190 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 -- 172.21.15.87:0/2871822519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 msgr2=0x55b2709e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 0x55b2709e5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 -- 172.21.15.87:0/2871822519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 msgr2=0x55b2709e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 0x55b2709e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c6d9d640 1 -- 172.21.15.87:0/2871822519 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2708ce1b0 con 0x55b2709da4d0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 0x55b2709e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.072+0000 7f73c659c640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 0x55b2709e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f73c0012890 tx=0x7f73c0012d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649eb5234c92f492 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73b4ff9640 1 -- 172.21.15.87:0/2871822519 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0003d40 con 0x55b2709da4d0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73b4ff9640 1 -- 172.21.15.87:0/2871822519 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c0004b60 con 0x55b2709da4d0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73b4ff9640 1 -- 172.21.15.87:0/2871822519 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0005050 con 0x55b2709da4d0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73cf568740 1 -- 172.21.15.87:0/2871822519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 msgr2=0x55b2709da8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 0x7f73b00a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73c001f000 tx=0x7f73c001e000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.073+0000 7f73cf568740 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f73c0012890 tx=0x7f73c0012d50 comp rx=0 tx=0).stop 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- 172.21.15.87:0/2871822519 shutdown_connections 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709e3a00 0x55b2709e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709dae80 0x55b2709e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- 172.21.15.87:0/2871822519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709da8b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- 172.21.15.87:0/2871822519 >> 172.21.15.87:0/2871822519 conn(0x55b2709d4ba0 msgr2=0x55b2709d4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- 172.21.15.87:0/2871822519 shutdown_connections 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- 172.21.15.87:0/2871822519 wait complete. 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- start start 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 Processor -- start 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b2709e6970 con 0x55b2709dae80 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2708f8500 con 0x55b2709e3a00 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b2709e6af0 con 0x55b2709da4d0 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73cf568740 10 client.0 _ll_register_callbacks cb 0x55b2708c6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35922/0 (socket says 172.21.15.87:35922) 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73c5d9b640 1 -- 172.21.15.87:0/905952705 learned_addr learned my addr 172.21.15.87:0/905952705 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.074+0000 7f73c659c640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 -- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 msgr2=0x55b2709e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 -- 172.21.15.87:0/905952705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 msgr2=0x55b2709e0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 -- 172.21.15.87:0/905952705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2706513f0 con 0x55b2709dae80 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f73b800e280 tx=0x7f73b800e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e65a7740e14cf13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8018020 con 0x55b2709dae80 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f73b8013660 con 0x55b2709dae80 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73c0004b60 con 0x55b2709dae80 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f73b8023050 con 0x55b2709dae80 2024-04-22T13:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 10 client.15837 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b27071b7d0 con 0x55b2709dae80 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c0004b60 con 0x55b2709dae80 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f73b80149d0 con 0x55b2709dae80 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 10 client.15837 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 10 client.15837 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 20 client.15837 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.075+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b2709f3d80 0x55b2709f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.076+0000 7f73cf568740 4 client.15837 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.076+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2708de930 con 0x55b2709f3d80 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.076+0000 7f73c659c640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b2709f3d80 0x55b2709f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.076+0000 7f73adffb640 10 client.15837 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.076+0000 7f73c659c640 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b2709f3d80 0x55b2709f61a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0e86102ccd3faa5 server_cookie=fc3efe9aa682beaf in_seq=0 out_seq=0 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73adffb640 1 -- 172.21.15.87:0/905952705 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2708de930 con 0x55b2709f3d80 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73adffb640 10 client.15837 handle_command_reply: tid=0 2024-04-22T13:32:30.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 client.15837 shutdown 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 msgr2=0x55b2709e06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f73b800e280 tx=0x7f73b800e740 comp rx=0 tx=0).stop 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 shutdown_connections 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b2709f3d80 0x55b2709f61a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73adffb640 0 client.15837 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e06d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 --2- 172.21.15.87:0/905952705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 >> 172.21.15.87:0/905952705 conn(0x55b2709d4ba0 msgr2=0x55b2709d68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 shutdown_connections 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 1 -- 172.21.15.87:0/905952705 wait complete. 2024-04-22T13:32:30.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.077+0000 7f73cf568740 20 client.15837 trim_cache size 0 max 16384 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 Processor -- start 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 -- start start 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709dae80 0x55b2709e2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81ae0f0 con 0x55b2709da4d0 2024-04-22T13:32:30.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81ab9c0 con 0x55b2709e3a00 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.090+0000 7f73cf568740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81a9890 con 0x55b2709dae80 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35928/0 (socket says 172.21.15.87:35928) 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 -- 172.21.15.87:0/3767843801 learned_addr learned my addr 172.21.15.87:0/3767843801 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 -- 172.21.15.87:0/3767843801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709dae80 msgr2=0x55b2709e2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709dae80 0x55b2709e2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 -- 172.21.15.87:0/3767843801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 msgr2=0x55b2709e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 -- 172.21.15.87:0/3767843801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2709f2880 con 0x55b2709da4d0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73c659c640 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0bec90 tx=0x7f73bc08e5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1aa36d6155fd49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73affff640 1 -- 172.21.15.87:0/3767843801 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc089810 con 0x55b2709da4d0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73affff640 1 -- 172.21.15.87:0/3767843801 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc0bf400 con 0x55b2709da4d0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.091+0000 7f73affff640 1 -- 172.21.15.87:0/3767843801 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc0bb370 con 0x55b2709da4d0 2024-04-22T13:32:30.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- 172.21.15.87:0/3767843801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 msgr2=0x55b2709ef830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0bec90 tx=0x7f73bc08e5e0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- 172.21.15.87:0/3767843801 shutdown_connections 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b2709e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709dae80 0x55b2709e2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- 172.21.15.87:0/3767843801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709da4d0 0x55b2709ef830 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- 172.21.15.87:0/3767843801 >> 172.21.15.87:0/3767843801 conn(0x55b2709d4ba0 msgr2=0x55b2709d6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- 172.21.15.87:0/3767843801 shutdown_connections 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- 172.21.15.87:0/3767843801 wait complete. 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- start start 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 Processor -- start 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b270a522a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81a9ee0 con 0x55b2709dae80 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81ab2f0 con 0x55b2709e3a00 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f73c81aa550 con 0x55b2709da4d0 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73cf568740 10 client.0 _ll_register_callbacks cb 0x7f73b800cc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35938/0 (socket says 172.21.15.87:35938) 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.092+0000 7f73c5d9b640 1 -- 172.21.15.87:0/360562058 learned_addr learned my addr 172.21.15.87:0/360562058 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c659c640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 -- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 msgr2=0x55b2709e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 -- 172.21.15.87:0/360562058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 msgr2=0x55b270a522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b270a522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b270a522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 -- 172.21.15.87:0/360562058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b270a52960 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c6d9d640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b270a522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73c5d9b640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f73b8000c00 tx=0x7f73b800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7d2c67c9e86cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b802b030 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f73b8014400 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73bc0bf400 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f73b8018040 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73cf568740 10 client.15843 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b270a5ae00 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc0bf400 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.093+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f73b8013660 con 0x55b2709dae80 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 10 client.15843 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 10 client.15843 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:32:30.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 20 client.15843 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b270a5b520 0x55b270a5d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 4 client.15843 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b27094cc10 con 0x55b270a5b520 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73c659c640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b270a5b520 0x55b270a5d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73adffb640 10 client.15843 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73c659c640 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b270a5b520 0x55b270a5d940 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b26432dcd7008f0f server_cookie=17ce61b5fda8baae in_seq=0 out_seq=0 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73adffb640 1 -- 172.21.15.87:0/360562058 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b27094cc10 con 0x55b270a5b520 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73adffb640 10 client.15843 handle_command_reply: tid=0 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 client.15843 shutdown 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 msgr2=0x55b2709e8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f73b8000c00 tx=0x7f73b800c4b0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 shutdown_connections 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55b270a5b520 0x55b270a5d940 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2709e3a00 0x55b270a522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73adffb640 0 client.15843 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.094+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2709dae80 0x55b2709e8820 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cf568740 1 --2- 172.21.15.87:0/360562058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2709da4d0 0x55b2709e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 >> 172.21.15.87:0/360562058 conn(0x55b2709d4ba0 msgr2=0x55b2709db260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 shutdown_connections 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cf568740 1 -- 172.21.15.87:0/360562058 wait complete. 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cf568740 20 client.15843 trim_cache size 0 max 16384 2024-04-22T13:32:30.098 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 msgr2=0x7f73b00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 0x7f73b00a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73c001f000 tx=0x7f73c001e000 comp rx=0 tx=0).stop 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 msgr2=0x7f73c819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f73bc005940 tx=0x7f73bc02c860 comp rx=0 tx=0).stop 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 shutdown_connections 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f73b00a3ed0 0x7f73b00a6390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73c81986a0 0x7f73c8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73c8107260 0x7f73c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 --2- 172.21.15.87:0/1234711300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73c8102190 0x7f73c819e590 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 >> 172.21.15.87:0/1234711300 conn(0x7f73c80793a0 msgr2=0x7f73c8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 shutdown_connections 2024-04-22T13:32:30.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.095+0000 7f73cc8a0640 1 -- 172.21.15.87:0/1234711300 wait complete. 2024-04-22T13:32:30.101 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:30 smithi087 ceph-mon[27999]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-22T13:32:30.101 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:30 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2059725230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:30.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- 172.21.15.87:0/250686319 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 msgr2=0x7f2a2010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 --2- 172.21.15.87:0/250686319 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a2010ee40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009ca0 tx=0x7f2a1802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- 172.21.15.87:0/250686319 shutdown_connections 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 --2- 172.21.15.87:0/250686319 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a2010ee40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 --2- 172.21.15.87:0/250686319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2006b1d0 0x7f2a20106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 --2- 172.21.15.87:0/250686319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a2006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- 172.21.15.87:0/250686319 >> 172.21.15.87:0/250686319 conn(0x7f2a20077eb0 msgr2=0x7f2a200782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- 172.21.15.87:0/250686319 shutdown_connections 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- 172.21.15.87:0/250686319 wait complete. 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 Processor -- start 2024-04-22T13:32:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.205+0000 7f2a280c1640 1 -- start start 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a201986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2019cf40 0x7f2a2019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201121b0 con 0x7f2a20107270 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20112030 con 0x7f2a2006a820 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20112330 con 0x7f2a2019cf40 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35954/0 (socket says 172.21.15.87:35954) 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 -- 172.21.15.87:0/128538225 learned_addr learned my addr 172.21.15.87:0/128538225 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2665e640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2019cf40 0x7f2a2019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a201986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 -- 172.21.15.87:0/128538225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2019cf40 msgr2=0x7f2a2019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2019cf40 0x7f2a2019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 -- 172.21.15.87:0/128538225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 msgr2=0x7f2a201986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a201986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 -- 172.21.15.87:0/128538225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2019da30 con 0x7f2a20107270 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a201986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a2565c640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000c970 tx=0x7f2a1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a16ffd640 1 -- 172.21.15.87:0/128538225 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a10007bb0 con 0x7f2a20107270 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a16ffd640 1 -- 172.21.15.87:0/128538225 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a10007d50 con 0x7f2a20107270 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2019dd20 con 0x7f2a20107270 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a16ffd640 1 -- 172.21.15.87:0/128538225 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000f450 con 0x7f2a20107270 2024-04-22T13:32:30.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.206+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a201a5510 con 0x7f2a20107270 2024-04-22T13:32:30.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.209+0000 7f2a16ffd640 1 -- 172.21.15.87:0/128538225 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f2a10004f20 con 0x7f2a20107270 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.209+0000 7f2a2ad8c740 1 Processor -- start 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.209+0000 7f2a2ad8c740 1 -- start start 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.209+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.209+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 0x5569698e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5569698421f0 con 0x5569698dde80 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569697df4b0 con 0x5569698e6a00 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55696983e1b0 con 0x5569698dd4d0 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35960/0 (socket says 172.21.15.87:35960) 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 -- 172.21.15.87:0/3393246257 learned_addr learned my addr 172.21.15.87:0/3393246257 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2665e640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a16ffd640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 0x7f2a0c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 -- 172.21.15.87:0/3393246257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 msgr2=0x5569698dd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698dd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2565c640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 0x5569698e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 -- 172.21.15.87:0/3393246257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 msgr2=0x5569698e8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 0x5569698e8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 -- 172.21.15.87:0/3393246257 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569697d11b0 con 0x5569698dde80 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2665e640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2565c640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 0x5569698e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a16ffd640 1 -- 172.21.15.87:0/128538225 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2a100d0a40 con 0x7f2a20107270 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a2665e640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 0x7f2a0c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00bcc0 tx=0x7f2a1c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9a50d3e5f92f7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a0bfff640 1 -- 172.21.15.87:0/3393246257 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c005020 con 0x5569698dde80 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a0bfff640 1 -- 172.21.15.87:0/3393246257 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a1c0047a0 con 0x5569698dde80 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.210+0000 7f2a0bfff640 1 -- 172.21.15.87:0/3393246257 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c011680 con 0x5569698dde80 2024-04-22T13:32:30.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3393246257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 msgr2=0x5569698e6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2665e640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 0x7f2a0c0a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a18004af0 tx=0x7f2a1802e040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00bcc0 tx=0x7f2a1c00f730 comp rx=0 tx=0).stop 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3393246257 shutdown_connections 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698e6a00 0x5569698e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e6430 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3393246257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3393246257 >> 172.21.15.87:0/3393246257 conn(0x5569698d7ba0 msgr2=0x5569698d7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3393246257 shutdown_connections 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.211+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3393246257 wait complete. 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 -- start start 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 Processor -- start 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698e6a00 0x5569698e3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5569698e9970 con 0x5569698dde80 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569697fb500 con 0x5569698dd4d0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569698e9af0 con 0x5569698e6a00 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2ad8c740 10 client.0 _ll_register_callbacks cb 0x5569697c9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35962/0 (socket says 172.21.15.87:35962) 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 -- 172.21.15.87:0/3857664210 learned_addr learned my addr 172.21.15.87:0/3857664210 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2665e640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698e6a00 0x5569698e3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 -- 172.21.15.87:0/3857664210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698e6a00 msgr2=0x5569698e3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698e6a00 0x5569698e3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 -- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 msgr2=0x5569698e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.212+0000 7f2a2565c640 1 -- 172.21.15.87:0/3857664210 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569695543f0 con 0x5569698dde80 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2565c640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2a1009c7a0 tx=0x7f2a10067360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a0d3f7a4f7af6f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1001d040 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2a10005210 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a1c0047a0 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a10005410 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 10 client.15855 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55696961e7d0 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a1c0047a0 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2a100a4ff0 con 0x5569698dde80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 10 client.15855 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 20 client.15855 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698f6b80 0x5569698f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 4 client.15855 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.213+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569697e1930 con 0x5569698f6b80 2024-04-22T13:32:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.214+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698f6b80 0x5569698f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.214+0000 7f2a097fa640 10 client.15855 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.214+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698f6b80 0x5569698f8f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7674cb7c021d30a server_cookie=9f846c19df9afcaf in_seq=0 out_seq=0 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a097fa640 1 -- 172.21.15.87:0/3857664210 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569697e1930 con 0x5569698f6b80 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a097fa640 10 client.15855 handle_command_reply: tid=0 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 client.15855 shutdown 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 msgr2=0x5569698e3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2a1009c7a0 tx=0x7f2a10067360 comp rx=0 tx=0).stop 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 shutdown_connections 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698f6b80 0x5569698f8f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698e6a00 0x5569698e3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e3630 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/3857664210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a097fa640 0 client.15855 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 >> 172.21.15.87:0/3857664210 conn(0x5569698d7ba0 msgr2=0x5569698d98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 shutdown_connections 2024-04-22T13:32:30.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/3857664210 wait complete. 2024-04-22T13:32:30.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.215+0000 7f2a2ad8c740 20 client.15855 trim_cache size 0 max 16384 2024-04-22T13:32:30.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.228+0000 7f2a2ad8c740 1 Processor -- start 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.228+0000 7f2a2ad8c740 1 -- start start 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 0x5569698f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201b01f0 con 0x5569698dde80 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201adac0 con 0x5569698d7ba0 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201ab930 con 0x5569698dd4d0 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35964/0 (socket says 172.21.15.87:35964) 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 -- 172.21.15.87:0/2159594988 learned_addr learned my addr 172.21.15.87:0/2159594988 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 0x5569698f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2565c640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 -- 172.21.15.87:0/2159594988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 msgr2=0x5569698d6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 -- 172.21.15.87:0/2159594988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 msgr2=0x5569698f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 0x5569698f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 -- 172.21.15.87:0/2159594988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569698ddc20 con 0x5569698dde80 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 0x5569698f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a2665e640 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2a18037040 tx=0x7f2a18040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e5e8fc24af9c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a0b7fe640 1 -- 172.21.15.87:0/2159594988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18039040 con 0x5569698dde80 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a0b7fe640 1 -- 172.21.15.87:0/2159594988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a18045040 con 0x5569698dde80 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.229+0000 7f2a0b7fe640 1 -- 172.21.15.87:0/2159594988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18034780 con 0x5569698dde80 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/2159594988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 msgr2=0x5569698e5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2a18037040 tx=0x7f2a18040000 comp rx=0 tx=0).stop 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/2159594988 shutdown_connections 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698e5d60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698dd4d0 0x5569698d6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/2159594988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698d7ba0 0x5569698f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/2159594988 >> 172.21.15.87:0/2159594988 conn(0x5569698e3cd0 msgr2=0x5569698d6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/2159594988 shutdown_connections 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/2159594988 wait complete. 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 -- start start 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.230+0000 7f2a2ad8c740 1 Processor -- start 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 0x5569698ea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201abf80 con 0x5569698dde80 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201ad3f0 con 0x5569698dd4d0 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201ac650 con 0x5569698d7ba0 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 10 client.0 _ll_register_callbacks cb 0x7f2a10098b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35976/0 (socket says 172.21.15.87:35976) 2024-04-22T13:32:30.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 -- 172.21.15.87:0/300743079 learned_addr learned my addr 172.21.15.87:0/300743079 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 0x5569698ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2565c640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 -- 172.21.15.87:0/300743079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 msgr2=0x5569698ea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 0x5569698ea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 -- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 msgr2=0x5569698eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698eadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 -- 172.21.15.87:0/300743079 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556969960560 con 0x5569698dde80 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 0x5569698ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2665e640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2a18005fc0 tx=0x7f2a18032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798069595509053f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a097fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18039040 con 0x5569698dde80 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2a18034bb0 con 0x5569698dde80 2024-04-22T13:32:30.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.231+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a18045040 con 0x5569698dde80 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a18034db0 con 0x5569698dde80 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 10 client.15861 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5569699617d0 con 0x5569698dde80 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a18045040 con 0x5569698dde80 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2a1804d5e0 con 0x5569698dde80 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 10 client.15861 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 20 client.15861 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698e6a00 0x556969963fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 4 client.15861 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55696984fc10 con 0x5569698e6a00 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.232+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698e6a00 0x556969963fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a097fa640 10 client.15861 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a25e5d640 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698e6a00 0x556969963fe0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78138feb384edffd server_cookie=729cc4d75a0ffe7 in_seq=0 out_seq=0 2024-04-22T13:32:30.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a097fa640 1 -- 172.21.15.87:0/300743079 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55696984fc10 con 0x5569698e6a00 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a097fa640 10 client.15861 handle_command_reply: tid=0 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 client.15861 shutdown 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 msgr2=0x5569698eb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2a18005fc0 tx=0x7f2a18032f60 comp rx=0 tx=0).stop 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 shutdown_connections 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569698e6a00 0x556969963fe0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569698dde80 0x5569698eb2f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569698dd4d0 0x5569698eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a097fa640 0 client.15861 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 --2- 172.21.15.87:0/300743079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569698d7ba0 0x5569698ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 >> 172.21.15.87:0/300743079 conn(0x5569698e3cd0 msgr2=0x5569698deb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 shutdown_connections 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 1 -- 172.21.15.87:0/300743079 wait complete. 2024-04-22T13:32:30.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.233+0000 7f2a2ad8c740 20 client.15861 trim_cache size 0 max 16384 2024-04-22T13:32:30.237 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 msgr2=0x7f2a0c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 0x7f2a0c0a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a18004af0 tx=0x7f2a1802e040 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 msgr2=0x7f2a20198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000c970 tx=0x7f2a1000ce30 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 shutdown_connections 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a0c0a3ed0 0x7f2a0c0a6390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2019cf40 0x7f2a2019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a20107270 0x7f2a20198c10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 --2- 172.21.15.87:0/128538225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2006a820 0x7f2a201986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 >> 172.21.15.87:0/128538225 conn(0x7f2a20077eb0 msgr2=0x7f2a200fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 shutdown_connections 2024-04-22T13:32:30.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:30.234+0000 7f2a280c1640 1 -- 172.21.15.87:0/128538225 wait complete. 2024-04-22T13:32:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:30 smithi120 ceph-mon[31101]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-22T13:32:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:30 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2059725230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:30.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:30 smithi132 ceph-mon[31531]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-22T13:32:30.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:30 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2059725230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:31.241 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:32:31.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:32:31.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:32:31.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 -- 172.21.15.87:0/899207937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c4102b40 msgr2=0x7f77c4106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/899207937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c4102b40 0x7f77c4106b80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009900 tx=0x7f77b802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 -- 172.21.15.87:0/899207937 shutdown_connections 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/899207937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4107260 0x7f77c410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/899207937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c4102b40 0x7f77c4106b80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/899207937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4102190 0x7f77c4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 -- 172.21.15.87:0/899207937 >> 172.21.15.87:0/899207937 conn(0x7f77c40fd990 msgr2=0x7f77c40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 -- 172.21.15.87:0/899207937 shutdown_connections 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.360+0000 7f77ca3f4640 1 -- 172.21.15.87:0/899207937 wait complete. 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 Processor -- start 2024-04-22T13:32:31.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 -- start start 2024-04-22T13:32:31.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 0x7f77c419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4107260 0x7f77c419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41121a0 con 0x7f77c41985a0 2024-04-22T13:32:31.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4112020 con 0x7f77c4102190 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.361+0000 7f77ca3f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4112320 con 0x7f77c4107260 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35996/0 (socket says 172.21.15.87:35996) 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 -- 172.21.15.87:0/1386324383 learned_addr learned my addr 172.21.15.87:0/1386324383 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4107260 0x7f77c419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 0x7f77c419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 -- 172.21.15.87:0/1386324383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4107260 msgr2=0x7f77c419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4107260 0x7f77c419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 -- 172.21.15.87:0/1386324383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 msgr2=0x7f77c419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 0x7f77c419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 -- 172.21.15.87:0/1386324383 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c41990c0 con 0x7f77c41985a0 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 0x7f77c419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c8991640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00eaa0 tx=0x7f77bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c17fa640 1 -- 172.21.15.87:0/1386324383 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00cdb0 con 0x7f77c41985a0 2024-04-22T13:32:31.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c41993b0 con 0x7f77c41985a0 2024-04-22T13:32:31.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c17fa640 1 -- 172.21.15.87:0/1386324383 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77bc005020 con 0x7f77c41985a0 2024-04-22T13:32:31.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77c17fa640 1 -- 172.21.15.87:0/1386324383 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc0116d0 con 0x7f77c41985a0 2024-04-22T13:32:31.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.362+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c41a55b0 con 0x7f77c41985a0 2024-04-22T13:32:31.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.365+0000 7f77c17fa640 1 -- 172.21.15.87:0/1386324383 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f77bc0051d0 con 0x7f77c41985a0 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.365+0000 7f77ccfeb740 1 Processor -- start 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.365+0000 7f77ccfeb740 1 -- start start 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.365+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd01216cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.365+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 0x55fd0121f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fd0117af00 con 0x55fd0121fe10 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd01118590 con 0x55fd01217290 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fd01176ec0 con 0x55fd012168e0 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36000/0 (socket says 172.21.15.87:36000) 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 -- 172.21.15.87:0/1390067458 learned_addr learned my addr 172.21.15.87:0/1390067458 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c17fa640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 0x7f77ac0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd01216cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 0x55fd0121f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c8991640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 0x7f77ac0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 -- 172.21.15.87:0/1390067458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 msgr2=0x55fd01216cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd01216cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 -- 172.21.15.87:0/1390067458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 msgr2=0x55fd0121f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 0x55fd0121f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 -- 172.21.15.87:0/1390067458 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd00f557e0 con 0x55fd0121fe10 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 0x55fd0121f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c17fa640 1 -- 172.21.15.87:0/1386324383 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f77bc0d0c10 con 0x7f77c41985a0 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f77b80096c0 tx=0x7f77b8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73449ab0ecff08a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77c8991640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 0x7f77ac0a6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0ce1f0 tx=0x7f77bc005890 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77aa7fc640 1 -- 172.21.15.87:0/1390067458 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b804a070 con 0x55fd0121fe10 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77aa7fc640 1 -- 172.21.15.87:0/1390067458 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b8004150 con 0x55fd0121fe10 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.366+0000 7f77aa7fc640 1 -- 172.21.15.87:0/1390067458 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8043450 con 0x55fd0121fe10 2024-04-22T13:32:31.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1390067458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 msgr2=0x55fd01222260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f77b80096c0 tx=0x7f77b8005e40 comp rx=0 tx=0).stop 2024-04-22T13:32:31.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1390067458 shutdown_connections 2024-04-22T13:32:31.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01222260 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd01217290 0x55fd0121f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1390067458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd01216cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1390067458 >> 172.21.15.87:0/1390067458 conn(0x55fd01210fb0 msgr2=0x55fd01211390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1390067458 shutdown_connections 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1390067458 wait complete. 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.367+0000 7f77ccfeb740 1 -- start start 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 Processor -- start 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0121ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd0121d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fd01222d20 con 0x55fd01217290 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd01134580 con 0x55fd0121fe10 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fd01222ea0 con 0x55fd012168e0 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77ccfeb740 10 client.0 _ll_register_callbacks cb 0x55fd0113a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:31.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36014/0 (socket says 172.21.15.87:36014) 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 -- 172.21.15.87:0/1496568928 learned_addr learned my addr 172.21.15.87:0/1496568928 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd0121d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0121ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 -- 172.21.15.87:0/1496568928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 msgr2=0x55fd0121ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0121ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 -- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 msgr2=0x55fd0121d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd0121d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c3fff640 1 -- 172.21.15.87:0/1496568928 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd00fb5f70 con 0x55fd01217290 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.368+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0121ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f77b400f760 tx=0x7f77b400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0e0cbdc296af64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4016070 con 0x55fd01217290 2024-04-22T13:32:31.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f77b400bd10 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77b8004150 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f77b400fe70 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 10 client.15873 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fd00e4d170 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b8004150 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f77b4022030 con 0x55fd01217290 2024-04-22T13:32:31.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 10 client.15873 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:31.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:31.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 20 client.15873 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:31.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd012303f0 0x55fd01232800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 4 client.15873 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.369+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd01188920 con 0x55fd012303f0 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.370+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd012303f0 0x55fd01232800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.370+0000 7f77a37fe640 10 client.15873 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.370+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd012303f0 0x55fd01232800 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c7e54acc200cd6 server_cookie=b8de36c12643a341 in_seq=0 out_seq=0 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77a37fe640 1 -- 172.21.15.87:0/1496568928 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd01188920 con 0x55fd012303f0 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77a37fe640 10 client.15873 handle_command_reply: tid=0 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 client.15873 shutdown 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 msgr2=0x55fd0121d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f77b400f760 tx=0x7f77b400fc20 comp rx=0 tx=0).stop 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 shutdown_connections 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd012303f0 0x55fd01232800 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd0121d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d380 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1496568928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0121ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 >> 172.21.15.87:0/1496568928 conn(0x55fd01210fb0 msgr2=0x55fd01212d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77a37fe640 0 client.15873 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 shutdown_connections 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1496568928 wait complete. 2024-04-22T13:32:31.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.371+0000 7f77ccfeb740 20 client.15873 trim_cache size 0 max 16384 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 Processor -- start 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 -- start start 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0122bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd012380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41b0370 con 0x55fd01217290 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41adc40 con 0x55fd0121fe10 2024-04-22T13:32:31.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41abab0 con 0x55fd012168e0 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36018/0 (socket says 172.21.15.87:36018) 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 -- 172.21.15.87:0/3113939281 learned_addr learned my addr 172.21.15.87:0/3113939281 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c37fe640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0122bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c37fe640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd012380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 -- 172.21.15.87:0/3113939281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 msgr2=0x55fd0122bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0122bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 -- 172.21.15.87:0/3113939281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 msgr2=0x55fd012380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd012380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 -- 172.21.15.87:0/3113939281 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd01210ae0 con 0x55fd01217290 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c37fe640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd012380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77c3fff640 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f77b4010010 tx=0x7f77b401a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4a8d2691c9df27 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77a9ffb640 1 -- 172.21.15.87:0/3113939281 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4031070 con 0x55fd01217290 2024-04-22T13:32:31.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77a9ffb640 1 -- 172.21.15.87:0/3113939281 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b401a9b0 con 0x55fd01217290 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.389+0000 7f77a9ffb640 1 -- 172.21.15.87:0/3113939281 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4016040 con 0x55fd01217290 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- 172.21.15.87:0/3113939281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 msgr2=0x55fd0121d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f77b4010010 tx=0x7f77b401a4b0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- 172.21.15.87:0/3113939281 shutdown_connections 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd0121fe10 0x55fd012380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd01217290 0x55fd0121d6e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/3113939281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd012168e0 0x55fd0122bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- 172.21.15.87:0/3113939281 >> 172.21.15.87:0/3113939281 conn(0x55fd01210fb0 msgr2=0x55fd01210010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- 172.21.15.87:0/3113939281 shutdown_connections 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- 172.21.15.87:0/3113939281 wait complete. 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.390+0000 7f77ccfeb740 1 -- start start 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 Processor -- start 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd012168e0 0x55fd012243b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd01217290 0x55fd012248f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41ac100 con 0x55fd0121fe10 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41ad570 con 0x55fd012168e0 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f77c41ac7d0 con 0x55fd01217290 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77ccfeb740 10 client.0 _ll_register_callbacks cb 0x7f77b400d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36024/0 (socket says 172.21.15.87:36024) 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 -- 172.21.15.87:0/1488965997 learned_addr learned my addr 172.21.15.87:0/1488965997 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd01217290 0x55fd012248f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd012168e0 0x55fd012243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 -- 172.21.15.87:0/1488965997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd01217290 msgr2=0x55fd012248f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd01217290 0x55fd012248f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 -- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd012168e0 msgr2=0x55fd012243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd012168e0 0x55fd012243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.391+0000 7f77c37fe640 1 -- 172.21.15.87:0/1488965997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd012943d0 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77c37fe640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f77b8036040 tx=0x7f77b400dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b248c1a4c43d5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b804a040 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f77b802cb30 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77b401a9b0 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f77b80035f0 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 10 client.15879 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fd0129b730 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b401a9b0 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f77b8043450 con 0x55fd0121fe10 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 10 client.15879 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 20 client.15879 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd0129ba00 0x55fd0129ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 4 client.15879 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.392+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fd01182790 con 0x55fd0129ba00 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.393+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd0129ba00 0x55fd0129ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.393+0000 7f77a37fe640 10 client.15879 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.393+0000 7f77c3fff640 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd0129ba00 0x55fd0129ddf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec707149e27da3c server_cookie=84078fd088c996be in_seq=0 out_seq=0 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77a37fe640 1 -- 172.21.15.87:0/1488965997 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fd01182790 con 0x55fd0129ba00 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77a37fe640 10 client.15879 handle_command_reply: tid=0 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 client.15879 shutdown 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 msgr2=0x55fd01293d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f77b8036040 tx=0x7f77b400dc50 comp rx=0 tx=0).stop 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 shutdown_connections 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd0129ba00 0x55fd0129ddf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd0121fe10 0x55fd01293d90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77a37fe640 0 client.15879 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd01217290 0x55fd012248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 --2- 172.21.15.87:0/1488965997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd012168e0 0x55fd012243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.394+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 >> 172.21.15.87:0/1488965997 conn(0x55fd01210fb0 msgr2=0x55fd01211fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 shutdown_connections 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ccfeb740 1 -- 172.21.15.87:0/1488965997 wait complete. 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ccfeb740 20 client.15879 trim_cache size 0 max 16384 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:32:31.400 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "16aa8f3d-06d0-401b-8ea6-944954aed150", 2024-04-22T13:32:31.401 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:32:31.401 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:31.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 msgr2=0x7f77ac0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 0x7f77ac0a6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0ce1f0 tx=0x7f77bc005890 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 msgr2=0x7f77c4198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00eaa0 tx=0x7f77bc00ef60 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 shutdown_connections 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f77ac0a3ed0 0x7f77ac0a6390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77c41985a0 0x7f77c4198a50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77c4107260 0x7f77c419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 --2- 172.21.15.87:0/1386324383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4102190 0x7f77c419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 >> 172.21.15.87:0/1386324383 conn(0x7f77c40fd990 msgr2=0x7f77c40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 shutdown_connections 2024-04-22T13:32:31.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.395+0000 7f77ca3f4640 1 -- 172.21.15.87:0/1386324383 wait complete. 2024-04-22T13:32:31.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:32:31.403 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:32:31.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 -- 172.21.15.87:0/2883365718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc102170 msgr2=0x7f0acc102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 --2- 172.21.15.87:0/2883365718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc102170 0x7f0acc102550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009900 tx=0x7f0abc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:31.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 -- 172.21.15.87:0/2883365718 shutdown_connections 2024-04-22T13:32:31.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 --2- 172.21.15.87:0/2883365718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc107240 0x7f0acc10ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 --2- 172.21.15.87:0/2883365718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc102b20 0x7f0acc106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 --2- 172.21.15.87:0/2883365718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc102170 0x7f0acc102550 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 -- 172.21.15.87:0/2883365718 >> 172.21.15.87:0/2883365718 conn(0x7f0acc0793a0 msgr2=0x7f0acc0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 -- 172.21.15.87:0/2883365718 shutdown_connections 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 -- 172.21.15.87:0/2883365718 wait complete. 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.507+0000 7f0ad2493640 1 Processor -- start 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 -- start start 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc102170 0x7f0acc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc19cfc0 0x7f0acc19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc112180 con 0x7f0acc107240 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc112000 con 0x7f0acc102170 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad2493640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc112300 con 0x7f0acc19cfc0 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36044/0 (socket says 172.21.15.87:36044) 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 -- 172.21.15.87:0/3960588362 learned_addr learned my addr 172.21.15.87:0/3960588362 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acbfff640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc102170 0x7f0acc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc19cfc0 0x7f0acc19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 -- 172.21.15.87:0/3960588362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc19cfc0 msgr2=0x7f0acc19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc19cfc0 0x7f0acc19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 -- 172.21.15.87:0/3960588362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc102170 msgr2=0x7f0acc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc102170 0x7f0acc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.508+0000 7f0acb7fe640 1 -- 172.21.15.87:0/3960588362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc19dab0 con 0x7f0acc107240 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0005fd0 tx=0x7f0ac000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0ac97fa640 1 -- 172.21.15.87:0/3960588362 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0017070 con 0x7f0acc107240 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0ac97fa640 1 -- 172.21.15.87:0/3960588362 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac0009c30 con 0x7f0acc107240 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc19dda0 con 0x7f0acc107240 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0ac97fa640 1 -- 172.21.15.87:0/3960588362 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0003d30 con 0x7f0acc107240 2024-04-22T13:32:31.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.509+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1a5590 con 0x7f0acc107240 2024-04-22T13:32:31.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ac97fa640 1 -- 172.21.15.87:0/3960588362 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0ac0003ed0 con 0x7f0acc107240 2024-04-22T13:32:31.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 Processor -- start 2024-04-22T13:32:31.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 -- start start 2024-04-22T13:32:31.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a7ffff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80008540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558a7ff641e0 con 0x558a80008a80 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a7ff01390 con 0x558a7ffff580 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0ad5164740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a7ff601a0 con 0x558a7ffffea0 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.511+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36056/0 (socket says 172.21.15.87:36056) 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 -- 172.21.15.87:0/1202270818 learned_addr learned my addr 172.21.15.87:0/1202270818 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a7ffff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ac97fa640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 0x7f0ab40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acbfff640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80008540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 0x7f0ab40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 -- 172.21.15.87:0/1202270818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 msgr2=0x558a80008540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80008540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 -- 172.21.15.87:0/1202270818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 msgr2=0x558a7ffff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a7ffff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ac97fa640 1 -- 172.21.15.87:0/3960588362 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f0ac00d0770 con 0x7f0acc107240 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 -- 172.21.15.87:0/1202270818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a7fd407d0 con 0x558a80008a80 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a7ffff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0ac009c240 tx=0x7f0ac000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f6ff8a122a3898 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ab27fc640 1 -- 172.21.15.87:0/1202270818 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0009de0 con 0x558a80008a80 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 0x7f0ab40a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4008fd0 tx=0x7f0ac4004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ab27fc640 1 -- 172.21.15.87:0/1202270818 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac009cc40 con 0x558a80008a80 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.512+0000 7f0ab27fc640 1 -- 172.21.15.87:0/1202270818 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0098bb0 con 0x558a80008a80 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- 172.21.15.87:0/1202270818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 msgr2=0x558a8000ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0ac009c240 tx=0x7f0ac000bf60 comp rx=0 tx=0).stop 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- 172.21.15.87:0/1202270818 shutdown_connections 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8000ae70 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80008540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 --2- 172.21.15.87:0/1202270818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a7ffff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- 172.21.15.87:0/1202270818 >> 172.21.15.87:0/1202270818 conn(0x558a7fff9cb0 msgr2=0x558a7fffa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- 172.21.15.87:0/1202270818 shutdown_connections 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- 172.21.15.87:0/1202270818 wait complete. 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.513+0000 7f0ad5164740 1 -- start start 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 Processor -- start 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 0x558a80005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80006520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558a8000b7b0 con 0x558a7ffff580 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a7ff1d2f0 con 0x558a7ffffea0 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a8000b930 con 0x558a80008a80 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad5164740 10 client.0 _ll_register_callbacks cb 0x558a7ff2a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:31.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36060/0 (socket says 172.21.15.87:36060) 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 -- 172.21.15.87:0/2427784968 learned_addr learned my addr 172.21.15.87:0/2427784968 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 0x558a80005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80006520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 -- 172.21.15.87:0/2427784968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 msgr2=0x558a80006520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80006520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 -- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 msgr2=0x558a80005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 0x558a80005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 -- 172.21.15.87:0/2427784968 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a7fc763f0 con 0x558a7ffff580 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 0x558a80005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80006520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:31.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0abc02c170 tx=0x7f0abc02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e5feb247d26324 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.514+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc005230 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f0abc0053d0 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac009cc40 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0abc0055d0 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 10 client.15891 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558a7fc360f0 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac009cc40 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0abc051050 con 0x558a7ffff580 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 10 client.15891 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 20 client.15891 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:31.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a80018930 0x558a8001adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 4 client.15891 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a7ff71c00 con 0x558a80018930 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.515+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a80018930 0x558a8001adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.516+0000 7f0aab7fe640 10 client.15891 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.516+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a80018930 0x558a8001adf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f8616205a15baf1 server_cookie=735a7baa25717bb5 in_seq=0 out_seq=0 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.516+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2427784968 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a7ff71c00 con 0x558a80018930 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.516+0000 7f0aab7fe640 10 client.15891 handle_command_reply: tid=0 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.516+0000 7f0ad5164740 1 client.15891 shutdown 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 msgr2=0x558a80005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0abc02c170 tx=0x7f0abc02c9f0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 shutdown_connections 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a80018930 0x558a8001adf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80006520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0aab7fe640 0 client.15891 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffffea0 0x558a80005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2427784968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffff580 0x558a80005aa0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 >> 172.21.15.87:0/2427784968 conn(0x558a7fff9cb0 msgr2=0x558a7fffba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 shutdown_connections 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 1 -- 172.21.15.87:0/2427784968 wait complete. 2024-04-22T13:32:31.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.517+0000 7f0ad5164740 20 client.15891 trim_cache size 0 max 16384 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.533+0000 7f0ad5164740 1 Processor -- start 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 -- start start 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a800081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80014670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1b0270 con 0x558a7ffffea0 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1adb40 con 0x558a7ffff580 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad5164740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1ab9d0 con 0x558a80008a80 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36064/0 (socket says 172.21.15.87:36064) 2024-04-22T13:32:31.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 -- 172.21.15.87:0/2979470188 learned_addr learned my addr 172.21.15.87:0/2979470188 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a800081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 -- 172.21.15.87:0/2979470188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 msgr2=0x558a80014670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80014670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80014670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 -- 172.21.15.87:0/2979470188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 msgr2=0x558a800081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a800081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 -- 172.21.15.87:0/2979470188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a8000ae70 con 0x558a7ffffea0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a800081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80014670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0ac009c4d0 tx=0x7f0ac00cdce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00ebc05ae7c8ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ab1ffb640 1 -- 172.21.15.87:0/2979470188 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0009de0 con 0x558a7ffffea0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ab1ffb640 1 -- 172.21.15.87:0/2979470188 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac009c540 con 0x558a7ffffea0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.534+0000 7f0ab1ffb640 1 -- 172.21.15.87:0/2979470188 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac009ca00 con 0x558a7ffffea0 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- 172.21.15.87:0/2979470188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 msgr2=0x558a800049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0ac009c4d0 tx=0x7f0ac00cdce0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- 172.21.15.87:0/2979470188 shutdown_connections 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a80008a80 0x558a80014670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a7ffffea0 0x558a800049e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2979470188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a800081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- 172.21.15.87:0/2979470188 >> 172.21.15.87:0/2979470188 conn(0x558a7fff9cb0 msgr2=0x558a7fff8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- 172.21.15.87:0/2979470188 shutdown_connections 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- 172.21.15.87:0/2979470188 wait complete. 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.535+0000 7f0ad5164740 1 -- start start 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 Processor -- start 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a80002a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80002f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1ac020 con 0x558a80008a80 2024-04-22T13:32:31.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1ad470 con 0x558a7ffff580 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1ac6d0 con 0x558a7ffffea0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad5164740 10 client.0 _ll_register_callbacks cb 0x7f0abc005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36074/0 (socket says 172.21.15.87:36074) 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 -- 172.21.15.87:0/2378166900 learned_addr learned my addr 172.21.15.87:0/2378166900 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a80002a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0acb7fe640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80002f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 -- 172.21.15.87:0/2378166900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 msgr2=0x558a80002f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80002f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 -- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 msgr2=0x558a80002a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a80002a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 -- 172.21.15.87:0/2378166900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a8007e4c0 con 0x558a80008a80 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0ad0a30640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4005fd0 tx=0x7f0ac4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc461b718c89983 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:31.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.536+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4018070 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f0ac401fd90 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac009c540 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0ac4003970 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 10 client.15897 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558a8007f500 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac009c540 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0ac4026070 con 0x558a80008a80 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 10 client.15897 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 20 client.15897 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a8007fcb0 0x558a800820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 4 client.15897 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a7ff6ba70 con 0x558a8007fcb0 2024-04-22T13:32:31.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.537+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a8007fcb0 0x558a800820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0acbfff640 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a8007fcb0 0x558a800820d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91b7847b080b9f1 server_cookie=42439f1a41f79615 in_seq=0 out_seq=0 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0aab7fe640 10 client.15897 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0aab7fe640 1 -- 172.21.15.87:0/2378166900 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558a7ff6ba70 con 0x558a8007fcb0 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0aab7fe640 10 client.15897 handle_command_reply: tid=0 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 client.15897 shutdown 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 msgr2=0x558a8007df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4005fd0 tx=0x7f0ac4004a40 comp rx=0 tx=0).stop 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 shutdown_connections 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a8007fcb0 0x558a800820d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a80008a80 0x558a8007df80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a7ffffea0 0x558a80002f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 --2- 172.21.15.87:0/2378166900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a7ffff580 0x558a80002a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0aab7fe640 0 client.15897 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 >> 172.21.15.87:0/2378166900 conn(0x558a7fff9cb0 msgr2=0x558a7fff84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 shutdown_connections 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 1 -- 172.21.15.87:0/2378166900 wait complete. 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.538+0000 7f0ad5164740 20 client.15897 trim_cache size 0 max 16384 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "16aa8f3d-06d0-401b-8ea6-944954aed150": { 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "16aa8f3d-06d0-401b-8ea6-944954aed150", 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:32:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:31.543 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:31.543 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:31.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 msgr2=0x7f0ab40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 0x7f0ab40a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4008fd0 tx=0x7f0ac4004ab0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 msgr2=0x7f0acc198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0005fd0 tx=0x7f0ac000dde0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 shutdown_connections 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0ab40a3ed0 0x7f0ab40a6390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0acc19cfc0 0x7f0acc19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0acc107240 0x7f0acc198c40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 --2- 172.21.15.87:0/3960588362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0acc102170 0x7f0acc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 >> 172.21.15.87:0/3960588362 conn(0x7f0acc0793a0 msgr2=0x7f0acc100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 shutdown_connections 2024-04-22T13:32:31.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:31.539+0000 7f0ad2493640 1 -- 172.21.15.87:0/3960588362 wait complete. 2024-04-22T13:32:31.546 INFO:tasks.cephfs.filesystem:scrub status for tag:16aa8f3d-06d0-401b-8ea6-944954aed150 - {'path': '/', 'tag': '16aa8f3d-06d0-401b-8ea6-944954aed150', 'options': 'recursive,force'} 2024-04-22T13:32:32.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:32 smithi120 ceph-mon[31101]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-04-22T13:32:32.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:32 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:32:32.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:32 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:32.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:32 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:32:32.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:32 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:32.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:32 smithi132 ceph-mon[31531]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-04-22T13:32:32.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:32 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:32:32.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:32 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:32.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:32 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:32:32.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:32 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:32.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:32 smithi087 ceph-mon[27999]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-04-22T13:32:32.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:32 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:32:32.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:32 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:32.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:32 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:32:32.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:32 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:32:34.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:34 smithi120 ceph-mon[31101]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:32:34.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:34 smithi132 ceph-mon[31531]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:32:34.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:34 smithi087 ceph-mon[27999]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-22T13:32:35.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:35 smithi120 ceph-mon[31101]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T13:32:35.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:35 smithi132 ceph-mon[31531]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T13:32:35.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:35 smithi087 ceph-mon[27999]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T13:32:36.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:36 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:32:36.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:36 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:32:36.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:36 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:32:36.847 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:32:36.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 -- 172.21.15.87:0/1802313835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21741011a0 msgr2=0x7f2174101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:36.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 --2- 172.21.15.87:0/1802313835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21741011a0 0x7f2174101620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2164009930 tx=0x7f216402bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:36.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 -- 172.21.15.87:0/1802313835 shutdown_connections 2024-04-22T13:32:36.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 --2- 172.21.15.87:0/1802313835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2174101b60 0x7f217410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:36.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 --2- 172.21.15.87:0/1802313835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21741011a0 0x7f2174101620 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 --2- 172.21.15.87:0/1802313835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 0x7f2174106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 -- 172.21.15.87:0/1802313835 >> 172.21.15.87:0/1802313835 conn(0x7f2174077e30 msgr2=0x7f2174078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 -- 172.21.15.87:0/1802313835 shutdown_connections 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 -- 172.21.15.87:0/1802313835 wait complete. 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.962+0000 7f217a1f4640 1 Processor -- start 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 -- start start 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21741011a0 0x7f21741995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 0x7f21741a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f21741110c0 con 0x7f2174101b60 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2174110f40 con 0x7f21741011a0 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f217a1f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2174111240 con 0x7f2174105e30 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36092/0 (socket says 172.21.15.87:36092) 2024-04-22T13:32:36.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 -- 172.21.15.87:0/781557058 learned_addr learned my addr 172.21.15.87:0/781557058 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f21737fe640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21741011a0 0x7f21741995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2173fff640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 0x7f21741a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 -- 172.21.15.87:0/781557058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 msgr2=0x7f21741a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 0x7f21741a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 -- 172.21.15.87:0/781557058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21741011a0 msgr2=0x7f21741995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21741011a0 0x7f21741995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.963+0000 7f2172ffd640 1 -- 172.21.15.87:0/781557058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21741a4b80 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f2172ffd640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2164009230 tx=0x7f216402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164043020 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21640043d0 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21741a4d50 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164003bc0 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.964+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21741a51c0 con 0x7f2174101b60 2024-04-22T13:32:36.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.965+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2174068de0 con 0x7f2174101b60 2024-04-22T13:32:36.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.966+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f2164046020 con 0x7f2174101b60 2024-04-22T13:32:36.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.966+0000 7f2170ff9640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 0x7f215c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:36.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.967+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f21640f2f60 con 0x7f2174101b60 2024-04-22T13:32:36.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.967+0000 7f21737fe640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 0x7f215c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:36.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.967+0000 7f21737fe640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 0x7f215c0a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2168005d00 tx=0x7f216800b410 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:36.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:36.970+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f216404d020 con 0x7f2174101b60 2024-04-22T13:32:37.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.188+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2174109380 con 0x7f2174101b60 2024-04-22T13:32:37.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.189+0000 7f2170ff9640 1 -- 172.21.15.87:0/781557058 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f21640bf540 con 0x7f2174101b60 2024-04-22T13:32:37.185 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:32:37.185 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:32:37.185 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:32:37.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 msgr2=0x7f215c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 0x7f215c0a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2168005d00 tx=0x7f216800b410 comp rx=0 tx=0).stop 2024-04-22T13:32:37.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 msgr2=0x7f2174199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2164009230 tx=0x7f216402c710 comp rx=0 tx=0).stop 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 shutdown_connections 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f215c0a3ed0 0x7f215c0a6390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2174105e30 0x7f21741a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2174101b60 0x7f2174199b00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 --2- 172.21.15.87:0/781557058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21741011a0 0x7f21741995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 >> 172.21.15.87:0/781557058 conn(0x7f2174077e30 msgr2=0x7f21740feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 shutdown_connections 2024-04-22T13:32:37.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.191+0000 7f217a1f4640 1 -- 172.21.15.87:0/781557058 wait complete. 2024-04-22T13:32:37.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:37 smithi087 ceph-mon[27999]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-22T13:32:37.200 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 3ca5eb30 --release --await' 2024-04-22T13:32:37.200 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 3ca5eb30 --release --await 2024-04-22T13:32:37.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- 172.21.15.87:0/463486019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 msgr2=0x7fc658101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 --2- 172.21.15.87:0/463486019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658101470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc648009900 tx=0x7fc64802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:37.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- 172.21.15.87:0/463486019 shutdown_connections 2024-04-22T13:32:37.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 --2- 172.21.15.87:0/463486019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc65810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 --2- 172.21.15.87:0/463486019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc658101a40 0x7fc658105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 --2- 172.21.15.87:0/463486019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658101470 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- 172.21.15.87:0/463486019 >> 172.21.15.87:0/463486019 conn(0x7fc658077f50 msgr2=0x7fc658078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- 172.21.15.87:0/463486019 shutdown_connections 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- 172.21.15.87:0/463486019 wait complete. 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 Processor -- start 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.310+0000 7fc65cea0640 1 -- start start 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc658198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 0x7fc65819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581121e0 con 0x7fc658101090 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc658112060 con 0x7fc65819cfe0 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc658112360 con 0x7fc658106160 2024-04-22T13:32:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36108/0 (socket says 172.21.15.87:36108) 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 -- 172.21.15.87:0/994398434 learned_addr learned my addr 172.21.15.87:0/994398434 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc64ffff640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc658198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc656d9d640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 0x7fc65819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 -- 172.21.15.87:0/994398434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 msgr2=0x7fc658198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc658198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 -- 172.21.15.87:0/994398434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 msgr2=0x7fc65819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 0x7fc65819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 -- 172.21.15.87:0/994398434 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65819dad0 con 0x7fc658101090 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc64ffff640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc658198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc656d9d640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 0x7fc65819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65659c640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc648005940 tx=0x7fc6480045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc64f7fe640 1 -- 172.21.15.87:0/994398434 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc648043020 con 0x7fc658101090 2024-04-22T13:32:37.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc64f7fe640 1 -- 172.21.15.87:0/994398434 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc64802caf0 con 0x7fc658101090 2024-04-22T13:32:37.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc64f7fe640 1 -- 172.21.15.87:0/994398434 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc648003af0 con 0x7fc658101090 2024-04-22T13:32:37.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65819dd60 con 0x7fc658101090 2024-04-22T13:32:37.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.311+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6581a55b0 con 0x7fc658101090 2024-04-22T13:32:37.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.313+0000 7fc64f7fe640 1 -- 172.21.15.87:0/994398434 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fc648046020 con 0x7fc658101090 2024-04-22T13:32:37.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 Processor -- start 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 -- start start 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a912d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc64f7fe640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 0x7fc6440a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5594c29ed500 con 0x5594c2a91810 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5594c29936a0 con 0x5594c2a88310 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc65fb68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5594c29e94c0 con 0x5594c2a88c30 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc656d9d640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 0x7fc6440a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36114/0 (socket says 172.21.15.87:36114) 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc64ffff640 1 -- 172.21.15.87:0/3989827786 learned_addr learned my addr 172.21.15.87:0/3989827786 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:37.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc64f7fe640 1 -- 172.21.15.87:0/994398434 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc6480f2b20 con 0x7fc658101090 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.314+0000 7fc656d9d640 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc65659c640 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 -- 172.21.15.87:0/3989827786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 msgr2=0x5594c2a912d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a912d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 -- 172.21.15.87:0/3989827786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 msgr2=0x5594c2a886f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a886f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 -- 172.21.15.87:0/3989827786 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594c29b25e0 con 0x5594c2a91810 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64ffff640 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc640009cb0 tx=0x7fc6400117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6d0325b421d972 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc656d9d640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 0x7fc6440a6410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc650005c40 tx=0x7fc6500127c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64cff9640 1 -- 172.21.15.87:0/3989827786 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640004020 con 0x5594c2a91810 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64cff9640 1 -- 172.21.15.87:0/3989827786 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6400041c0 con 0x5594c2a91810 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.315+0000 7fc64cff9640 1 -- 172.21.15.87:0/3989827786 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6400127a0 con 0x5594c2a91810 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- 172.21.15.87:0/3989827786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 msgr2=0x5594c2a93c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc640009cb0 tx=0x7fc6400117b0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- 172.21.15.87:0/3989827786 shutdown_connections 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a93c00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a912d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3989827786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a886f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- 172.21.15.87:0/3989827786 >> 172.21.15.87:0/3989827786 conn(0x5594c2a82a40 msgr2=0x5594c2a82e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- 172.21.15.87:0/3989827786 shutdown_connections 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- 172.21.15.87:0/3989827786 wait complete. 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- start start 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 Processor -- start 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a8e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5594c2a94600 con 0x5594c2a91810 2024-04-22T13:32:37.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5594c29a6a60 con 0x5594c2a88310 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.316+0000 7fc65fb68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5594c2a94780 con 0x5594c2a88c30 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65fb68740 10 client.0 _ll_register_callbacks cb 0x5594c26ff110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36120/0 (socket says 172.21.15.87:36120) 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 -- 172.21.15.87:0/3647250807 learned_addr learned my addr 172.21.15.87:0/3647250807 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc64ffff640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a8e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65659c640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 -- 172.21.15.87:0/3647250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 msgr2=0x5594c2a8e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a8e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 -- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 msgr2=0x5594c2a911b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a911b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 -- 172.21.15.87:0/3647250807 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594c26ff920 con 0x5594c2a91810 2024-04-22T13:32:37.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65659c640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc656d9d640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc6500109f0 tx=0x7fc650005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887284520ad28d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc63dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650022030 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc650018070 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6400041c0 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc65000fa70 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65fb68740 10 client.15915 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5594c27cbd30 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.317+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6400041c0 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc65002f050 con 0x5594c2a91810 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 10 client.15915 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 10 client.15915 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 20 client.15915 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:37.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2aa18f0 0x5594c2aa3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 4 client.15915 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594c29c0a10 con 0x5594c2aa18f0 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65659c640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2aa18f0 0x5594c2aa3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc63dffb640 10 client.15915 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.318+0000 7fc65659c640 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2aa18f0 0x5594c2aa3d50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef868ae595fb5eab server_cookie=1e801c11df65b424 in_seq=0 out_seq=0 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc63dffb640 1 -- 172.21.15.87:0/3647250807 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594c29c0a10 con 0x5594c2aa18f0 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc63dffb640 10 client.15915 handle_command_reply: tid=0 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 client.15915 shutdown 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 msgr2=0x5594c2a8ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc6500109f0 tx=0x7fc650005bd0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 shutdown_connections 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2aa18f0 0x5594c2aa3d50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a91810 0x5594c2a8ebe0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc63dffb640 0 client.15915 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88c30 0x5594c2a8e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 --2- 172.21.15.87:0/3647250807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a88310 0x5594c2a911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 >> 172.21.15.87:0/3647250807 conn(0x5594c2a82a40 msgr2=0x5594c2a84790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 shutdown_connections 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 1 -- 172.21.15.87:0/3647250807 wait complete. 2024-04-22T13:32:37.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.319+0000 7fc65fb68740 20 client.15915 trim_cache size 0 max 16384 2024-04-22T13:32:37.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.336+0000 7fc65fb68740 1 Processor -- start 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.336+0000 7fc65fb68740 1 -- start start 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a9d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2aa9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581b0220 con 0x5594c2a88c30 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581adaf0 con 0x5594c2a91810 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65fb68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581ab960 con 0x5594c2a88310 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36136/0 (socket says 172.21.15.87:36136) 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 -- 172.21.15.87:0/4005753050 learned_addr learned my addr 172.21.15.87:0/4005753050 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc65659c640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a9d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 -- 172.21.15.87:0/4005753050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 msgr2=0x5594c2a9d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a9d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 -- 172.21.15.87:0/4005753050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 msgr2=0x5594c2aa9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc656d9d640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2aa9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2aa9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 -- 172.21.15.87:0/4005753050 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594c2a8e380 con 0x5594c2a88c30 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc656d9d640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2aa9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc64ffff640 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc640011cb0 tx=0x7fc640009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33db40caaf891286 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc63ffff640 1 -- 172.21.15.87:0/4005753050 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640028070 con 0x5594c2a88c30 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc63ffff640 1 -- 172.21.15.87:0/4005753050 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc640003aa0 con 0x5594c2a88c30 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.337+0000 7fc63ffff640 1 -- 172.21.15.87:0/4005753050 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640012450 con 0x5594c2a88c30 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- 172.21.15.87:0/4005753050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 msgr2=0x5594c2a99db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc640011cb0 tx=0x7fc640009620 comp rx=0 tx=0).stop 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- 172.21.15.87:0/4005753050 shutdown_connections 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2aa9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a99db0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- 172.21.15.87:0/4005753050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a9d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- 172.21.15.87:0/4005753050 >> 172.21.15.87:0/4005753050 conn(0x5594c2a82a40 msgr2=0x5594c2a8ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- 172.21.15.87:0/4005753050 shutdown_connections 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- 172.21.15.87:0/4005753050 wait complete. 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- start start 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 Processor -- start 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a8a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2a8b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581abfb0 con 0x5594c2a88c30 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581ad420 con 0x5594c2a91810 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581ac680 con 0x5594c2a88310 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.338+0000 7fc65fb68740 10 client.0 _ll_register_callbacks cb 0x7fc6500103e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36140/0 (socket says 172.21.15.87:36140) 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 -- 172.21.15.87:0/1481226399 learned_addr learned my addr 172.21.15.87:0/1481226399 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc65659c640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a8a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 -- 172.21.15.87:0/1481226399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 msgr2=0x5594c2a8a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a8a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 -- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 msgr2=0x5594c2a8b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2a8b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 -- 172.21.15.87:0/1481226399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594c2b0bf50 con 0x5594c2a88c30 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc64ffff640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc640026040 tx=0x7fc640009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a97a70daa548c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc63dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640028040 con 0x5594c2a88c30 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc6400035f0 con 0x5594c2a88c30 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc640003aa0 con 0x5594c2a88c30 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc6400037f0 con 0x5594c2a88c30 2024-04-22T13:32:37.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc65fb68740 10 client.15921 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5594c2b0d1c0 con 0x5594c2a88c30 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.339+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc640003aa0 con 0x5594c2a88c30 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc640034030 con 0x5594c2a88c30 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 10 client.15921 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 10 client.15921 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 20 client.15921 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2b0d720 0x5594c2b0fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 4 client.15921 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3ca5eb30", "release": true, "await": true}] 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3ca5eb30", "release": true, "await": true}) -- 0x5594c29faf20 con 0x5594c2b0d720 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65659c640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2b0d720 0x5594c2b0fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc63dffb640 10 client.15921 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.340+0000 7fc65659c640 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2b0d720 0x5594c2b0fb40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15b432bb7c4144f server_cookie=85b993101de80756 in_seq=0 out_seq=0 2024-04-22T13:32:37.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc63dffb640 1 -- 172.21.15.87:0/1481226399 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5594c29faf20 con 0x5594c2b0d720 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc63dffb640 10 client.15921 handle_command_reply: tid=0 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 client.15921 shutdown 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 msgr2=0x5594c2a8adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc640026040 tx=0x7fc640009850 comp rx=0 tx=0).stop 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 shutdown_connections 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5594c2b0d720 0x5594c2b0fb40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5594c2a91810 0x5594c2a8b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc63dffb640 0 client.15921 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5594c2a88c30 0x5594c2a8adc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 --2- 172.21.15.87:0/1481226399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594c2a88310 0x5594c2a8a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 >> 172.21.15.87:0/1481226399 conn(0x5594c2a82a40 msgr2=0x5594c2a83e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 shutdown_connections 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 1 -- 172.21.15.87:0/1481226399 wait complete. 2024-04-22T13:32:37.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.342+0000 7fc65fb68740 20 client.15921 trim_cache size 0 max 16384 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 4, 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "3ca5eb30": { 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 4, 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:32:37.348 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 74.2, 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:37.349 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:37.350 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:32:37.350 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 msgr2=0x7fc6440a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 0x7fc6440a6410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc650005c40 tx=0x7fc6500127c0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 msgr2=0x7fc658198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc648005940 tx=0x7fc6480045e0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 shutdown_connections 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6440a3f50 0x7fc6440a6410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc65819cfe0 0x7fc65819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc658106160 0x7fc658198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 --2- 172.21.15.87:0/994398434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc658101090 0x7fc658198720 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:37.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 >> 172.21.15.87:0/994398434 conn(0x7fc658077f50 msgr2=0x7fc6580ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:37.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 shutdown_connections 2024-04-22T13:32:37.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:32:37.343+0000 7fc65cea0640 1 -- 172.21.15.87:0/994398434 wait complete. 2024-04-22T13:32:37.352 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 3ca5eb30, seconds elapsed: 15.3 2024-04-22T13:32:37.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:37 smithi120 ceph-mon[31101]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-22T13:32:37.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:37 smithi132 ceph-mon[31531]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-22T13:32:38.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:38 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/781557058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:38.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:38 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/781557058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:38.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:38 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/781557058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:32:39.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:39 smithi120 ceph-mon[31101]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T13:32:39.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:39 smithi132 ceph-mon[31531]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T13:32:39.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:39 smithi087 ceph-mon[27999]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 -- 172.21.15.87:0/2801588175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001021d0 msgr2=0x7fe2001025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2801588175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001021d0 0x7fe2001025b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009ca0 tx=0x7fe1f002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 -- 172.21.15.87:0/2801588175 shutdown_connections 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2801588175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2001072a0 0x7fe20010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2801588175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe200102b80 0x7fe200106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2801588175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001021d0 0x7fe2001025b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 -- 172.21.15.87:0/2801588175 >> 172.21.15.87:0/2801588175 conn(0x7fe2000fda10 msgr2=0x7fe2000ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:39.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 -- 172.21.15.87:0/2801588175 shutdown_connections 2024-04-22T13:32:39.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.990+0000 7fe205a5c640 1 -- 172.21.15.87:0/2801588175 wait complete. 2024-04-22T13:32:39.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 Processor -- start 2024-04-22T13:32:39.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 -- start start 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 0x7fe200198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20019cfb0 0x7fe20019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe200112200 con 0x7fe2001072a0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe200112080 con 0x7fe20019cfb0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe205a5c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe200112380 con 0x7fe2001021d0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40602/0 (socket says 172.21.15.87:40602) 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/2351576629 learned_addr learned my addr 172.21.15.87:0/2351576629 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1feffd640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 0x7fe200198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20019cfb0 0x7fe20019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/2351576629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 msgr2=0x7fe200198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 0x7fe200198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/2351576629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20019cfb0 msgr2=0x7fe20019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20019cfb0 0x7fe20019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/2351576629 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20019daa0 con 0x7fe2001072a0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.991+0000 7fe1feffd640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 0x7fe200198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400e9f0 tx=0x7fe1f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe204a5a640 1 -- 172.21.15.87:0/2351576629 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f400ccc0 con 0x7fe2001072a0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20019dd90 con 0x7fe2001072a0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe204a5a640 1 -- 172.21.15.87:0/2351576629 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f40054d0 con 0x7fe2001072a0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe204a5a640 1 -- 172.21.15.87:0/2351576629 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4010430 con 0x7fe2001072a0 2024-04-22T13:32:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.992+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2001a5530 con 0x7fe2001072a0 2024-04-22T13:32:39.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe204a5a640 1 -- 172.21.15.87:0/2351576629 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe1f40046c0 con 0x7fe2001072a0 2024-04-22T13:32:39.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 Processor -- start 2024-04-22T13:32:39.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 -- start start 2024-04-22T13:32:39.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b2ce0 0x562a444b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 0x562a444b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562a444128d0 con 0x562a444bb4f0 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a443afe80 con 0x562a444b2ce0 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe208653740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562a4440e890 con 0x562a444b1f70 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe1fe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe1fe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40612/0 (socket says 172.21.15.87:40612) 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/1089094838 learned_addr learned my addr 172.21.15.87:0/1089094838 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:39.998 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe1feffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 0x562a444b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b2ce0 0x562a444b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.995+0000 7fe204a5a640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 0x7fe1e80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 0x7fe1e80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/1089094838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 msgr2=0x562a444b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 0x562a444b23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/1089094838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b2ce0 msgr2=0x562a444b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b2ce0 0x562a444b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe204a5a640 1 -- 172.21.15.87:0/2351576629 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe1f40d0f50 con 0x7fe2001072a0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 -- 172.21.15.87:0/1089094838 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a443d1320 con 0x562a444bb4f0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1feffd640 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 0x562a444b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe1f409ca20 tx=0x7fe1f400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8f870a1d97d8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1e5ffb640 1 -- 172.21.15.87:0/1089094838 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40cee60 con 0x562a444bb4f0 2024-04-22T13:32:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 0x7fe1e80a6430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800bfd0 tx=0x7fe1f8004730 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1e5ffb640 1 -- 172.21.15.87:0/1089094838 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f40a5990 con 0x562a444bb4f0 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.996+0000 7fe1e5ffb640 1 -- 172.21.15.87:0/1089094838 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40a10d0 con 0x562a444bb4f0 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- 172.21.15.87:0/1089094838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 msgr2=0x562a444bd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe1f409ca20 tx=0x7fe1f400ce60 comp rx=0 tx=0).stop 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- 172.21.15.87:0/1089094838 shutdown_connections 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444bd8e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b1f70 0x562a444b23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 --2- 172.21.15.87:0/1089094838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b2ce0 0x562a444b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- 172.21.15.87:0/1089094838 >> 172.21.15.87:0/1089094838 conn(0x562a44474c90 msgr2=0x562a444acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- 172.21.15.87:0/1089094838 shutdown_connections 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- 172.21.15.87:0/1089094838 wait complete. 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 -- start start 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.997+0000 7fe208653740 1 Processor -- start 2024-04-22T13:32:40.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 0x562a444bae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562a444be160 con 0x562a444bb4f0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a4439aeb0 con 0x562a444b1f70 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562a444be2e0 con 0x562a444b2ce0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe208653740 10 client.0 _ll_register_callbacks cb 0x562a443c7b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40618/0 (socket says 172.21.15.87:40618) 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 -- 172.21.15.87:0/119356077 learned_addr learned my addr 172.21.15.87:0/119356077 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1feffd640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 0x562a444bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 -- 172.21.15.87:0/119356077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 msgr2=0x562a444b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 -- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 msgr2=0x562a444bae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 0x562a444bae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 -- 172.21.15.87:0/119356077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a44125210 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1feffd640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 0x562a444bae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80108a0 tx=0x7fe1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabf0c13fe6b2aa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.998+0000 7fe1deffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8019070 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe1f8011b60 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a44426880 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe1f8011da0 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 10 client.15933 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562a441f1620 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562a44426880 con 0x562a444bb4f0 2024-04-22T13:32:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe1f8030020 con 0x562a444bb4f0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 10 client.15933 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 20 client.15933 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a444cb4a0 0x562a444cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 4 client.15933 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:39.999+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a443b23d0 con 0x562a444cb4a0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.000+0000 7fe1feffd640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a444cb4a0 0x562a444cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.000+0000 7fe1deffd640 10 client.15933 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.000+0000 7fe1feffd640 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a444cb4a0 0x562a444cd890 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26704dd6944c2ff server_cookie=e11a57b40c822f11 in_seq=0 out_seq=0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe1deffd640 1 -- 172.21.15.87:0/119356077 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a443b23d0 con 0x562a444cb4a0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe1deffd640 10 client.15933 handle_command_reply: tid=0 2024-04-22T13:32:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 client.15933 shutdown 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 msgr2=0x562a444b87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80108a0 tx=0x7fe1f80046c0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 shutdown_connections 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a444cb4a0 0x562a444cd890 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444bb4f0 0x562a444b87c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe1deffd640 0 client.15933 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 --2- 172.21.15.87:0/119356077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444b1f70 0x562a444bae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 >> 172.21.15.87:0/119356077 conn(0x562a44474c90 msgr2=0x562a444ae9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 shutdown_connections 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 1 -- 172.21.15.87:0/119356077 wait complete. 2024-04-22T13:32:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.001+0000 7fe208653740 20 client.15933 trim_cache size 0 max 16384 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 Processor -- start 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 -- start start 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001ae0b0 con 0x562a444b1f70 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001ab980 con 0x562a444bb4f0 2024-04-22T13:32:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe208653740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001a9810 con 0x562a444b2ce0 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40634/0 (socket says 172.21.15.87:40634) 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 -- 172.21.15.87:0/3186625790 learned_addr learned my addr 172.21.15.87:0/3186625790 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 -- 172.21.15.87:0/3186625790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 msgr2=0x562a444ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444ac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 -- 172.21.15.87:0/3186625790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 msgr2=0x562a444d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1feffd640 1 -- 172.21.15.87:0/3186625790 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a444b1240 con 0x562a444b1f70 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.016+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe1feffd640 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0004af0 tx=0x7fe1f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2581606a0be7581 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe1e57fa640 1 -- 172.21.15.87:0/3186625790 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f004a070 con 0x562a444b1f70 2024-04-22T13:32:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe1e57fa640 1 -- 172.21.15.87:0/3186625790 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f0032970 con 0x562a444b1f70 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe1e57fa640 1 -- 172.21.15.87:0/3186625790 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0039040 con 0x562a444b1f70 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 -- 172.21.15.87:0/3186625790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 msgr2=0x562a444c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0004af0 tx=0x7fe1f0004a40 comp rx=0 tx=0).stop 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 -- 172.21.15.87:0/3186625790 shutdown_connections 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a444ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 --2- 172.21.15.87:0/3186625790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a444c7060 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 -- 172.21.15.87:0/3186625790 >> 172.21.15.87:0/3186625790 conn(0x562a44474c90 msgr2=0x562a444b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 -- 172.21.15.87:0/3186625790 shutdown_connections 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.017+0000 7fe208653740 1 -- 172.21.15.87:0/3186625790 wait complete. 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 -- start start 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 Processor -- start 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a4452a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444be750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001a9e60 con 0x562a444b1f70 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001ab2b0 con 0x562a444bb4f0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001aa510 con 0x562a444b2ce0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe208653740 10 client.0 _ll_register_callbacks cb 0x7fe1f800bd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe1feffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe1feffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40646/0 (socket says 172.21.15.87:40646) 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe1feffd640 1 -- 172.21.15.87:0/2554384098 learned_addr learned my addr 172.21.15.87:0/2554384098 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.018+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a4452a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444be750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 -- 172.21.15.87:0/2554384098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 msgr2=0x562a4452a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a4452a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 -- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 msgr2=0x562a444be750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444be750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 -- 172.21.15.87:0/2554384098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a444bee40 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1ff7fe640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444be750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1feffd640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00099f0 tx=0x7fe1f00324c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e1faf4f4c1aad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0039040 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe1f0003970 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1f0032970 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe1f005e050 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 10 client.15939 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562a44531410 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f0032970 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe1f003f040 con 0x562a444b1f70 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 10 client.15939 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 20 client.15939 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a44531be0 0x562a44534000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 4 client.15939 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.019+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a444202f0 con 0x562a44531be0 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.020+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a44531be0 0x562a44534000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.020+0000 7fe1fe7fc640 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a44531be0 0x562a44534000 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86bf5ae55ed5caf4 server_cookie=f1f643a4f5d0c706 in_seq=0 out_seq=0 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.020+0000 7fe1deffd640 10 client.15939 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe1deffd640 1 -- 172.21.15.87:0/2554384098 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a444202f0 con 0x562a44531be0 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe1deffd640 10 client.15939 handle_command_reply: tid=0 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 client.15939 shutdown 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 msgr2=0x562a44529cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00099f0 tx=0x7fe1f00324c0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 shutdown_connections 2024-04-22T13:32:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562a44531be0 0x562a44534000 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a444bb4f0 0x562a444be750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a444b2ce0 0x562a4452a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 --2- 172.21.15.87:0/2554384098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562a444b1f70 0x562a44529cf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe1deffd640 0 client.15939 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 >> 172.21.15.87:0/2554384098 conn(0x562a44474c90 msgr2=0x562a444b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 shutdown_connections 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 1 -- 172.21.15.87:0/2554384098 wait complete. 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.024+0000 7fe208653740 20 client.15939 trim_cache size 0 max 16384 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:32:40.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:32:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:32:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:32:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 msgr2=0x7fe1e80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 0x7fe1e80a6430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800bfd0 tx=0x7fe1f8004730 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 msgr2=0x7fe200198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400e9f0 tx=0x7fe1f400eeb0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 shutdown_connections 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1e80a3f70 0x7fe1e80a6430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe20019cfb0 0x7fe20019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2001072a0 0x7fe200198c80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 --2- 172.21.15.87:0/2351576629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2001021d0 0x7fe200198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 >> 172.21.15.87:0/2351576629 conn(0x7fe2000fda10 msgr2=0x7fe2000fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 shutdown_connections 2024-04-22T13:32:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:40.025+0000 7fe205a5c640 1 -- 172.21.15.87:0/2351576629 wait complete. 2024-04-22T13:32:41.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:41 smithi120 ceph-mon[31101]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T13:32:41.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:41 smithi132 ceph-mon[31531]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T13:32:41.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:41 smithi087 ceph-mon[27999]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T13:32:43.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:43 smithi120 ceph-mon[31101]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-22T13:32:43.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:43 smithi132 ceph-mon[31531]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-22T13:32:43.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:43 smithi087 ceph-mon[27999]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-22T13:32:45.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:45 smithi120 ceph-mon[31101]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-22T13:32:45.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:45 smithi132 ceph-mon[31531]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-22T13:32:45.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:45 smithi087 ceph-mon[27999]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-22T13:32:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:47 smithi120 ceph-mon[31101]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:47.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:47 smithi132 ceph-mon[31531]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:47.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:47 smithi087 ceph-mon[27999]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:49.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:49 smithi120 ceph-mon[31101]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:49.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:49 smithi132 ceph-mon[31531]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:49.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:49 smithi087 ceph-mon[27999]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:51.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:51 smithi120 ceph-mon[31101]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:51.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:51 smithi132 ceph-mon[31531]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:51.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:51 smithi087 ceph-mon[27999]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 -- 172.21.15.87:0/2751950937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb4101aa0 msgr2=0x7f5fb4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 --2- 172.21.15.87:0/2751950937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb4101aa0 0x7f5fb4105a90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009900 tx=0x7f5fa802bc30 comp rx=0 tx=0).stop 2024-04-22T13:32:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 -- 172.21.15.87:0/2751950937 shutdown_connections 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 --2- 172.21.15.87:0/2751950937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 0x7f5fb410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 --2- 172.21.15.87:0/2751950937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb4101aa0 0x7f5fb4105a90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 --2- 172.21.15.87:0/2751950937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 0x7f5fb41014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 -- 172.21.15.87:0/2751950937 >> 172.21.15.87:0/2751950937 conn(0x7f5fb4077fd0 msgr2=0x7f5fb40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 -- 172.21.15.87:0/2751950937 shutdown_connections 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 -- 172.21.15.87:0/2751950937 wait complete. 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.115+0000 7f5fba132640 1 Processor -- start 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 -- start start 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 0x7f5fb41a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 0x7f5fb41a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41121f0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb4112070 con 0x7f5fb4106170 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fba132640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb4112370 con 0x7f5fb41010f0 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46502/0 (socket says 172.21.15.87:46502) 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 -- 172.21.15.87:0/3947792376 learned_addr learned my addr 172.21.15.87:0/3947792376 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 0x7f5fb41a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 0x7f5fb41a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 -- 172.21.15.87:0/3947792376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 msgr2=0x7f5fb41a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 0x7f5fb41a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 -- 172.21.15.87:0/3947792376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 msgr2=0x7f5fb41a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 0x7f5fb41a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 -- 172.21.15.87:0/3947792376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb419b2f0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.116+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00eaa0 tx=0x7f5fac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.117+0000 7f5fb0ff9640 1 -- 172.21.15.87:0/3947792376 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac00cdb0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.117+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb41a73d0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.117+0000 7f5fb0ff9640 1 -- 172.21.15.87:0/3947792376 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fac005020 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.117+0000 7f5fb0ff9640 1 -- 172.21.15.87:0/3947792376 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0116d0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.117+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb41a7780 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.119+0000 7f5fb0ff9640 1 -- 172.21.15.87:0/3947792376 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f5fac0051d0 con 0x7f5fb419a6f0 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 Processor -- start 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 -- start start 2024-04-22T13:32:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb0ff9640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 0x7f5f9c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512967fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 0x7f5f9c0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e5129734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e5128cc8d0 con 0x55e5129688d0 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e512869e80 con 0x55e5129710e0 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fbce04740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e5128c8890 con 0x55e512967b60 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb0ff9640 1 -- 172.21.15.87:0/3947792376 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5fac0d0b70 con 0x7f5fb419a6f0 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46516/0 (socket says 172.21.15.87:46516) 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb37fe640 1 -- 172.21.15.87:0/3354308336 learned_addr learned my addr 172.21.15.87:0/3354308336 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512967fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.120+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e5129734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 -- 172.21.15.87:0/3354308336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 msgr2=0x55e512967fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512967fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 0x7f5f9c0a64b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009230 tx=0x7f5fa8004310 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 -- 172.21.15.87:0/3354308336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 msgr2=0x55e5129734d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e5129734d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 -- 172.21.15.87:0/3354308336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51288b320 con 0x55e5129688d0 2024-04-22T13:32:53.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.121+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400bcf0 tx=0x7f5fa400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e1d0cb6d20f0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.122+0000 7f5f99ffb640 1 -- 172.21.15.87:0/3354308336 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4015070 con 0x55e5129688d0 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.122+0000 7f5f99ffb640 1 -- 172.21.15.87:0/3354308336 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa40118f0 con 0x55e5129688d0 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.122+0000 7f5f99ffb640 1 -- 172.21.15.87:0/3354308336 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40107d0 con 0x55e5129688d0 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- 172.21.15.87:0/3354308336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 msgr2=0x55e512967620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400bcf0 tx=0x7f5fa400f730 comp rx=0 tx=0).stop 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- 172.21.15.87:0/3354308336 shutdown_connections 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e5129734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512967fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 --2- 172.21.15.87:0/3354308336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512967620 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- 172.21.15.87:0/3354308336 >> 172.21.15.87:0/3354308336 conn(0x55e51292ec90 msgr2=0x55e512962850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- 172.21.15.87:0/3354308336 shutdown_connections 2024-04-22T13:32:53.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- 172.21.15.87:0/3354308336 wait complete. 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.123+0000 7f5fbce04740 1 -- start start 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 Processor -- start 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129688d0 0x55e51296dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5129710e0 0x55e51296e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e512973e10 con 0x55e512967b60 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e512854eb0 con 0x55e5129688d0 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e512973f90 con 0x55e5129710e0 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fbce04740 10 client.0 _ll_register_callbacks cb 0x55e512881b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46524/0 (socket says 172.21.15.87:46524) 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 -- 172.21.15.87:0/724115147 learned_addr learned my addr 172.21.15.87:0/724115147 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5129710e0 0x55e51296e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129688d0 0x55e51296dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 -- 172.21.15.87:0/724115147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5129710e0 msgr2=0x55e51296e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5129710e0 0x55e51296e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 -- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129688d0 msgr2=0x55e51296dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129688d0 0x55e51296dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.124+0000 7f5fb37fe640 1 -- 172.21.15.87:0/724115147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5125df210 con 0x55e512967b60 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400fc30 tx=0x7f5fa400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f79411f85917fb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4015040 con 0x55e512967b60 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5fa4011ca0 con 0x55e512967b60 2024-04-22T13:32:53.133 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa40118f0 con 0x55e512967b60 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5fa4005040 con 0x55e512967b60 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 10 client.15951 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e5126ab620 con 0x55e512967b60 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa40118f0 con 0x55e512967b60 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5fa4022070 con 0x55e512967b60 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 10 client.15951 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 10 client.15951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 20 client.15951 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e512980ff0 0x55e5129833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 4 client.15951 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.125+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5128e0880 con 0x55e512980ff0 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.126+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e512980ff0 0x55e5129833e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.126+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e512980ff0 0x55e5129833e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4839bb046b64ad7c server_cookie=d4eb5112f6859362 in_seq=0 out_seq=0 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.126+0000 7f5f92ffd640 10 client.15951 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5f92ffd640 1 -- 172.21.15.87:0/724115147 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5128e0880 con 0x55e512980ff0 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5f92ffd640 10 client.15951 handle_command_reply: tid=0 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 client.15951 shutdown 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 msgr2=0x55e512970a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400fc30 tx=0x7f5fa400bf50 comp rx=0 tx=0).stop 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 shutdown_connections 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e512980ff0 0x55e5129833e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5129710e0 0x55e51296e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129688d0 0x55e51296dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 --2- 172.21.15.87:0/724115147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e512967b60 0x55e512970a50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 >> 172.21.15.87:0/724115147 conn(0x55e51292ec90 msgr2=0x55e5129645a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5f92ffd640 0 client.15951 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 shutdown_connections 2024-04-22T13:32:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 1 -- 172.21.15.87:0/724115147 wait complete. 2024-04-22T13:32:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.127+0000 7f5fbce04740 20 client.15951 trim_cache size 0 max 16384 2024-04-22T13:32:53.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.143+0000 7f5fbce04740 1 Processor -- start 2024-04-22T13:32:53.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.143+0000 7f5fbce04740 1 -- start start 2024-04-22T13:32:53.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e51297cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e512970a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41b2440 con 0x55e5129688d0 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41afd10 con 0x55e5129710e0 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fbce04740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41adb60 con 0x55e512967b60 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46532/0 (socket says 172.21.15.87:46532) 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/809095259 learned_addr learned my addr 172.21.15.87:0/809095259 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e512970a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e51297cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/809095259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 msgr2=0x55e51297cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e51297cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/809095259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 msgr2=0x55e512970a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e512970a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/809095259 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51295df30 con 0x55e5129688d0 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009850 tx=0x7f5fa8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be55251f969eebad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5f997fa640 1 -- 172.21.15.87:0/809095259 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8049070 con 0x55e5129688d0 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5f997fa640 1 -- 172.21.15.87:0/809095259 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa8058850 con 0x55e5129688d0 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.144+0000 7f5f997fa640 1 -- 172.21.15.87:0/809095259 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8004ab0 con 0x55e5129688d0 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 -- 172.21.15.87:0/809095259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 msgr2=0x55e512970510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009850 tx=0x7f5fa8045000 comp rx=0 tx=0).stop 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 -- 172.21.15.87:0/809095259 shutdown_connections 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e512970a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512970510 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 --2- 172.21.15.87:0/809095259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e51297cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 -- 172.21.15.87:0/809095259 >> 172.21.15.87:0/809095259 conn(0x55e51292ec90 msgr2=0x55e512962380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 -- 172.21.15.87:0/809095259 shutdown_connections 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.145+0000 7f5fbce04740 1 -- 172.21.15.87:0/809095259 wait complete. 2024-04-22T13:32:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 -- start start 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 Processor -- start 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512979220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e51296a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41ae1d0 con 0x55e5129688d0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41af640 con 0x55e5129710e0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb41ae8a0 con 0x55e512967b60 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fbce04740 10 client.0 _ll_register_callbacks cb 0x7f5fa80045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46536/0 (socket says 172.21.15.87:46536) 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/2855158806 learned_addr learned my addr 172.21.15.87:0/2855158806 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb3fff640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e51296a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512979220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/2855158806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 msgr2=0x55e512979220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512979220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 msgr2=0x55e51296a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e51296a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.146+0000 7f5fb2ffd640 1 -- 172.21.15.87:0/2855158806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51296a9d0 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fb2ffd640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80043f0 tx=0x7f5fa8004110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6caa7d8d27892977 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8063030 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5fa80583a0 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa8058850 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5fa8070050 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 10 client.15957 fetch_fsmap learned FSMap version 21 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e5129e6c00 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa8058850 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5fa8040040 con 0x55e5129688d0 2024-04-22T13:32:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 10 client.15957 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 20 client.15957 populate_metadata read hostname 'smithi087' 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e5129e72a0 0x55e5129e96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 4 client.15957 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.147+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e51286c3d0 con 0x55e5129e72a0 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.148+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e5129e72a0 0x55e5129e96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.148+0000 7f5f92ffd640 10 client.15957 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.148+0000 7f5fb37fe640 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e5129e72a0 0x55e5129e96c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=473fbaabb4f307c1 server_cookie=eefde2170018e6ee in_seq=0 out_seq=0 2024-04-22T13:32:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5f92ffd640 1 -- 172.21.15.87:0/2855158806 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e51286c3d0 con 0x55e5129e72a0 2024-04-22T13:32:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5f92ffd640 10 client.15957 handle_command_reply: tid=0 2024-04-22T13:32:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 client.15957 shutdown 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 msgr2=0x55e512979760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80043f0 tx=0x7f5fa8004110 comp rx=0 tx=0).stop 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 shutdown_connections 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e5129e72a0 0x55e5129e96c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5129710e0 0x55e51296a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5129688d0 0x55e512979760 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5f92ffd640 0 client.15957 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 --2- 172.21.15.87:0/2855158806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e512967b60 0x55e512979220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 >> 172.21.15.87:0/2855158806 conn(0x55e51292ec90 msgr2=0x55e512968ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 shutdown_connections 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 1 -- 172.21.15.87:0/2855158806 wait complete. 2024-04-22T13:32:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fbce04740 20 client.15957 trim_cache size 0 max 16384 2024-04-22T13:32:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:32:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:32:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:32:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 msgr2=0x7f5f9c0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.152+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 0x7f5f9c0a64b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009230 tx=0x7f5fa8004310 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 msgr2=0x7f5fb419ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00eaa0 tx=0x7f5fac00ef60 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 shutdown_connections 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5f9c0a3ff0 0x7f5f9c0a64b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fb419a6f0 0x7f5fb419ab70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb4106170 0x7f5fb41a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 --2- 172.21.15.87:0/3947792376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fb41010f0 0x7f5fb41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 >> 172.21.15.87:0/3947792376 conn(0x7f5fb4077fd0 msgr2=0x7f5fb410c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 shutdown_connections 2024-04-22T13:32:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:32:53.153+0000 7f5fba132640 1 -- 172.21.15.87:0/3947792376 wait complete. 2024-04-22T13:32:53.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:53 smithi120 ceph-mon[31101]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:53.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:53 smithi132 ceph-mon[31531]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:53.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:53 smithi087 ceph-mon[27999]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:32:55.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:55 smithi120 ceph-mon[31101]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:55.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:55 smithi132 ceph-mon[31531]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:55.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:55 smithi087 ceph-mon[27999]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:57.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:57 smithi120 ceph-mon[31101]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:32:57.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:57 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:32:57.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:57 smithi132 ceph-mon[31531]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:32:57.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:57 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:32:57.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:57 smithi087 ceph-mon[27999]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:32:57.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:57 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:32:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:32:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:32:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:32:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:32:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:58.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:32:58.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:32:58.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:32:59.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:32:59 smithi120 ceph-mon[31101]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:59.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:32:59 smithi132 ceph-mon[31531]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:32:59.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:32:59 smithi087 ceph-mon[27999]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:33:01.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:33:01.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:33:01.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:01 smithi087 ceph-mon[27999]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:01.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:01 smithi120 ceph-mon[31101]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:01.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 -- 172.21.15.87:0/3162883264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 msgr2=0x7f7acc10bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3162883264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc10bfc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009900 tx=0x7f7abc02bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 -- 172.21.15.87:0/3162883264 shutdown_connections 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3162883264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc10c500 0x7f7acc10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3162883264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc10bfc0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3162883264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc1031c0 0x7f7acc1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 -- 172.21.15.87:0/3162883264 >> 172.21.15.87:0/3162883264 conn(0x7f7acc078000 msgr2=0x7f7acc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 -- 172.21.15.87:0/3162883264 shutdown_connections 2024-04-22T13:33:01.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.677+0000 7f7ad3396640 1 -- 172.21.15.87:0/3162883264 wait complete. 2024-04-22T13:33:01.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 Processor -- start 2024-04-22T13:33:01.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 -- start start 2024-04-22T13:33:01.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 0x7f7acc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc10c500 0x7f7acc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc112170 con 0x7f7acc103ae0 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc111ff0 con 0x7f7acc1031c0 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.678+0000 7f7ad3396640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc1122f0 con 0x7f7acc10c500 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38238/0 (socket says 172.21.15.87:38238) 2024-04-22T13:33:01.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 -- 172.21.15.87:0/3110286031 learned_addr learned my addr 172.21.15.87:0/3110286031 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad1933640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc10c500 0x7f7acc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad1132640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 0x7f7acc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 -- 172.21.15.87:0/3110286031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc10c500 msgr2=0x7f7acc19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc10c500 0x7f7acc19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 -- 172.21.15.87:0/3110286031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 msgr2=0x7f7acc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 0x7f7acc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 -- 172.21.15.87:0/3110286031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc19d6c0 con 0x7f7acc103ae0 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad1132640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 0x7f7acc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:01.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005940 tx=0x7f7abc0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ac27fc640 1 -- 172.21.15.87:0/3110286031 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc043020 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc19d950 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ac27fc640 1 -- 172.21.15.87:0/3110286031 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7abc02dc30 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc1a5240 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.679+0000 7f7ac27fc640 1 -- 172.21.15.87:0/3110286031 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc003a10 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.682+0000 7f7ac27fc640 1 -- 172.21.15.87:0/3110286031 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f7abc046020 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.682+0000 7f7ac27fc640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 0x7f7ab80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ac27fc640 1 -- 172.21.15.87:0/3110286031 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7abc0f2940 con 0x7f7acc103ae0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ad1132640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 0x7f7ab80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ad1132640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 0x7f7ab80a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80058c0 tx=0x7f7ac8005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ad5f86740 1 Processor -- start 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ad5f86740 1 -- start start 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.683+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696705540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x559696707e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5596966611e0 con 0x5596966fc580 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5596965fe390 con 0x559696705a80 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55969665d1a0 con 0x5596966fcea0 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38240/0 (socket says 172.21.15.87:38240) 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 -- 172.21.15.87:0/2668754028 learned_addr learned my addr 172.21.15.87:0/2668754028 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696705540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1933640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x559696707e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 -- 172.21.15.87:0/2668754028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 msgr2=0x559696705540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696705540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 -- 172.21.15.87:0/2668754028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 msgr2=0x559696707e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x559696707e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 -- 172.21.15.87:0/2668754028 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55969643d7d0 con 0x5596966fc580 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1933640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x559696707e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8010010 tx=0x7f7ac8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cccc7ca7d6fb569 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ab77fe640 1 -- 172.21.15.87:0/2668754028 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8022070 con 0x5596966fc580 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ab77fe640 1 -- 172.21.15.87:0/2668754028 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac801eb00 con 0x5596966fc580 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.684+0000 7f7ab77fe640 1 -- 172.21.15.87:0/2668754028 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8010aa0 con 0x5596966fc580 2024-04-22T13:33:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2668754028 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 msgr2=0x5596966fc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8010010 tx=0x7f7ac8012000 comp rx=0 tx=0).stop 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2668754028 shutdown_connections 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x559696707e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696705540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2668754028 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x5596966fc960 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.685+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2668754028 >> 172.21.15.87:0/2668754028 conn(0x5596966f6cb0 msgr2=0x5596966f7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.686+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2668754028 shutdown_connections 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.686+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2668754028 wait complete. 2024-04-22T13:33:01.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.686+0000 7f7ad5f86740 1 -- start start 2024-04-22T13:33:01.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.686+0000 7f7ad5f86740 1 Processor -- start 2024-04-22T13:33:01.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.686+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696702f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x5596967034d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559696708870 con 0x5596966fc580 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38256/0 (socket says 172.21.15.87:38256) 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 -- 172.21.15.87:0/2451116694 learned_addr learned my addr 172.21.15.87:0/2451116694 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696702f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55969661a2f0 con 0x559696705a80 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5596967089f0 con 0x5596966fcea0 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1933640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x5596967034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad5f86740 10 client.0 _ll_register_callbacks cb 0x559696627580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:01.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 -- 172.21.15.87:0/2451116694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 msgr2=0x559696702f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696702f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 -- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 msgr2=0x5596967034d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x5596967034d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.687+0000 7f7ad1132640 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596963733f0 con 0x5596966fc580 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7ac801eec0 tx=0x7f7ac801eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbc2551337e4fec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8022070 con 0x5596966fc580 2024-04-22T13:33:01.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7ac801e660 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ac801eb00 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7ac801e860 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ad5f86740 10 client.15969 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5596963330f0 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.688+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac801eb00 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7ac8010b80 con 0x5596966fc580 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 10 client.15969 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 20 client.15969 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:01.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559696715900 0x559696717cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 4 client.15969 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55969666ec00 con 0x559696715900 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559696715900 0x559696717cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559696715900 0x559696717cf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eefc2c029f6dc77f server_cookie=f014ab6b79b557f9 in_seq=0 out_seq=0 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.689+0000 7f7ab4ff9640 10 client.15969 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2451116694 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55969666ec00 con 0x559696715900 2024-04-22T13:33:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ab4ff9640 10 client.15969 handle_command_reply: tid=0 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 client.15969 shutdown 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 msgr2=0x559696702a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7ac801eec0 tx=0x7f7ac801eef0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 shutdown_connections 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559696715900 0x559696717cf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ab4ff9640 0 client.15969 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x5596967034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fcea0 0x559696702f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2451116694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fc580 0x559696702a50 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.690+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 >> 172.21.15.87:0/2451116694 conn(0x5596966f6cb0 msgr2=0x5596966f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.691+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 shutdown_connections 2024-04-22T13:33:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.691+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2451116694 wait complete. 2024-04-22T13:33:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.691+0000 7f7ad5f86740 20 client.15969 trim_cache size 0 max 16384 2024-04-22T13:33:01.719 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:01 smithi132 ceph-mon[31531]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:01.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 Processor -- start 2024-04-22T13:33:01.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 -- start start 2024-04-22T13:33:01.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fc580 0x559696701990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x55969671d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab0010e90 con 0x5596966fcea0 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab000e760 con 0x559696705a80 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab000c5b0 con 0x5596966fc580 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38272/0 (socket says 172.21.15.87:38272) 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 -- 172.21.15.87:0/3713042515 learned_addr learned my addr 172.21.15.87:0/3713042515 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad1933640 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x55969671d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad1132640 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fc580 0x559696701990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 -- 172.21.15.87:0/3713042515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fc580 msgr2=0x559696701990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fc580 0x559696701990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 -- 172.21.15.87:0/3713042515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 msgr2=0x55969671d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x55969671d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.716+0000 7f7ad0931640 1 -- 172.21.15.87:0/3713042515 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596966f6ae0 con 0x5596966fcea0 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.717+0000 7f7ad0931640 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000c00 tx=0x7f7abc0babf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf052c287d1ae20c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.717+0000 7f7ab6ffd640 1 -- 172.21.15.87:0/3713042515 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0c7490 con 0x5596966fcea0 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.717+0000 7f7ab6ffd640 1 -- 172.21.15.87:0/3713042515 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7abc0bc8c0 con 0x5596966fcea0 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.717+0000 7f7ab6ffd640 1 -- 172.21.15.87:0/3713042515 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc003bb0 con 0x5596966fcea0 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- 172.21.15.87:0/3713042515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 msgr2=0x559696711790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000c00 tx=0x7f7abc0babf0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- 172.21.15.87:0/3713042515 shutdown_connections 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559696705a80 0x55969671d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x559696711790 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/3713042515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5596966fc580 0x559696701990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- 172.21.15.87:0/3713042515 >> 172.21.15.87:0/3713042515 conn(0x5596966f6cb0 msgr2=0x5596966f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- 172.21.15.87:0/3713042515 shutdown_connections 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- 172.21.15.87:0/3713042515 wait complete. 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- start start 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 Processor -- start 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596966fc580 0x55969670d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 0x559696709de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab000cc20 con 0x5596966fcea0 2024-04-22T13:33:01.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab000e090 con 0x5596966fc580 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.718+0000 7f7ad5f86740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab000d2f0 con 0x559696705a80 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad5f86740 10 client.0 _ll_register_callbacks cb 0x7f7abc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38280/0 (socket says 172.21.15.87:38280) 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 -- 172.21.15.87:0/2103275433 learned_addr learned my addr 172.21.15.87:0/2103275433 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596966fc580 0x55969670d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad1933640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 0x559696709de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 -- 172.21.15.87:0/2103275433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 msgr2=0x559696709de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 0x559696709de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 -- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596966fc580 msgr2=0x55969670d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596966fc580 0x55969670d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 -- 172.21.15.87:0/2103275433 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596966f64f0 con 0x5596966fcea0 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad1933640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 0x559696709de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:01.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad0931640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0f00c0 tx=0x7f7abc0c7490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140e7a4bb4c49432 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0db070 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7abc0c7000 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.719+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7abc0bc8c0 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7abc0c7220 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 10 client.15975 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55969677c690 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7abc0bc8c0 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7abc0f01d0 con 0x5596966fcea0 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 10 client.15975 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 20 client.15975 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55969677cdd0 0x55969677f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 4 client.15975 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:33:01.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559696668a70 con 0x55969677cdd0 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.720+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55969677cdd0 0x55969677f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ab4ff9640 10 client.15975 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad1132640 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55969677cdd0 0x55969677f1f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c4667bb5398a73 server_cookie=c7e44118954db81c in_seq=0 out_seq=0 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ab4ff9640 1 -- 172.21.15.87:0/2103275433 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559696668a70 con 0x55969677cdd0 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ab4ff9640 10 client.15975 handle_command_reply: tid=0 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 client.15975 shutdown 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 msgr2=0x55969670de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0f00c0 tx=0x7f7abc0c7490 comp rx=0 tx=0).stop 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 shutdown_connections 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55969677cdd0 0x55969677f1f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559696705a80 0x559696709de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5596966fcea0 0x55969670de80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 --2- 172.21.15.87:0/2103275433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596966fc580 0x55969670d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 >> 172.21.15.87:0/2103275433 conn(0x5596966f6cb0 msgr2=0x5596966f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ab4ff9640 0 client.15975 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 shutdown_connections 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 1 -- 172.21.15.87:0/2103275433 wait complete. 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.721+0000 7f7ad5f86740 20 client.15975 trim_cache size 0 max 16384 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:33:01.726 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 msgr2=0x7f7ab80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 0x7f7ab80a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80058c0 tx=0x7f7ac8005850 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 msgr2=0x7f7acc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005940 tx=0x7f7abc0045e0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 shutdown_connections 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7ab80a3ed0 0x7f7ab80a6390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7acc10c500 0x7f7acc19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc103ae0 0x7f7acc198c10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 --2- 172.21.15.87:0/3110286031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7acc1031c0 0x7f7acc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 >> 172.21.15.87:0/3110286031 conn(0x7f7acc078000 msgr2=0x7f7acc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 shutdown_connections 2024-04-22T13:33:01.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.722+0000 7f7ad3396640 1 -- 172.21.15.87:0/3110286031 wait complete. 2024-04-22T13:33:01.731 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:33:01.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 -- 172.21.15.87:0/2497114266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 msgr2=0x7fd20c069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 --2- 172.21.15.87:0/2497114266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c069560 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009900 tx=0x7fd1fc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 -- 172.21.15.87:0/2497114266 shutdown_connections 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 --2- 172.21.15.87:0/2497114266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c069aa0 0x7fd20c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 --2- 172.21.15.87:0/2497114266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c069560 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 --2- 172.21.15.87:0/2497114266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c106f50 0x7fd20c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 -- 172.21.15.87:0/2497114266 >> 172.21.15.87:0/2497114266 conn(0x7fd20c077e30 msgr2=0x7fd20c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 -- 172.21.15.87:0/2497114266 shutdown_connections 2024-04-22T13:33:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.912+0000 7fd214270640 1 -- 172.21.15.87:0/2497114266 wait complete. 2024-04-22T13:33:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 Processor -- start 2024-04-22T13:33:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 -- start start 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c069aa0 0x7fd20c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c106f50 0x7fd20c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c112150 con 0x7fd20c0690e0 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c111fd0 con 0x7fd20c069aa0 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd214270640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c1122d0 con 0x7fd20c106f50 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38302/0 (socket says 172.21.15.87:38302) 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 -- 172.21.15.87:0/3308403343 learned_addr learned my addr 172.21.15.87:0/3308403343 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21180b640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c069aa0 0x7fd20c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21280d640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c106f50 0x7fd20c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 -- 172.21.15.87:0/3308403343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c106f50 msgr2=0x7fd20c101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c106f50 0x7fd20c101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 -- 172.21.15.87:0/3308403343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c069aa0 msgr2=0x7fd20c1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c069aa0 0x7fd20c1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.913+0000 7fd21200c640 1 -- 172.21.15.87:0/3308403343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c1a6b50 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd21200c640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd20800d950 tx=0x7fd20800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208017030 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd208004020 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c1a6e40 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2080047a0 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.914+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c1a7300 con 0x7fd20c0690e0 2024-04-22T13:33:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.915+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c068de0 con 0x7fd20c0690e0 2024-04-22T13:33:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.916+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd208009a30 con 0x7fd20c0690e0 2024-04-22T13:33:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.917+0000 7fd2037fe640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 0x7fd1f80a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.917+0000 7fd21180b640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 0x7fd1f80a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.917+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd2080d0320 con 0x7fd20c0690e0 2024-04-22T13:33:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.918+0000 7fd21180b640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 0x7fd1f80a6440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006010 tx=0x7fd1fc02c6c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:01.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:01.922+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd20809c7f0 con 0x7fd20c0690e0 2024-04-22T13:33:02.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.314+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd20c102750 con 0x7fd20c0690e0 2024-04-22T13:33:02.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.315+0000 7fd2037fe640 1 -- 172.21.15.87:0/3308403343 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fd20c102750 con 0x7fd20c0690e0 2024-04-22T13:33:02.310 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:02.310 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:02.311 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:02.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.317+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 msgr2=0x7fd1f80a6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.317+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 0x7fd1f80a6440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006010 tx=0x7fd1fc02c6c0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.317+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 msgr2=0x7fd20c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.317+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd20800d950 tx=0x7fd20800de10 comp rx=0 tx=0).stop 2024-04-22T13:33:02.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 shutdown_connections 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd1f80a3f80 0x7fd1f80a6440 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd20c106f50 0x7fd20c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd20c069aa0 0x7fd20c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 --2- 172.21.15.87:0/3308403343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd20c0690e0 0x7fd20c101270 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 >> 172.21.15.87:0/3308403343 conn(0x7fd20c077e30 msgr2=0x7fd20c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 shutdown_connections 2024-04-22T13:33:02.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.318+0000 7fd214270640 1 -- 172.21.15.87:0/3308403343 wait complete. 2024-04-22T13:33:02.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:33:02.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 -- 172.21.15.87:0/2655634552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 msgr2=0x7f7b8810e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/2655634552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b8810e3c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009900 tx=0x7f7b7402bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 -- 172.21.15.87:0/2655634552 shutdown_connections 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/2655634552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b88110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/2655634552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b8810e3c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/2655634552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 0x7f7b88101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 -- 172.21.15.87:0/2655634552 >> 172.21.15.87:0/2655634552 conn(0x7f7b88077f50 msgr2=0x7f7b88078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 -- 172.21.15.87:0/2655634552 shutdown_connections 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.434+0000 7f7b8ea70640 1 -- 172.21.15.87:0/2655634552 wait complete. 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 Processor -- start 2024-04-22T13:33:02.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 -- start start 2024-04-22T13:33:02.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 0x7f7b881a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b8819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7b88114430 con 0x7f7b88101670 2024-04-22T13:33:02.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881142b0 con 0x7f7b8810e900 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b8ea70640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881145b0 con 0x7f7b88100d50 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.435+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38318/0 (socket says 172.21.15.87:38318) 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 -- 172.21.15.87:0/491685399 learned_addr learned my addr 172.21.15.87:0/491685399 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 0x7f7b881a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b8819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 -- 172.21.15.87:0/491685399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 msgr2=0x7f7b881a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 0x7f7b881a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 -- 172.21.15.87:0/491685399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 msgr2=0x7f7b8819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b8819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 -- 172.21.15.87:0/491685399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8819afe0 con 0x7f7b88101670 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b8819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:02.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b87fff640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000c00 tx=0x7f7b7402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b85ffb640 1 -- 172.21.15.87:0/491685399 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74043020 con 0x7f7b88101670 2024-04-22T13:33:02.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b85ffb640 1 -- 172.21.15.87:0/491685399 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b740052b0 con 0x7f7b88101670 2024-04-22T13:33:02.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8819b270 con 0x7f7b88101670 2024-04-22T13:33:02.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.436+0000 7f7b85ffb640 1 -- 172.21.15.87:0/491685399 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74003c60 con 0x7f7b88101670 2024-04-22T13:33:02.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.437+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b881a74a0 con 0x7f7b88101670 2024-04-22T13:33:02.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b85ffb640 1 -- 172.21.15.87:0/491685399 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f7b74046020 con 0x7f7b88101670 2024-04-22T13:33:02.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 Processor -- start 2024-04-22T13:33:02.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 -- start start 2024-04-22T13:33:02.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5acf4d0 0x5616e5acf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ad8a00 0x5616e5adae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5616e5a341f0 con 0x5616e5acfe80 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5616e59d14b0 con 0x5616e5ad8a00 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b9165a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5616e5a301b0 con 0x5616e5acf4d0 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38320/0 (socket says 172.21.15.87:38320) 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2171424891 learned_addr learned my addr 172.21.15.87:0/2171424891 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5acf4d0 0x5616e5acf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b85ffb640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 0x7f7b780a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b87fff640 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ad8a00 0x5616e5adae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.440+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 0x7f7b780a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2171424891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5acf4d0 msgr2=0x5616e5acf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5acf4d0 0x5616e5acf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2171424891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ad8a00 msgr2=0x5616e5adae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ad8a00 0x5616e5adae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2171424891 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e59c31b0 con 0x5616e5acfe80 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b85ffb640 1 -- 172.21.15.87:0/491685399 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7b740f2bd0 con 0x7f7b88101670 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b760 tx=0x7f7b7c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a7385ae23041d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 0x7f7b780a6390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b8806b6e0 tx=0x7f7b80004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b72ffd640 1 -- 172.21.15.87:0/2171424891 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c003dd0 con 0x5616e5acfe80 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b72ffd640 1 -- 172.21.15.87:0/2171424891 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b7c015070 con 0x5616e5acfe80 2024-04-22T13:33:02.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.441+0000 7f7b72ffd640 1 -- 172.21.15.87:0/2171424891 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c004e00 con 0x5616e5acfe80 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- 172.21.15.87:0/2171424891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 msgr2=0x5616e5ad8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00b760 tx=0x7f7b7c00bc20 comp rx=0 tx=0).stop 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- 172.21.15.87:0/2171424891 shutdown_connections 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ad8a00 0x5616e5adae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad8430 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2171424891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5acf4d0 0x5616e5acf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- 172.21.15.87:0/2171424891 >> 172.21.15.87:0/2171424891 conn(0x5616e5ac9ba0 msgr2=0x5616e5ac9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- 172.21.15.87:0/2171424891 shutdown_connections 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- 172.21.15.87:0/2171424891 wait complete. 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 -- start start 2024-04-22T13:33:02.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.442+0000 7f7b9165a740 1 Processor -- start 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 0x5616e5ad5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5616e5adb970 con 0x5616e5acf4d0 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5616e59ed500 con 0x5616e5acfe80 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5616e5adbaf0 con 0x5616e5ad8a00 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b9165a740 10 client.0 _ll_register_callbacks cb 0x5616e59bbb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38324/0 (socket says 172.21.15.87:38324) 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 -- 172.21.15.87:0/160441437 learned_addr learned my addr 172.21.15.87:0/160441437 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b87fff640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 0x5616e5ad5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 -- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 msgr2=0x5616e5ad5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 -- 172.21.15.87:0/160441437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 msgr2=0x5616e5ad5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 0x5616e5ad5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b8c80c640 1 -- 172.21.15.87:0/160441437 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e57463f0 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.443+0000 7f7b87fff640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 0x5616e5ad5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0057c0 tx=0x7f7b7c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26b048f364838b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b67fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c003de0 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7b7c011440 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b7c015070 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b7c011640 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b9165a740 10 client.15993 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.444+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5616e58107d0 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b7c015070 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7b7c025880 con 0x5616e5acf4d0 2024-04-22T13:33:02.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 10 client.15993 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 10 client.15993 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 20 client.15993 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5af1350 0x5616e5af3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 4 client.15993 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616e59d3930 con 0x5616e5af1350 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.445+0000 7f7b87fff640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5af1350 0x5616e5af3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.446+0000 7f7b67fff640 10 client.15993 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.446+0000 7f7b87fff640 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5af1350 0x5616e5af3740 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e162e47c4185392b server_cookie=8ff238d58f1c0d69 in_seq=0 out_seq=0 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.446+0000 7f7b67fff640 1 -- 172.21.15.87:0/160441437 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616e59d3930 con 0x5616e5af1350 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.446+0000 7f7b67fff640 10 client.15993 handle_command_reply: tid=0 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.446+0000 7f7b9165a740 1 client.15993 shutdown 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 msgr2=0x5616e5ad82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0057c0 tx=0x7f7b7c00bf20 comp rx=0 tx=0).stop 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 shutdown_connections 2024-04-22T13:33:02.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5af1350 0x5616e5af3740 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b67fff640 0 client.15993 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5acfe80 0x5616e5ad5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 --2- 172.21.15.87:0/160441437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acf4d0 0x5616e5ad82c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 >> 172.21.15.87:0/160441437 conn(0x5616e5ac9ba0 msgr2=0x5616e5acb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 shutdown_connections 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 1 -- 172.21.15.87:0/160441437 wait complete. 2024-04-22T13:33:02.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.447+0000 7f7b9165a740 20 client.15993 trim_cache size 0 max 16384 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 Processor -- start 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 -- start start 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5af1350 0x5616e5ad82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881ae180 con 0x5616e5acfe80 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881aba50 con 0x5616e5af1350 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b9165a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881a98a0 con 0x5616e5ad8a00 2024-04-22T13:33:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38330/0 (socket says 172.21.15.87:38330) 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2924923110 learned_addr learned my addr 172.21.15.87:0/2924923110 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b87fff640 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5af1350 0x5616e5ad82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2924923110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 msgr2=0x5616e5ad7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2924923110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5af1350 msgr2=0x5616e5ad82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5af1350 0x5616e5ad82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.472+0000 7f7b8c80c640 1 -- 172.21.15.87:0/2924923110 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e5ad5f20 con 0x5616e5acfe80 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.473+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c025e90 tx=0x7f7b7c004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109092084484e29a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.473+0000 7f7b727fc640 1 -- 172.21.15.87:0/2924923110 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c03b020 con 0x5616e5acfe80 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.473+0000 7f7b727fc640 1 -- 172.21.15.87:0/2924923110 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b7c015040 con 0x5616e5acfe80 2024-04-22T13:33:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.473+0000 7f7b727fc640 1 -- 172.21.15.87:0/2924923110 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c011c90 con 0x5616e5acfe80 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 -- 172.21.15.87:0/2924923110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 msgr2=0x5616e5ad0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c025e90 tx=0x7f7b7c004640 comp rx=0 tx=0).stop 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 -- 172.21.15.87:0/2924923110 shutdown_connections 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5af1350 0x5616e5ad82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ad7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 --2- 172.21.15.87:0/2924923110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5acfe80 0x5616e5ad0ae0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 -- 172.21.15.87:0/2924923110 >> 172.21.15.87:0/2924923110 conn(0x5616e5ac9ba0 msgr2=0x5616e5ad03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 -- 172.21.15.87:0/2924923110 shutdown_connections 2024-04-22T13:33:02.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.474+0000 7f7b9165a740 1 -- 172.21.15.87:0/2924923110 wait complete. 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 -- start start 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 Processor -- start 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ae9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ae5100 0x5616e5ae5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881a9f10 con 0x5616e5af1350 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881ab380 con 0x5616e5ae5100 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b881aa5e0 con 0x5616e5ad8a00 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b9165a740 10 client.0 _ll_register_callbacks cb 0x7f7b7c004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b87fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38340/0 (socket says 172.21.15.87:38340) 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b87fff640 1 -- 172.21.15.87:0/1273820266 learned_addr learned my addr 172.21.15.87:0/1273820266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b8d00d640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ae5100 0x5616e5ae5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.475+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ae9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 -- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 msgr2=0x5616e5ae9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ae9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 -- 172.21.15.87:0/1273820266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ae5100 msgr2=0x5616e5ae5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ae5100 0x5616e5ae5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 -- 172.21.15.87:0/1273820266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e5b56720 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b87fff640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7b740f02c0 tx=0x7f7b740df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b800797b71c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b67fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b740d7070 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7b74005460 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b7c015040 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b74088050 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b9165a740 10 client.15999 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.476+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5616e5b57920 con 0x5616e5af1350 2024-04-22T13:33:02.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b7c015040 con 0x5616e5af1350 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7b740bb120 con 0x5616e5af1350 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 10 client.15999 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 10 client.15999 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 20 client.15999 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5b58050 0x5616e5b5a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 4 client.15999 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616e5a41c10 con 0x5616e5b58050 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.477+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5b58050 0x5616e5b5a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b67fff640 10 client.15999 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b8c80c640 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5b58050 0x5616e5b5a470 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a03b240c949daa9 server_cookie=32673541e216e214 in_seq=0 out_seq=0 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b67fff640 1 -- 172.21.15.87:0/1273820266 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616e5a41c10 con 0x5616e5b58050 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b67fff640 10 client.15999 handle_command_reply: tid=0 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 client.15999 shutdown 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 msgr2=0x5616e5ae4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7b740f02c0 tx=0x7f7b740df040 comp rx=0 tx=0).stop 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 shutdown_connections 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5616e5b58050 0x5616e5b5a470 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616e5ae5100 0x5616e5ae5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5616e5af1350 0x5616e5ae4bc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 --2- 172.21.15.87:0/1273820266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616e5ad8a00 0x5616e5ae9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 >> 172.21.15.87:0/1273820266 conn(0x5616e5ac9ba0 msgr2=0x5616e5ad0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b67fff640 0 client.15999 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 shutdown_connections 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 1 -- 172.21.15.87:0/1273820266 wait complete. 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.478+0000 7f7b9165a740 20 client.15999 trim_cache size 0 max 16384 2024-04-22T13:33:02.481 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 msgr2=0x7f7b780a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 0x7f7b780a6390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b8806b6e0 tx=0x7f7b80004ab0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 msgr2=0x7f7b881a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000c00 tx=0x7f7b7402c7e0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 shutdown_connections 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7b780a3ed0 0x7f7b780a6390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8810e900 0x7f7b8819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b88101670 0x7f7b881a0b90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 --2- 172.21.15.87:0/491685399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b88100d50 0x7f7b881a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 >> 172.21.15.87:0/491685399 conn(0x7f7b88077f50 msgr2=0x7f7b8810ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 shutdown_connections 2024-04-22T13:33:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.479+0000 7f7b8ea70640 1 -- 172.21.15.87:0/491685399 wait complete. 2024-04-22T13:33:02.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:33:02.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 -- 172.21.15.87:0/3861495401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 msgr2=0x7f942010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 --2- 172.21.15.87:0/3861495401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f942010dd30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ca0 tx=0x7f941802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 -- 172.21.15.87:0/3861495401 shutdown_connections 2024-04-22T13:33:02.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 --2- 172.21.15.87:0/3861495401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f942010dd30 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 --2- 172.21.15.87:0/3861495401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101a40 0x7f9420105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 --2- 172.21.15.87:0/3861495401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9420101090 0x7f9420101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 -- 172.21.15.87:0/3861495401 >> 172.21.15.87:0/3861495401 conn(0x7f9420077f50 msgr2=0x7f9420078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 -- 172.21.15.87:0/3861495401 shutdown_connections 2024-04-22T13:33:02.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.595+0000 7f94276eb640 1 -- 172.21.15.87:0/3861495401 wait complete. 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 Processor -- start 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 -- start start 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 0x7f94201986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 0x7f942019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f94201121e0 con 0x7f9420106160 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112060 con 0x7f9420101090 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f94276eb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112360 con 0x7f942019cfa0 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.596+0000 7f9424c86640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38356/0 (socket says 172.21.15.87:38356) 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 -- 172.21.15.87:0/629565332 learned_addr learned my addr 172.21.15.87:0/629565332 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9425487640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 0x7f94201986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9425c88640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 0x7f942019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 -- 172.21.15.87:0/629565332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 msgr2=0x7f942019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 0x7f942019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 -- 172.21.15.87:0/629565332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 msgr2=0x7f94201986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 0x7f94201986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 -- 172.21.15.87:0/629565332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942019da90 con 0x7f9420106160 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9425487640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 0x7f94201986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9425c88640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 0x7f942019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f9424c86640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f941000d770 tx=0x7f941000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f94167fc640 1 -- 172.21.15.87:0/629565332 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410017070 con 0x7f9420106160 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942019dd80 con 0x7f9420106160 2024-04-22T13:33:02.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f94167fc640 1 -- 172.21.15.87:0/629565332 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9410009c30 con 0x7f9420106160 2024-04-22T13:33:02.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f94167fc640 1 -- 172.21.15.87:0/629565332 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941000f620 con 0x7f9420106160 2024-04-22T13:33:02.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.597+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942010e910 con 0x7f9420106160 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.600+0000 7f94167fc640 1 -- 172.21.15.87:0/629565332 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f941001a070 con 0x7f9420106160 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 Processor -- start 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f94167fc640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 0x7f940c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 -- start start 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425c88640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 0x7f940c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f94167fc640 1 -- 172.21.15.87:0/629565332 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f94100d0600 con 0x7f9420106160 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d28430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b56c841f0 con 0x558b56d1f4d0 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b56c214b0 con 0x558b56d28a00 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f942a3b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b56c801b0 con 0x558b56d1fe80 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38360/0 (socket says 172.21.15.87:38360) 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 -- 172.21.15.87:0/2611783275 learned_addr learned my addr 172.21.15.87:0/2611783275 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425c88640 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9424c86640 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d28430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 -- 172.21.15.87:0/2611783275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 msgr2=0x558b56d28430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d28430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 -- 172.21.15.87:0/2611783275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 msgr2=0x558b56d2ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.601+0000 7f9425487640 1 -- 172.21.15.87:0/2611783275 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b56c131b0 con 0x558b56d1f4d0 2024-04-22T13:33:02.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.602+0000 7f9425c88640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 0x7f940c0a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9418005e30 tx=0x7f941802cf90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.602+0000 7f9425487640 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f941c00bd40 tx=0x7f941c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e8b7299700eda2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.602+0000 7f940b7fe640 1 -- 172.21.15.87:0/2611783275 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c015070 con 0x558b56d1f4d0 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.602+0000 7f940b7fe640 1 -- 172.21.15.87:0/2611783275 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f941c00be30 con 0x558b56d1f4d0 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.602+0000 7f940b7fe640 1 -- 172.21.15.87:0/2611783275 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c0118b0 con 0x558b56d1f4d0 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 -- 172.21.15.87:0/2611783275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 msgr2=0x558b56d1f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f941c00bd40 tx=0x7f941c00f730 comp rx=0 tx=0).stop 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 -- 172.21.15.87:0/2611783275 shutdown_connections 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d28430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 --2- 172.21.15.87:0/2611783275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d1f8b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 -- 172.21.15.87:0/2611783275 >> 172.21.15.87:0/2611783275 conn(0x558b56d19ba0 msgr2=0x558b56d19f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 -- 172.21.15.87:0/2611783275 shutdown_connections 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.603+0000 7f942a3b8740 1 -- 172.21.15.87:0/2611783275 wait complete. 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 -- start start 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 Processor -- start 2024-04-22T13:33:02.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d28380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d25710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b56d2b970 con 0x558b56d28a00 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b56c3d500 con 0x558b56d1f4d0 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b56d2baf0 con 0x558b56d1fe80 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.604+0000 7f942a3b8740 10 client.0 _ll_register_callbacks cb 0x558b56c0bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38364/0 (socket says 172.21.15.87:38364) 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 -- 172.21.15.87:0/3122271261 learned_addr learned my addr 172.21.15.87:0/3122271261 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9424c86640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d25710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425487640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d28380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 -- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 msgr2=0x558b56d25710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d25710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 -- 172.21.15.87:0/3122271261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 msgr2=0x558b56d28380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d28380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 -- 172.21.15.87:0/3122271261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b569963f0 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425487640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d28380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9425c88640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ca0 tx=0x7f941802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0732f7492bcda2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9408ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418050020 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9418003d80 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.605+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f941c00be30 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f941805b050 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 10 client.16011 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b56a607d0 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f941c00be30 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9418034710 con 0x558b56d28a00 2024-04-22T13:33:02.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 10 client.16011 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 10 client.16011 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 20 client.16011 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56d3b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 4 client.16011 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b56c23930 con 0x558b56d38ce0 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.606+0000 7f9425487640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56d3b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.607+0000 7f9425487640 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56d3b110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=235db0e7aa9c40fe server_cookie=3309e42f50a0fbe2 in_seq=0 out_seq=0 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.607+0000 7f9408ff9640 10 client.16011 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.607+0000 7f9408ff9640 1 -- 172.21.15.87:0/3122271261 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b56c23930 con 0x558b56d38ce0 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.607+0000 7f9408ff9640 10 client.16011 handle_command_reply: tid=0 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.607+0000 7f942a3b8740 1 client.16011 shutdown 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 msgr2=0x558b56d25d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ca0 tx=0x7f941802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 shutdown_connections 2024-04-22T13:33:02.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56d3b110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f9408ff9640 0 client.16011 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d28a00 0x558b56d25d20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d25710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3122271261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d28380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 >> 172.21.15.87:0/3122271261 conn(0x558b56d19ba0 msgr2=0x558b56d1b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 shutdown_connections 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 1 -- 172.21.15.87:0/3122271261 wait complete. 2024-04-22T13:33:02.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.608+0000 7f942a3b8740 20 client.16011 trim_cache size 0 max 16384 2024-04-22T13:33:02.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 Processor -- start 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 -- start start 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d34830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d28a00 0x558b56d27e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b24b0 con 0x558b56d1fe80 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94201afd80 con 0x558b56d1f4d0 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f942a3b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94201adbf0 con 0x558b56d28a00 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38366/0 (socket says 172.21.15.87:38366) 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 -- 172.21.15.87:0/3370954848 learned_addr learned my addr 172.21.15.87:0/3370954848 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9425c88640 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d28a00 0x558b56d27e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9425487640 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d34830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 -- 172.21.15.87:0/3370954848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d28a00 msgr2=0x558b56d27e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d28a00 0x558b56d27e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 -- 172.21.15.87:0/3370954848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 msgr2=0x558b56d34830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d34830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.632+0000 7f9424c86640 1 -- 172.21.15.87:0/3370954848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b56d1be90 con 0x558b56d1fe80 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.633+0000 7f9424c86640 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9410098830 tx=0x7f941000de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538a3dca6395d853 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.633+0000 7f940affd640 1 -- 172.21.15.87:0/3370954848 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410021040 con 0x558b56d1fe80 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.633+0000 7f940affd640 1 -- 172.21.15.87:0/3370954848 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94100a0780 con 0x558b56d1fe80 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.633+0000 7f940affd640 1 -- 172.21.15.87:0/3370954848 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410098120 con 0x558b56d1fe80 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- 172.21.15.87:0/3370954848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 msgr2=0x558b56d24650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9410098830 tx=0x7f941000de80 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- 172.21.15.87:0/3370954848 shutdown_connections 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d28a00 0x558b56d27e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1fe80 0x558b56d24650 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3370954848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d1f4d0 0x558b56d34830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- 172.21.15.87:0/3370954848 >> 172.21.15.87:0/3370954848 conn(0x558b56d19ba0 msgr2=0x558b56d1b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- 172.21.15.87:0/3370954848 shutdown_connections 2024-04-22T13:33:02.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- 172.21.15.87:0/3370954848 wait complete. 2024-04-22T13:33:02.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 -- start start 2024-04-22T13:33:02.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.634+0000 7f942a3b8740 1 Processor -- start 2024-04-22T13:33:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d2c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ae240 con 0x558b56d1f4d0 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94201af6b0 con 0x558b56d28a00 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94201ae910 con 0x558b56d1fe80 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f942a3b8740 10 client.0 _ll_register_callbacks cb 0x7f9418009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38382/0 (socket says 172.21.15.87:38382) 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 -- 172.21.15.87:0/3733855598 learned_addr learned my addr 172.21.15.87:0/3733855598 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9424c86640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d2c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425c88640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 -- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 msgr2=0x558b56d2c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d2c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 -- 172.21.15.87:0/3733855598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 msgr2=0x558b56d2cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 -- 172.21.15.87:0/3733855598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b56d9e2a0 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9425487640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f941c005c50 tx=0x7f941c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6372439eb7a35591 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9408ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.635+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c015040 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f941c0051c0 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94100a0780 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f941c038050 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 10 client.16017 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558b56d9f420 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94100a0780 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f941c004680 con 0x558b56d1f4d0 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 10 client.16017 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 10 client.16017 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 20 client.16017 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56da1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 4 client.16017 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b56c91c10 con 0x558b56d38ce0 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.636+0000 7f9424c86640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56da1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f9408ff9640 10 client.16017 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f9424c86640 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56da1af0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5fb6238aded7cff4 server_cookie=5def2b4e0a4d1725 in_seq=0 out_seq=0 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f9408ff9640 1 -- 172.21.15.87:0/3733855598 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b56c91c10 con 0x558b56d38ce0 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f9408ff9640 10 client.16017 handle_command_reply: tid=0 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 client.16017 shutdown 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 msgr2=0x558b56d31270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f941c005c50 tx=0x7f941c00ba10 comp rx=0 tx=0).stop 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 shutdown_connections 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558b56d38ce0 0x558b56da1af0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b56d28a00 0x558b56d2cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b56d1fe80 0x558b56d2c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f9408ff9640 0 client.16017 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 --2- 172.21.15.87:0/3733855598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b56d1f4d0 0x558b56d31270 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.637+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 >> 172.21.15.87:0/3733855598 conn(0x558b56d19ba0 msgr2=0x558b56d18370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 shutdown_connections 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f942a3b8740 1 -- 172.21.15.87:0/3733855598 wait complete. 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f942a3b8740 20 client.16017 trim_cache size 0 max 16384 2024-04-22T13:33:02.639 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 msgr2=0x7f940c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 0x7f940c0a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9418005e30 tx=0x7f941802cf90 comp rx=0 tx=0).stop 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 msgr2=0x7f9420198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f941000d770 tx=0x7f941000dc30 comp rx=0 tx=0).stop 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 shutdown_connections 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f940c0a3ed0 0x7f940c0a6390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f942019cfa0 0x7f942019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9420106160 0x7f9420198c20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 --2- 172.21.15.87:0/629565332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9420101090 0x7f94201986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 >> 172.21.15.87:0/629565332 conn(0x7f9420077f50 msgr2=0x7f94200fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 shutdown_connections 2024-04-22T13:33:02.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.638+0000 7f94276eb640 1 -- 172.21.15.87:0/629565332 wait complete. 2024-04-22T13:33:02.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 -- 172.21.15.87:0/1502621541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc102190 msgr2=0x7f78bc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 --2- 172.21.15.87:0/1502621541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc102190 0x7f78bc102570 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f78b8009900 tx=0x7f78b802bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 -- 172.21.15.87:0/1502621541 shutdown_connections 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 --2- 172.21.15.87:0/1502621541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc107260 0x7f78bc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 --2- 172.21.15.87:0/1502621541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102b40 0x7f78bc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 --2- 172.21.15.87:0/1502621541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc102190 0x7f78bc102570 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 -- 172.21.15.87:0/1502621541 >> 172.21.15.87:0/1502621541 conn(0x7f78bc0793a0 msgr2=0x7f78bc0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 -- 172.21.15.87:0/1502621541 shutdown_connections 2024-04-22T13:33:02.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.752+0000 7f78c434b640 1 -- 172.21.15.87:0/1502621541 wait complete. 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 Processor -- start 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 -- start start 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102190 0x7f78bc19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 0x7f78bc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1121c0 con 0x7f78bc107260 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc112040 con 0x7f78bc1985e0 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c434b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc112340 con 0x7f78bc102190 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c18e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c18e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38404/0 (socket says 172.21.15.87:38404) 2024-04-22T13:33:02.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c18e6640 1 -- 172.21.15.87:0/2246327648 learned_addr learned my addr 172.21.15.87:0/2246327648 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c20e7640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102190 0x7f78bc19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.753+0000 7f78c28e8640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 0x7f78bc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 -- 172.21.15.87:0/2246327648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102190 msgr2=0x7f78bc19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102190 0x7f78bc19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 -- 172.21.15.87:0/2246327648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 msgr2=0x7f78bc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 0x7f78bc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 -- 172.21.15.87:0/2246327648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc199100 con 0x7f78bc107260 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c28e8640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 0x7f78bc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c18e6640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00d740 tx=0x7f78ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78b37fe640 1 -- 172.21.15.87:0/2246327648 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac017070 con 0x7f78bc107260 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78bc1993f0 con 0x7f78bc107260 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78b37fe640 1 -- 172.21.15.87:0/2246327648 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78ac00de50 con 0x7f78bc107260 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78b37fe640 1 -- 172.21.15.87:0/2246327648 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac004ce0 con 0x7f78bc107260 2024-04-22T13:33:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.754+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78bc1a55f0 con 0x7f78bc107260 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.757+0000 7f78c7014740 1 Processor -- start 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.757+0000 7f78c7014740 1 -- start start 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.757+0000 7f78b37fe640 1 -- 172.21.15.87:0/2246327648 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f78ac004e80 con 0x7f78bc107260 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 0x5572a67af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5572a67041f0 con 0x5572a679fd20 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5572a66a14b0 con 0x5572a679f370 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c7014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5572a67001b0 con 0x5572a67ace10 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c20e7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60868/0 (socket says 172.21.15.87:60868) 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 -- 172.21.15.87:0/1992598612 learned_addr learned my addr 172.21.15.87:0/1992598612 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c18e6640 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 0x5572a67af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 -- 172.21.15.87:0/1992598612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 msgr2=0x5572a67af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 0x5572a67af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 -- 172.21.15.87:0/1992598612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 msgr2=0x5572a67ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78b37fe640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 0x7f78a80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c28e8640 1 -- 172.21.15.87:0/1992598612 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a66931b0 con 0x5572a679f370 2024-04-22T13:33:02.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.758+0000 7f78c18e6640 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 0x5572a67af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78b37fe640 1 -- 172.21.15.87:0/2246327648 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78ac0d0150 con 0x7f78bc107260 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78c28e8640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 0x7f78a80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78c28e8640 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009e40 tx=0x7f78b400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ac1776ddc6068 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78b0bfe640 1 -- 172.21.15.87:0/1992598612 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4021070 con 0x5572a679f370 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78b0bfe640 1 -- 172.21.15.87:0/1992598612 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b401bc20 con 0x5572a679f370 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.759+0000 7f78b0bfe640 1 -- 172.21.15.87:0/1992598612 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4010ac0 con 0x5572a679f370 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 -- 172.21.15.87:0/1992598612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 msgr2=0x5572a679f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c28e8640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 0x7f78a80a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f78b401e000 tx=0x7f78b4021000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009e40 tx=0x7f78b400f730 comp rx=0 tx=0).stop 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 -- 172.21.15.87:0/1992598612 shutdown_connections 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a67ace10 0x5572a67af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 --2- 172.21.15.87:0/1992598612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a679f750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 -- 172.21.15.87:0/1992598612 >> 172.21.15.87:0/1992598612 conn(0x5572a6799ba0 msgr2=0x5572a6799f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 -- 172.21.15.87:0/1992598612 shutdown_connections 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.760+0000 7f78c7014740 1 -- 172.21.15.87:0/1992598612 wait complete. 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 -- start start 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 Processor -- start 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a67ace10 0x5572a67a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5572a67afd40 con 0x5572a679f370 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c20e7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c20e7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38432/0 (socket says 172.21.15.87:38432) 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c20e7640 1 -- 172.21.15.87:0/1548396682 learned_addr learned my addr 172.21.15.87:0/1548396682 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5572a66bd500 con 0x5572a67ace10 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5572a67afec0 con 0x5572a679fd20 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c28e8640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a67ace10 0x5572a67a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.761+0000 7f78c18e6640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 -- 172.21.15.87:0/1548396682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 msgr2=0x5572a67a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 -- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a67ace10 msgr2=0x5572a67a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a67ace10 0x5572a67a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a64163f0 con 0x5572a679f370 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c7014740 10 client.0 _ll_register_callbacks cb 0x5572a668bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b401bc20 con 0x5572a679f370 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78a5ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c20e7640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f78b802c170 tx=0x7f78b802c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64b626f252bf8da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78a5ffb640 1 -- 172.21.15.87:0/1548396682 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8005230 con 0x5572a679f370 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78a5ffb640 1 -- 172.21.15.87:0/1548396682 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78b80053d0 con 0x5572a679f370 2024-04-22T13:33:02.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78a5ffb640 1 -- 172.21.15.87:0/1548396682 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78b802c5d0 con 0x5572a679f370 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c7014740 10 client.16029 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.762+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5572a64e07d0 con 0x5572a679f370 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78a5ffb640 1 -- 172.21.15.87:0/1548396682 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f78b80046c0 con 0x5572a679f370 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 10 client.16029 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 20 client.16029 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67bccd0 0x5572a67bf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 4 client.16029 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572a66a3930 con 0x5572a67bccd0 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c18e6640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67bccd0 0x5572a67bf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78a5ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.763+0000 7f78c18e6640 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67bccd0 0x5572a67bf170 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef47bb23df834bfc server_cookie=3f5b8b19a8a547ce in_seq=0 out_seq=0 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78a5ffb640 1 -- 172.21.15.87:0/1548396682 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572a66a3930 con 0x5572a67bccd0 2024-04-22T13:33:02.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78a5ffb640 10 client.16029 handle_command_reply: tid=0 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 client.16029 shutdown 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 msgr2=0x5572a67ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f78b802c170 tx=0x7f78b802c9f0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 shutdown_connections 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67bccd0 0x5572a67bf170 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a67ace10 0x5572a67a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 --2- 172.21.15.87:0/1548396682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679f370 0x5572a67ac6b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 >> 172.21.15.87:0/1548396682 conn(0x5572a6799ba0 msgr2=0x5572a679b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78a5ffb640 0 client.16029 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 shutdown_connections 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 1 -- 172.21.15.87:0/1548396682 wait complete. 2024-04-22T13:33:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.764+0000 7f78c7014740 20 client.16029 trim_cache size 0 max 16384 2024-04-22T13:33:02.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 Processor -- start 2024-04-22T13:33:02.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 -- start start 2024-04-22T13:33:02.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a67a8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67ac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1ae170 con 0x5572a6799ba0 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1aba40 con 0x5572a679f370 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c7014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1a98b0 con 0x5572a679fd20 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38448/0 (socket says 172.21.15.87:38448) 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 -- 172.21.15.87:0/2598145757 learned_addr learned my addr 172.21.15.87:0/2598145757 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c28e8640 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c18e6640 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a67a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 -- 172.21.15.87:0/2598145757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 msgr2=0x5572a67ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 -- 172.21.15.87:0/2598145757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 msgr2=0x5572a67a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a67a8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.782+0000 7f78c20e7640 1 -- 172.21.15.87:0/2598145757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a67a89f0 con 0x5572a6799ba0 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78c20e7640 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f78b803ebd0 tx=0x7f78b8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904ef1921c73321 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78a7fff640 1 -- 172.21.15.87:0/2598145757 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b80523d0 con 0x5572a6799ba0 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78a7fff640 1 -- 172.21.15.87:0/2598145757 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b8058070 con 0x5572a6799ba0 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78a7fff640 1 -- 172.21.15.87:0/2598145757 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8005d50 con 0x5572a6799ba0 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78c7014740 1 -- 172.21.15.87:0/2598145757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 msgr2=0x5572a67b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.783+0000 7f78c7014740 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f78b803ebd0 tx=0x7f78b8043000 comp rx=0 tx=0).stop 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- 172.21.15.87:0/2598145757 shutdown_connections 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679fd20 0x5572a67ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a679f370 0x5572a67a8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- 172.21.15.87:0/2598145757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a6799ba0 0x5572a67b8c40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- 172.21.15.87:0/2598145757 >> 172.21.15.87:0/2598145757 conn(0x5572a67aa0e0 msgr2=0x5572a67a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- 172.21.15.87:0/2598145757 shutdown_connections 2024-04-22T13:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- 172.21.15.87:0/2598145757 wait complete. 2024-04-22T13:33:02.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- start start 2024-04-22T13:33:02.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 Processor -- start 2024-04-22T13:33:02.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 0x5572a67b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679f370 0x5572a67b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1a9f00 con 0x5572a679fd20 2024-04-22T13:33:02.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1ab370 con 0x5572a6799ba0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc1aa5d0 con 0x5572a679f370 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c7014740 10 client.0 _ll_register_callbacks cb 0x7f78b8004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c28e8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c28e8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38452/0 (socket says 172.21.15.87:38452) 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c28e8640 1 -- 172.21.15.87:0/3303702268 learned_addr learned my addr 172.21.15.87:0/3303702268 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.784+0000 7f78c20e7640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 0x5572a67b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c18e6640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679f370 0x5572a67b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 -- 172.21.15.87:0/3303702268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679f370 msgr2=0x5572a67b13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679f370 0x5572a67b13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 -- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 msgr2=0x5572a67b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 0x5572a67b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 -- 172.21.15.87:0/3303702268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a68225d0 con 0x5572a679fd20 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c20e7640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 0x5572a67b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c28e8640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f78b4023000 tx=0x7f78b4009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c57faac56bd1da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:02.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4009920 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78b40106d0 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b8058070 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78b40108d0 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c7014740 10 client.16035 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.785+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5572a6823870 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b8058070 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f78b4031500 con 0x5572a679fd20 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 10 client.16035 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 20 client.16035 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67ace10 0x5572a6825ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 4 client.16035 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5572a6711c10 con 0x5572a67ace10 2024-04-22T13:33:02.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c20e7640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67ace10 0x5572a6825ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78c20e7640 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67ace10 0x5572a6825ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e60912d82347b888 server_cookie=b63ba0a13071db4d in_seq=0 out_seq=0 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.786+0000 7f78a5ffb640 10 client.16035 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78a5ffb640 1 -- 172.21.15.87:0/3303702268 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5572a6711c10 con 0x5572a67ace10 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78a5ffb640 10 client.16035 handle_command_reply: tid=0 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 client.16035 shutdown 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 msgr2=0x5572a67b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f78b4023000 tx=0x7f78b4009d70 comp rx=0 tx=0).stop 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 shutdown_connections 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5572a67ace10 0x5572a6825ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5572a679fd20 0x5572a67b1910 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572a679f370 0x5572a67b13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 --2- 172.21.15.87:0/3303702268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a6799ba0 0x5572a67b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 >> 172.21.15.87:0/3303702268 conn(0x5572a67aa0e0 msgr2=0x5572a679afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78a5ffb640 0 client.16035 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:02.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 shutdown_connections 2024-04-22T13:33:02.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 1 -- 172.21.15.87:0/3303702268 wait complete. 2024-04-22T13:33:02.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.787+0000 7f78c7014740 20 client.16035 trim_cache size 0 max 16384 2024-04-22T13:33:02.789 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 msgr2=0x7f78a80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 0x7f78a80a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f78b401e000 tx=0x7f78b4021000 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 msgr2=0x7f78bc19e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00d740 tx=0x7f78ac00dc00 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 shutdown_connections 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78a80a3ed0 0x7f78a80a6390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78bc1985e0 0x7f78bc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78bc107260 0x7f78bc19e970 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 --2- 172.21.15.87:0/2246327648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78bc102190 0x7f78bc19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 >> 172.21.15.87:0/2246327648 conn(0x7f78bc0793a0 msgr2=0x7f78bc100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 shutdown_connections 2024-04-22T13:33:02.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:02.788+0000 7f78c434b640 1 -- 172.21.15.87:0/2246327648 wait complete. 2024-04-22T13:33:03.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:03 smithi120 ceph-mon[31101]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:03.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:03 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3308403343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:03.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:03 smithi132 ceph-mon[31531]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:03.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:03 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3308403343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:03.725 INFO:tasks.quiescer.fs.[cephfs]:Total duration (47.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T13:33:03.725 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.55 2024-04-22T13:33:03.725 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:03.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:03 smithi087 ceph-mon[27999]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:03.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:03 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3308403343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:03.796 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:33:03.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:33:03.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/2646603260 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc1005a0 msgr2=0x7f8edc100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/2646603260 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc1005a0 0x7f8edc100a20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009900 tx=0x7f8ed002bcd0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/2646603260 shutdown_connections 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/2646603260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc100f60 0x7f8edc10d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/2646603260 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc1005a0 0x7f8edc100a20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/2646603260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/2646603260 >> 172.21.15.87:0/2646603260 conn(0x7f8edc0774f0 msgr2=0x7f8edc0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/2646603260 shutdown_connections 2024-04-22T13:33:03.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.867+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/2646603260 wait complete. 2024-04-22T13:33:03.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 Processor -- start 2024-04-22T13:33:03.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 -- start start 2024-04-22T13:33:03.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 0x7f8edc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc1121f0 con 0x7f8edc100f60 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc112070 con 0x7f8edc1005a0 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ee0e3f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc112370 con 0x7f8edc105560 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ed9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ed9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38474/0 (socket says 172.21.15.87:38474) 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8ed9d9b640 1 -- 172.21.15.87:0/3089524756 learned_addr learned my addr 172.21.15.87:0/3089524756 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8eda59c640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 0x7f8edc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.868+0000 7f8edad9d640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 -- 172.21.15.87:0/3089524756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 msgr2=0x7f8edc19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 -- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 msgr2=0x7f8edc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 -- 172.21.15.87:0/3089524756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc19d700 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8edad9d640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ed9d9b640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8eda59c640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 0x7f8edc198710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9c0 tx=0x7f8ecc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00ccc0 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc19d9f0 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ecc0035f0 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00ce60 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.869+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc1a5280 con 0x7f8edc1005a0 2024-04-22T13:33:03.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.870+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc068550 con 0x7f8edc1005a0 2024-04-22T13:33:03.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.873+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f8ecc004000 con 0x7f8edc1005a0 2024-04-22T13:33:03.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.873+0000 7f8ecb7fe640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 0x7f8ec40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.873+0000 7f8ed9d9b640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 0x7f8ec40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.874+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8ecc0d0e40 con 0x7f8edc1005a0 2024-04-22T13:33:03.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.874+0000 7f8ed9d9b640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 0x7f8ec40a6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000c00 tx=0x7f8ed00058d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.876+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ecc09d310 con 0x7f8edc1005a0 2024-04-22T13:33:03.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 -- 172.21.15.87:0/220020280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 msgr2=0x7fe2c010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/220020280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c010eeb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009ca0 tx=0x7fe2b802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 -- 172.21.15.87:0/220020280 shutdown_connections 2024-04-22T13:33:03.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/220020280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c010eeb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/220020280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102bc0 0x7fe2c0106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/220020280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c0102210 0x7fe2c01025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 -- 172.21.15.87:0/220020280 >> 172.21.15.87:0/220020280 conn(0x7fe2c00fdab0 msgr2=0x7fe2c00ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 -- 172.21.15.87:0/220020280 shutdown_connections 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.948+0000 7fe2c7d96640 1 -- 172.21.15.87:0/220020280 wait complete. 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 Processor -- start 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 -- start start 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 0x7fe2c019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c01986b0 0x7fe2c0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c0112230 con 0x7fe2c01072e0 2024-04-22T13:33:03.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01120b0 con 0x7fe2c0102210 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c7d96640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01123b0 con 0x7fe2c01986b0 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c019eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38482/0 (socket says 172.21.15.87:38482) 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c5331640 1 -- 172.21.15.87:0/3776121348 learned_addr learned my addr 172.21.15.87:0/3776121348 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.949+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 0x7fe2c019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c01986b0 0x7fe2c0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 -- 172.21.15.87:0/3776121348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c01986b0 msgr2=0x7fe2c0198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c01986b0 0x7fe2c0198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 -- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 msgr2=0x7fe2c019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 -- 172.21.15.87:0/3776121348 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c01992b0 con 0x7fe2c0102210 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 0x7fe2c019e5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc00d6d0 tx=0x7fe2bc00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2b6ffd640 1 -- 172.21.15.87:0/3776121348 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc017070 con 0x7fe2c0102210 2024-04-22T13:33:03.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2b6ffd640 1 -- 172.21.15.87:0/3776121348 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2bc009820 con 0x7fe2c0102210 2024-04-22T13:33:03.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2b6ffd640 1 -- 172.21.15.87:0/3776121348 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc004c40 con 0x7fe2c0102210 2024-04-22T13:33:03.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c0074180 con 0x7fe2c0102210 2024-04-22T13:33:03.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.950+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c00746d0 con 0x7fe2c0102210 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.953+0000 7fe2b6ffd640 1 -- 172.21.15.87:0/3776121348 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe2bc005220 con 0x7fe2c0102210 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 Processor -- start 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 -- start start 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db047c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db05ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5628dafabf00 con 0x5628db048230 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628daf49590 con 0x5628db047880 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2ca987740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5628dafa7ec0 con 0x5628db059620 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.954+0000 7fe2b6ffd640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 0x7fe2ac0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38496/0 (socket says 172.21.15.87:38496) 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 -- 172.21.15.87:0/117317247 learned_addr learned my addr 172.21.15.87:0/117317247 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 0x7fe2ac0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5331640 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db05ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2b6ffd640 1 -- 172.21.15.87:0/3776121348 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe2bc0d04c0 con 0x7fe2c0102210 2024-04-22T13:33:03.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c6333640 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db047c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 -- 172.21.15.87:0/117317247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 msgr2=0x5628db05ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db05ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 -- 172.21.15.87:0/117317247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 msgr2=0x5628db047c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db047c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 -- 172.21.15.87:0/117317247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628dad867e0 con 0x5628db048230 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc098d10 tx=0x7fe2bc09c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4834a813d13be0cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.955+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 0x7fe2ac0a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8004af0 tx=0x7fe2b80323f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.956+0000 7fe2abfff640 1 -- 172.21.15.87:0/117317247 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc020050 con 0x5628db048230 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.956+0000 7fe2abfff640 1 -- 172.21.15.87:0/117317247 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2bc09c340 con 0x5628db048230 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.956+0000 7fe2abfff640 1 -- 172.21.15.87:0/117317247 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0a0170 con 0x5628db048230 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 -- 172.21.15.87:0/117317247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 msgr2=0x5628db059050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc098d10 tx=0x7fe2bc09c210 comp rx=0 tx=0).stop 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 -- 172.21.15.87:0/117317247 shutdown_connections 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db05ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db059050 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 --2- 172.21.15.87:0/117317247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db047c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 -- 172.21.15.87:0/117317247 >> 172.21.15.87:0/117317247 conn(0x5628db041fb0 msgr2=0x5628db042390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 -- 172.21.15.87:0/117317247 shutdown_connections 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.957+0000 7fe2ca987740 1 -- 172.21.15.87:0/117317247 wait complete. 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.958+0000 7fe2ca987740 1 -- start start 2024-04-22T13:33:03.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.958+0000 7fe2ca987740 1 Processor -- start 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db048230 0x5628db056b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 0x5628db0570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5628db05c550 con 0x5628db059620 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2c5b32640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2c5b32640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:40814/0 (socket says 172.21.15.87:40814) 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2c5b32640 1 -- 172.21.15.87:0/945843464 learned_addr learned my addr 172.21.15.87:0/945843464 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2c6333640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 0x5628db0570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2c5331640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db048230 0x5628db056b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628daf65580 con 0x5628db048230 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5628db05c6d0 con 0x5628db047880 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.959+0000 7fe2ca987740 10 client.0 _ll_register_callbacks cb 0x5628daf6b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 -- 172.21.15.87:0/945843464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 msgr2=0x5628db056620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 -- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db048230 msgr2=0x5628db056b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db048230 0x5628db056b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628dade6f70 con 0x5628db059620 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2bc09c340 con 0x5628db059620 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2c6333640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 0x5628db0570a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe2b802c700 tx=0x7fe2b802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7aadf5627c3e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2a97fa640 10 client.16053 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2a97fa640 1 -- 172.21.15.87:0/945843464 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8039040 con 0x5628db059620 2024-04-22T13:33:03.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2a97fa640 1 -- 172.21.15.87:0/945843464 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe2b8004240 con 0x5628db059620 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2a97fa640 1 -- 172.21.15.87:0/945843464 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe2b8004440 con 0x5628db059620 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2ca987740 10 client.16053 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.960+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5628dac7e170 con 0x5628db059620 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2a97fa640 1 -- 172.21.15.87:0/945843464 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe2b80045e0 con 0x5628db059620 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 10 client.16053 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 20 client.16053 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 4 client.16053 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628dafb9920 con 0x5628db0699c0 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.961+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.962+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db06be80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a960270729da989 server_cookie=387b1619cfe46ec0 in_seq=0 out_seq=0 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.962+0000 7fe2a97fa640 10 client.16053 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2a97fa640 1 -- 172.21.15.87:0/945843464 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628dafb9920 con 0x5628db0699c0 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2a97fa640 10 client.16053 handle_command_reply: tid=0 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 client.16053 shutdown 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 msgr2=0x5628db0570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 0x5628db0570a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe2b802c700 tx=0x7fe2b802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 shutdown_connections 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db06be80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db059620 0x5628db0570a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2a97fa640 0 client.16053 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db048230 0x5628db056b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 --2- 172.21.15.87:0/945843464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db056620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 >> 172.21.15.87:0/945843464 conn(0x5628db041fb0 msgr2=0x5628db043d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 shutdown_connections 2024-04-22T13:33:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 1 -- 172.21.15.87:0/945843464 wait complete. 2024-04-22T13:33:03.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.963+0000 7fe2ca987740 20 client.16053 trim_cache size 0 max 16384 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 Processor -- start 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 -- start start 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db058d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db040e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01ae140 con 0x5628db048230 2024-04-22T13:33:03.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01aba10 con 0x5628db047880 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2ca987740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01a9880 con 0x5628db059620 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38512/0 (socket says 172.21.15.87:38512) 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2c5331640 1 -- 172.21.15.87:0/3974080985 learned_addr learned my addr 172.21.15.87:0/3974080985 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db058d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.997+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db040e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 -- 172.21.15.87:0/3974080985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 msgr2=0x5628db040e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db040e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 -- 172.21.15.87:0/3974080985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 msgr2=0x5628db058d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:03.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db058d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 -- 172.21.15.87:0/3974080985 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628db0555d0 con 0x5628db048230 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db040e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00058c0 tx=0x7fe2b0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31cf15d95e36271 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2ab7fe640 1 -- 172.21.15.87:0/3974080985 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0022070 con 0x5628db048230 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2ab7fe640 1 -- 172.21.15.87:0/3974080985 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2b0004250 con 0x5628db048230 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.998+0000 7fe2ab7fe640 1 -- 172.21.15.87:0/3974080985 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b001d750 con 0x5628db048230 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- 172.21.15.87:0/3974080985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 msgr2=0x5628db065450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00058c0 tx=0x7fe2b0005f90 comp rx=0 tx=0).stop 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- 172.21.15.87:0/3974080985 shutdown_connections 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db059620 0x5628db040e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db065450 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3974080985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db047880 0x5628db058d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:03.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- 172.21.15.87:0/3974080985 >> 172.21.15.87:0/3974080985 conn(0x5628db041fb0 msgr2=0x5628db040780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:03.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- 172.21.15.87:0/3974080985 shutdown_connections 2024-04-22T13:33:03.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- 172.21.15.87:0/3974080985 wait complete. 2024-04-22T13:33:03.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 -- start start 2024-04-22T13:33:03.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:03.999+0000 7fe2ca987740 1 Processor -- start 2024-04-22T13:33:04.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db05db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db059620 0x5628db0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01a9ed0 con 0x5628db048230 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01ab340 con 0x5628db059620 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38522/0 (socket says 172.21.15.87:38522) 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 -- 172.21.15.87:0/3936904561 learned_addr learned my addr 172.21.15.87:0/3936904561 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c01aa5a0 con 0x5628db047880 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db05db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2ca987740 10 client.0 _ll_register_callbacks cb 0x7fe2b8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c6333640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db059620 0x5628db0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 -- 172.21.15.87:0/3936904561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 msgr2=0x5628db05db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db05db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 -- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db059620 msgr2=0x5628db0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db059620 0x5628db0cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.000+0000 7fe2c5331640 1 -- 172.21.15.87:0/3936904561 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628db0cf620 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2c5331640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0000c00 tx=0x7fe2b0003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8daa8775b69ea552 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0022070 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe2b0005030 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2b0004250 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe2b0005270 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2ca987740 10 client.16059 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5628db0d0960 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.001+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2b0004250 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe2b001d4d0 con 0x5628db048230 2024-04-22T13:33:04.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 10 client.16059 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 20 client.16059 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db0d3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 4 client.16059 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5628dafb3790 con 0x5628db0699c0 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db0d3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2a97fa640 10 client.16059 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.002+0000 7fe2c5b32640 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db0d3070 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ee0d57d1a3ea8d server_cookie=2b2a3db517ca24e5 in_seq=0 out_seq=0 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.003+0000 7fe2a97fa640 1 -- 172.21.15.87:0/3936904561 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5628dafb3790 con 0x5628db0699c0 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.003+0000 7fe2a97fa640 10 client.16059 handle_command_reply: tid=0 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 client.16059 shutdown 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 msgr2=0x5628db05e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0000c00 tx=0x7fe2b0003ae0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 shutdown_connections 2024-04-22T13:33:04.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5628db0699c0 0x5628db0d3070 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628db059620 0x5628db0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2a97fa640 0 client.16059 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5628db048230 0x5628db05e0c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 --2- 172.21.15.87:0/3936904561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628db047880 0x5628db05db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 >> 172.21.15.87:0/3936904561 conn(0x5628db041fb0 msgr2=0x5628db041010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 shutdown_connections 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 1 -- 172.21.15.87:0/3936904561 wait complete. 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.004+0000 7fe2ca987740 20 client.16059 trim_cache size 0 max 16384 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:04.005 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:33:04.006 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "57fe0af0-2b6b-4f8a-847d-069c83dce4dd", 2024-04-22T13:33:04.006 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:33:04.006 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 msgr2=0x7fe2ac0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 0x7fe2ac0a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8004af0 tx=0x7fe2b80323f0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 msgr2=0x7fe2c019e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 0x7fe2c019e5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc00d6d0 tx=0x7fe2bc00db90 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 shutdown_connections 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2ac0a3ed0 0x7fe2ac0a6390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2c01986b0 0x7fe2c0198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2c01072e0 0x7fe2c019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 --2- 172.21.15.87:0/3776121348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c0102210 0x7fe2c019e5a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 >> 172.21.15.87:0/3776121348 conn(0x7fe2c00fdab0 msgr2=0x7fe2c00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 shutdown_connections 2024-04-22T13:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.005+0000 7fe2c7d96640 1 -- 172.21.15.87:0/3776121348 wait complete. 2024-04-22T13:33:04.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:33:04.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.157+0000 7fdfe4515640 1 -- 172.21.15.87:0/2971469295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc101a60 msgr2=0x7fdfdc105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.157+0000 7fdfe4515640 1 --2- 172.21.15.87:0/2971469295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc101a60 0x7fdfdc105aa0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009900 tx=0x7fdfcc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 -- 172.21.15.87:0/2971469295 shutdown_connections 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 --2- 172.21.15.87:0/2971469295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc106180 0x7fdfdc10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 --2- 172.21.15.87:0/2971469295 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc101a60 0x7fdfdc105aa0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 --2- 172.21.15.87:0/2971469295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 -- 172.21.15.87:0/2971469295 >> 172.21.15.87:0/2971469295 conn(0x7fdfdc0771d0 msgr2=0x7fdfdc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 -- 172.21.15.87:0/2971469295 shutdown_connections 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 -- 172.21.15.87:0/2971469295 wait complete. 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 Processor -- start 2024-04-22T13:33:04.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.158+0000 7fdfe4515640 1 -- start start 2024-04-22T13:33:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc106180 0x7fdfdc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc112200 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc112080 con 0x7fdfdc1010b0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe4515640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc112380 con 0x7fdfdc106180 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38538/0 (socket says 172.21.15.87:38538) 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 -- 172.21.15.87:0/3665847797 learned_addr learned my addr 172.21.15.87:0/3665847797 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc106180 0x7fdfdc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 -- 172.21.15.87:0/3665847797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc106180 msgr2=0x7fdfdc198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc106180 0x7fdfdc198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 -- 172.21.15.87:0/3665847797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 msgr2=0x7fdfdc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 -- 172.21.15.87:0/3665847797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc19dab0 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.159+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009fd0 tx=0x7fdfd400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.160+0000 7fdfd37fe640 1 -- 172.21.15.87:0/3665847797 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4017070 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.160+0000 7fdfd37fe640 1 -- 172.21.15.87:0/3665847797 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfd4003e20 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.160+0000 7fdfd37fe640 1 -- 172.21.15.87:0/3665847797 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4010460 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.160+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfdc19dda0 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.160+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfdc1a5510 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.162+0000 7fdfd37fe640 1 -- 172.21.15.87:0/3665847797 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fdfd4003970 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.163+0000 7fdfd37fe640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 0x7fdfc80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.163+0000 7fdfd37fe640 1 -- 172.21.15.87:0/3665847797 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdfd40d0730 con 0x7fdfdc19cfc0 2024-04-22T13:33:04.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.163+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 0x7fdfc80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 Processor -- start 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 0x7fdfc80a6410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd8005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 -- start start 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a7760 0x55588c2a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55588c2121e0 con 0x55588c2b9c10 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55588c1af390 con 0x55588c2a8120 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55588c20e1a0 con 0x55588c2a7760 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38540/0 (socket says 172.21.15.87:38540) 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe22b1640 1 -- 172.21.15.87:0/3695144868 learned_addr learned my addr 172.21.15.87:0/3695144868 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.164+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a7760 0x55588c2a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 -- 172.21.15.87:0/3695144868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a7760 msgr2=0x55588c2a7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a7760 0x55588c2a7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 -- 172.21.15.87:0/3695144868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 msgr2=0x55588c2c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 -- 172.21.15.87:0/3695144868 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588bfee7d0 con 0x55588c2b9c10 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8004160 tx=0x7fdfd8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14708bc58046325 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfd0bfe640 1 -- 172.21.15.87:0/3695144868 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8003e30 con 0x55588c2b9c10 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfd0bfe640 1 -- 172.21.15.87:0/3695144868 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfd8025070 con 0x55588c2b9c10 2024-04-22T13:33:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.165+0000 7fdfd0bfe640 1 -- 172.21.15.87:0/3695144868 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8020760 con 0x55588c2b9c10 2024-04-22T13:33:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 -- 172.21.15.87:0/3695144868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 msgr2=0x55588c2b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8004160 tx=0x7fdfd8014000 comp rx=0 tx=0).stop 2024-04-22T13:33:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 -- 172.21.15.87:0/3695144868 shutdown_connections 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2c1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a7760 0x55588c2a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/3695144868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2b9c10 0x55588c2b9ff0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 -- 172.21.15.87:0/3695144868 >> 172.21.15.87:0/3695144868 conn(0x55588c2aa220 msgr2=0x55588c2aa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 -- 172.21.15.87:0/3695144868 shutdown_connections 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.166+0000 7fdfe71e6740 1 -- 172.21.15.87:0/3695144868 wait complete. 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 -- start start 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 Processor -- start 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2bf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2bfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55588c2c2140 con 0x55588c2a7760 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55588c1cb2f0 con 0x55588c2b9c10 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55588c2c22c0 con 0x55588c2a8120 2024-04-22T13:33:04.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38552/0 (socket says 172.21.15.87:38552) 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2691282933 learned_addr learned my addr 172.21.15.87:0/2691282933 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.167+0000 7fdfe71e6740 10 client.0 _ll_register_callbacks cb 0x55588c1d8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2bf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2bfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2691282933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 msgr2=0x55588c2bf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2bf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 msgr2=0x55588c2bfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2bfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2691282933 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588bf243f0 con 0x55588c2a7760 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2bfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8011c40 tx=0x7fdfd8003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494f5d1ccd97d188 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd80129a0 con 0x55588c2a7760 2024-04-22T13:33:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdfd80204a0 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.168+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd8025070 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdfd80206e0 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 10 client.16071 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55588bee40f0 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfd8016070 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdfd803d020 con 0x55588c2a7760 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 10 client.16071 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 20 client.16071 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c2d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 4 client.16071 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.169+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdfd8025070 con 0x55588c2cf280 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.170+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c2d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.170+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c2d1670 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4440e901f200ad30 server_cookie=f5fc36352aa8da6c in_seq=0 out_seq=0 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.170+0000 7fdfc5ffb640 10 client.16071 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2691282933 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdfd8025070 con 0x55588c2cf280 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfc5ffb640 10 client.16071 handle_command_reply: tid=0 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 client.16071 shutdown 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 msgr2=0x55588c2bf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8011c40 tx=0x7fdfd8003e60 comp rx=0 tx=0).stop 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 shutdown_connections 2024-04-22T13:33:04.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c2d1670 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2bfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfc5ffb640 0 client.16071 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2bf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2691282933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2bf010 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 >> 172.21.15.87:0/2691282933 conn(0x55588c2aa220 msgr2=0x55588c2a9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 shutdown_connections 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2691282933 wait complete. 2024-04-22T13:33:04.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.171+0000 7fdfe71e6740 20 client.16071 trim_cache size 0 max 16384 2024-04-22T13:33:04.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.200+0000 7fdfe71e6740 1 Processor -- start 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.200+0000 7fdfe71e6740 1 -- start start 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.200+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.200+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.200+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38556/0 (socket says 172.21.15.87:38556) 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2424475778 learned_addr learned my addr 172.21.15.87:0/2424475778 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ae080 con 0x55588c2a7760 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ab950 con 0x55588c2b9c10 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1a97e0 con 0x55588c2a8120 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2424475778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 msgr2=0x55588c2cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2424475778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 msgr2=0x55588c2b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2424475778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588c2b7bc0 con 0x55588c2a7760 2024-04-22T13:33:04.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.201+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800a010 tx=0x7fdfd80097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30110237990dbf25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.202+0000 7fdfc7fff640 1 -- 172.21.15.87:0/2424475778 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8044070 con 0x55588c2a7760 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.202+0000 7fdfc7fff640 1 -- 172.21.15.87:0/2424475778 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfd8016070 con 0x55588c2a7760 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.202+0000 7fdfc7fff640 1 -- 172.21.15.87:0/2424475778 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd80129e0 con 0x55588c2a7760 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 msgr2=0x55588c2c1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800a010 tx=0x7fdfd80097b0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 shutdown_connections 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2b9c10 0x55588c2b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2a8120 0x55588c2cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2424475778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2c1740 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 >> 172.21.15.87:0/2424475778 conn(0x55588c2aa220 msgr2=0x55588c2befd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 shutdown_connections 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2424475778 wait complete. 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 -- start start 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.203+0000 7fdfe71e6740 1 Processor -- start 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38560/0 (socket says 172.21.15.87:38560) 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2411469271 learned_addr learned my addr 172.21.15.87:0/2411469271 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2d6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2b9c10 0x55588c334650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1a9e30 con 0x55588c2a7760 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1ab280 con 0x55588c2a8120 2024-04-22T13:33:04.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdfdc1aa4e0 con 0x55588c2b9c10 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe2ab2640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2b9c10 0x55588c334650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2d6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2411469271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2b9c10 msgr2=0x55588c334650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2b9c10 0x55588c334650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 msgr2=0x55588c2d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2d6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.204+0000 7fdfe22b1640 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588c334b90 con 0x55588c2a7760 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfe22b1640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005b80 tx=0x7fdfd802a8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f07248fffc89fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfe71e6740 10 client.0 _ll_register_callbacks cb 0x7fdfd8004030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd8016070 con 0x55588c2a7760 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfc5ffb640 10 client.16077 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2411469271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd804e050 con 0x55588c2a7760 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2411469271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdfd80129a0 con 0x55588c2a7760 2024-04-22T13:33:04.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2411469271 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdfd8044070 con 0x55588c2a7760 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfe71e6740 10 client.16077 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.205+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55588c3358f0 con 0x55588c2a7760 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2411469271 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdfd803d070 con 0x55588c2a7760 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 10 client.16077 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 20 client.16077 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c337eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 4 client.16077 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55588c21fc00 con 0x55588c2cf280 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.206+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c337eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfc5ffb640 10 client.16077 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe1ab0640 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c337eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f569a0127d758ab7 server_cookie=9544bf6d53d6c2cf in_seq=0 out_seq=0 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfc5ffb640 1 -- 172.21.15.87:0/2411469271 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55588c21fc00 con 0x55588c2cf280 2024-04-22T13:33:04.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfc5ffb640 10 client.16077 handle_command_reply: tid=0 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 client.16077 shutdown 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 msgr2=0x55588c2d6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005b80 tx=0x7fdfd802a8d0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 shutdown_connections 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55588c2cf280 0x55588c337eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55588c2b9c10 0x55588c334650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55588c2a8120 0x55588c2d6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 --2- 172.21.15.87:0/2411469271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55588c2a7760 0x55588c2d6620 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 >> 172.21.15.87:0/2411469271 conn(0x55588c2aa220 msgr2=0x55588c2bf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfc5ffb640 0 client.16077 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 shutdown_connections 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 1 -- 172.21.15.87:0/2411469271 wait complete. 2024-04-22T13:33:04.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.207+0000 7fdfe71e6740 20 client.16077 trim_cache size 0 max 16384 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "57fe0af0-2b6b-4f8a-847d-069c83dce4dd": { 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "57fe0af0-2b6b-4f8a-847d-069c83dce4dd", 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.208 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 msgr2=0x7fdfc80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 0x7fdfc80a6410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd8005f90 comp rx=0 tx=0).stop 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 msgr2=0x7fdfdc19d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009fd0 tx=0x7fdfd400b260 comp rx=0 tx=0).stop 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 shutdown_connections 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdfc80a3f50 0x7fdfc80a6410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfdc19cfc0 0x7fdfdc19d470 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfdc106180 0x7fdfdc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 --2- 172.21.15.87:0/3665847797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfdc1010b0 0x7fdfdc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 >> 172.21.15.87:0/3665847797 conn(0x7fdfdc0771d0 msgr2=0x7fdfdc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 shutdown_connections 2024-04-22T13:33:04.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.211+0000 7fdfe4515640 1 -- 172.21.15.87:0/3665847797 wait complete. 2024-04-22T13:33:04.222 INFO:tasks.cephfs.filesystem:scrub status for tag:57fe0af0-2b6b-4f8a-847d-069c83dce4dd - {'path': '/', 'tag': '57fe0af0-2b6b-4f8a-847d-069c83dce4dd', 'options': 'recursive,force'} 2024-04-22T13:33:04.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.255+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8edc108b00 con 0x7f8edc1005a0 2024-04-22T13:33:04.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.256+0000 7f8ecb7fe640 1 -- 172.21.15.87:0/3089524756 <== mon.2 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f8ecc09d310 con 0x7f8edc1005a0 2024-04-22T13:33:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:04.252 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:04.252 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 msgr2=0x7f8ec40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 0x7f8ec40a6390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000c00 tx=0x7f8ed00058d0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 msgr2=0x7f8edc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 0x7f8edc198710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9c0 tx=0x7f8ecc00ee80 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 shutdown_connections 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ec40a3ed0 0x7f8ec40a6390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8edc105560 0x7f8edc19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc100f60 0x7f8edc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 --2- 172.21.15.87:0/3089524756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8edc1005a0 0x7f8edc198710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 >> 172.21.15.87:0/3089524756 conn(0x7f8edc0774f0 msgr2=0x7f8edc103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 shutdown_connections 2024-04-22T13:33:04.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.259+0000 7f8ee0e3f640 1 -- 172.21.15.87:0/3089524756 wait complete. 2024-04-22T13:33:04.270 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 83.55 --await' 2024-04-22T13:33:04.270 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 83.55 --await 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 -- 172.21.15.87:0/1741225230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 msgr2=0x7f78a4100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 --2- 172.21.15.87:0/1741225230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a4100a30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7894009900 tx=0x7f789402bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 -- 172.21.15.87:0/1741225230 shutdown_connections 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 --2- 172.21.15.87:0/1741225230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4100f70 0x7f78a410d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 --2- 172.21.15.87:0/1741225230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a4100a30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 --2- 172.21.15.87:0/1741225230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4106670 0x7f78a4106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 -- 172.21.15.87:0/1741225230 >> 172.21.15.87:0/1741225230 conn(0x7f78a40773f0 msgr2=0x7f78a40fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 -- 172.21.15.87:0/1741225230 shutdown_connections 2024-04-22T13:33:04.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.423+0000 7f78abad2640 1 -- 172.21.15.87:0/1741225230 wait complete. 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 Processor -- start 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 -- start start 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 0x7f78a4198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4106670 0x7f78a419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78a410ffe0 con 0x7f78a41005b0 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78a410fe60 con 0x7f78a4100f70 2024-04-22T13:33:04.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78abad2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4110160 con 0x7f78a4106670 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.424+0000 7f78a986e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38570/0 (socket says 172.21.15.87:38570) 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 -- 172.21.15.87:0/2946814888 learned_addr learned my addr 172.21.15.87:0/2946814888 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a906d640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 0x7f78a4198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78aa06f640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4106670 0x7f78a419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 -- 172.21.15.87:0/2946814888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4106670 msgr2=0x7f78a419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4106670 0x7f78a419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 -- 172.21.15.87:0/2946814888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 msgr2=0x7f78a4198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 0x7f78a4198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 -- 172.21.15.87:0/2946814888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a419d6c0 con 0x7f78a41005b0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a906d640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 0x7f78a4198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78a986e640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f78a000ea20 tx=0x7f78a000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f789affd640 1 -- 172.21.15.87:0/2946814888 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a000cd40 con 0x7f78a41005b0 2024-04-22T13:33:04.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f789affd640 1 -- 172.21.15.87:0/2946814888 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a00168c0 con 0x7f78a41005b0 2024-04-22T13:33:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a419d9b0 con 0x7f78a41005b0 2024-04-22T13:33:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f789affd640 1 -- 172.21.15.87:0/2946814888 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a000b840 con 0x7f78a41005b0 2024-04-22T13:33:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.425+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a41a5240 con 0x7f78a41005b0 2024-04-22T13:33:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 Processor -- start 2024-04-22T13:33:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 -- start start 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffba980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffbd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5605fff120b0 con 0x5605fffb22e0 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605ffec3680 con 0x5605fffbaec0 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f78ae79e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5605ffecfc60 con 0x5605fffb19c0 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.429+0000 7f789affd640 1 -- 172.21.15.87:0/2946814888 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f78a001d020 con 0x7f78a41005b0 2024-04-22T13:33:04.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78a906d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffbd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:40882/0 (socket says 172.21.15.87:40882) 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 -- 172.21.15.87:0/1552805321 learned_addr learned my addr 172.21.15.87:0/1552805321 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78a906d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffbd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60998/0 (socket says 172.21.15.87:60998) 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78a986e640 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffba980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 -- 172.21.15.87:0/1552805321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 msgr2=0x5605fffbd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffbd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 -- 172.21.15.87:0/1552805321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 msgr2=0x5605fffba980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffba980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 -- 172.21.15.87:0/1552805321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ffcf4c90 con 0x5605fffb19c0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f789c009fd0 tx=0x7f789c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9942e46008358c80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f789affd640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 0x7f78900a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f788ffff640 1 -- 172.21.15.87:0/1552805321 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c004d20 con 0x5605fffb19c0 2024-04-22T13:33:04.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.430+0000 7f788ffff640 1 -- 172.21.15.87:0/1552805321 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c004050 con 0x5605fffb19c0 2024-04-22T13:33:04.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f788ffff640 1 -- 172.21.15.87:0/1552805321 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c005240 con 0x5605fffb19c0 2024-04-22T13:33:04.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f789affd640 1 -- 172.21.15.87:0/2946814888 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78a00d05e0 con 0x7f78a41005b0 2024-04-22T13:33:04.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78a986e640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 0x7f78900a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 -- 172.21.15.87:0/1552805321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 msgr2=0x5605fffb1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f789c009fd0 tx=0x7f789c0118d0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78a986e640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 0x7f78900a64e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a00cdbc0 tx=0x7f78a000a830 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 -- 172.21.15.87:0/1552805321 shutdown_connections 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffbd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffba980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1552805321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb19c0 0x5605fffb1da0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.431+0000 7f78ae79e740 1 -- 172.21.15.87:0/1552805321 >> 172.21.15.87:0/1552805321 conn(0x5605fffac0f0 msgr2=0x5605fffac4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 -- 172.21.15.87:0/1552805321 shutdown_connections 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 -- 172.21.15.87:0/1552805321 wait complete. 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 -- start start 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 Processor -- start 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffb7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.432+0000 7f78ae79e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5605fff15400 con 0x5605fffbaec0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78ae79e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605ffed1840 con 0x5605fffb19c0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78ae79e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5605fff1a720 con 0x5605fffb22e0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78ae79e740 10 client.0 _ll_register_callbacks cb 0x5605fff22ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38592/0 (socket says 172.21.15.87:38592) 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:40884/0 (socket says 172.21.15.87:40884) 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78aa06f640 1 -- 172.21.15.87:0/1173142757 learned_addr learned my addr 172.21.15.87:0/1173142757 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffb7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 -- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 msgr2=0x5605fffb7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffb7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 -- 172.21.15.87:0/1173142757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 msgr2=0x5605fffb8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 -- 172.21.15.87:0/1173142757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ffd55420 con 0x5605fffb22e0 2024-04-22T13:33:04.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78a906d640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7894009230 tx=0x7f7894004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c02e5f754843cc70 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f788d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894005230 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c004050 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.433+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78940053d0 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f789402cbf0 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 10 client.25462 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5605ffbe8580 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c004050 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7894003890 con 0x5605fffb22e0 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 10 client.25462 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 10 client.25462 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 20 client.25462 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:04.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5605fffcf2e0 0x5605fffd16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 4 client.25462 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605fff23b10 con 0x5605fffcf2e0 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.434+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5605fffcf2e0 0x5605fffd16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.435+0000 7f78aa06f640 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5605fffcf2e0 0x5605fffd16d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a2ffc5b8025f7e server_cookie=2276f438d55c287b in_seq=0 out_seq=0 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.435+0000 7f788d7fa640 10 client.25462 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.435+0000 7f788d7fa640 1 -- 172.21.15.87:0/1173142757 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605fff23b10 con 0x5605fffcf2e0 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.435+0000 7f788d7fa640 10 client.25462 handle_command_reply: tid=0 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.435+0000 7f78ae79e740 1 client.25462 shutdown 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 msgr2=0x5605fffb8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7894009230 tx=0x7f7894004270 comp rx=0 tx=0).stop 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 shutdown_connections 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5605fffcf2e0 0x5605fffd16d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffbaec0 0x5605fffb8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffb8500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 --2- 172.21.15.87:0/1173142757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffb7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f788d7fa640 0 client.25462 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 >> 172.21.15.87:0/1173142757 conn(0x5605fffac0f0 msgr2=0x5605fffade40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 shutdown_connections 2024-04-22T13:33:04.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 1 -- 172.21.15.87:0/1173142757 wait complete. 2024-04-22T13:33:04.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.436+0000 7f78ae79e740 20 client.25462 trim_cache size 0 max 16384 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 Processor -- start 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 -- start start 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffab050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffd6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41ab980 con 0x5605fffb19c0 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41a97d0 con 0x5605fffbaec0 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78ae79e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41ab2b0 con 0x5605fffb22e0 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78aa06f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38596/0 (socket says 172.21.15.87:38596) 2024-04-22T13:33:04.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78aa06f640 1 -- 172.21.15.87:0/703733304 learned_addr learned my addr 172.21.15.87:0/703733304 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78a906d640 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.453+0000 7f78aa06f640 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffd6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 -- 172.21.15.87:0/703733304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 msgr2=0x5605fffab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 -- 172.21.15.87:0/703733304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 msgr2=0x5605fffd6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffd6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 -- 172.21.15.87:0/703733304 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605fffadb90 con 0x5605fffb19c0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f78aa06f640 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f789c009de0 tx=0x7f789c0047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a65d2c01edf64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f788f7fe640 1 -- 172.21.15.87:0/703733304 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c034070 con 0x5605fffb19c0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f788f7fe640 1 -- 172.21.15.87:0/703733304 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c02b030 con 0x5605fffb19c0 2024-04-22T13:33:04.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.454+0000 7f788f7fe640 1 -- 172.21.15.87:0/703733304 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c02e160 con 0x5605fffb19c0 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- 172.21.15.87:0/703733304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 msgr2=0x5605fffb3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f789c009de0 tx=0x7f789c0047f0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- 172.21.15.87:0/703733304 shutdown_connections 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffbaec0 0x5605fffd6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffb22e0 0x5605fffab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 --2- 172.21.15.87:0/703733304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb19c0 0x5605fffb3240 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- 172.21.15.87:0/703733304 >> 172.21.15.87:0/703733304 conn(0x5605fffac0f0 msgr2=0x5605fffaa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- 172.21.15.87:0/703733304 shutdown_connections 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- 172.21.15.87:0/703733304 wait complete. 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.455+0000 7f78ae79e740 1 -- start start 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 Processor -- start 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffc33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffbaec0 0x56060002e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41aa510 con 0x5605fffb22e0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41a9e40 con 0x5605fffb19c0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78ae79e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41aabe0 con 0x5605fffbaec0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78a906d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78a906d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38612/0 (socket says 172.21.15.87:38612) 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.456+0000 7f78a906d640 1 -- 172.21.15.87:0/708284679 learned_addr learned my addr 172.21.15.87:0/708284679 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78ae79e740 10 client.0 _ll_register_callbacks cb 0x7f789c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78aa06f640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffc33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78aa06f640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffbaec0 0x56060002e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 -- 172.21.15.87:0/708284679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffbaec0 msgr2=0x56060002e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffbaec0 0x56060002e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 -- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 msgr2=0x5605fffc33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffc33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 -- 172.21.15.87:0/708284679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605fffb2110 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f788d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78a906d640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7894006010 tx=0x7f7894003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc68ce0b17be4047 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78940055d0 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f78940035f0 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c02b030 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.457+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7894009520 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 10 client.16095 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560600035c80 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c02b030 con 0x5605fffb22e0 2024-04-22T13:33:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f789405c030 con 0x5605fffb22e0 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 10 client.16095 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 10 client.16095 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 20 client.16095 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5606000364f0 0x560600038910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 4 client.16095 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.55, "await": true}] 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.55, "await": true}) -- 0x5605fff1d980 con 0x5606000364f0 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.458+0000 7f78aa06f640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5606000364f0 0x560600038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.459+0000 7f788d7fa640 10 client.16095 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.459+0000 7f78aa06f640 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5606000364f0 0x560600038910 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5d361d7ba2e9823 server_cookie=e210d27d74968927 in_seq=0 out_seq=0 2024-04-22T13:33:04.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f788d7fa640 1 -- 172.21.15.87:0/708284679 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5605fff1d980 con 0x5606000364f0 2024-04-22T13:33:04.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f788d7fa640 10 client.16095 handle_command_reply: tid=0 2024-04-22T13:33:04.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 client.16095 shutdown 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 msgr2=0x5605fffc3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7894006010 tx=0x7f7894003e80 comp rx=0 tx=0).stop 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 shutdown_connections 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5606000364f0 0x560600038910 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f788d7fa640 0 client.16095 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605fffbaec0 0x56060002e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605fffb22e0 0x5605fffc3920 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 --2- 172.21.15.87:0/708284679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605fffb19c0 0x5605fffc33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.469+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 >> 172.21.15.87:0/708284679 conn(0x5605fffac0f0 msgr2=0x5605fffab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 shutdown_connections 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78ae79e740 1 -- 172.21.15.87:0/708284679 wait complete. 2024-04-22T13:33:04.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78ae79e740 20 client.16095 trim_cache size 0 max 16384 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 6, 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "86afa3a0": { 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 6, 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:33:04.467 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 83.5, 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:04.468 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 msgr2=0x7f78900a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 0x7f78900a64e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a00cdbc0 tx=0x7f78a000a830 comp rx=0 tx=0).stop 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 msgr2=0x7f78a41986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f78a000ea20 tx=0x7f78a000eee0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 shutdown_connections 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78900a4020 0x7f78900a64e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4106670 0x7f78a419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78a4100f70 0x7f78a4198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 --2- 172.21.15.87:0/2946814888 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78a41005b0 0x7f78a41986d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:04.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 >> 172.21.15.87:0/2946814888 conn(0x7f78a40773f0 msgr2=0x7f78a40fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:04.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 shutdown_connections 2024-04-22T13:33:04.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:04.470+0000 7f78abad2640 1 -- 172.21.15.87:0/2946814888 wait complete. 2024-04-22T13:33:04.482 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 86afa3a0, quiesce took 0.8 seconds. Will release after: 23.55 2024-04-22T13:33:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:04 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3089524756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:04.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:04 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3089524756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:04.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:04 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3089524756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:05 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:33:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:05 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:05 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:33:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:05 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:05 smithi120 ceph-mon[31101]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-22T13:33:05.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:05 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:33:05.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:05 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:05 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:33:05.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:05 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:05 smithi132 ceph-mon[31531]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-22T13:33:05.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:05 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:33:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:05 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:05 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:33:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:05 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:05 smithi087 ceph-mon[27999]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-22T13:33:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.247+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1679077673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 msgr2=0x7fd96410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.247+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1679077673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd96410dd40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009ca0 tx=0x7fd95c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1679077673 shutdown_connections 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1679077673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd96410dd40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1679077673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964101aa0 0x7fd964105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1679077673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 0x7fd9641014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1679077673 >> 172.21.15.87:0/1679077673 conn(0x7fd964077fd0 msgr2=0x7fd9640783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1679077673 shutdown_connections 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1679077673 wait complete. 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 Processor -- start 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.248+0000 7fd96b7a4640 1 -- start start 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 0x7fd964197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96419be40 0x7fd96419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641110e0 con 0x7fd964106170 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd964110f60 con 0x7fd96419be40 2024-04-22T13:33:06.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd96b7a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd964111260 con 0x7fd9641010f0 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38626/0 (socket says 172.21.15.87:38626) 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 -- 172.21.15.87:0/1211223111 learned_addr learned my addr 172.21.15.87:0/1211223111 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd969540640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 0x7fd964197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd969d41640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96419be40 0x7fd96419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 -- 172.21.15.87:0/1211223111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 msgr2=0x7fd964197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 0x7fd964197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 -- 172.21.15.87:0/1211223111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96419be40 msgr2=0x7fd96419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96419be40 0x7fd96419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 -- 172.21.15.87:0/1211223111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96419c930 con 0x7fd964106170 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.249+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd954009fa0 tx=0x7fd95400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.250+0000 7fd95a7fc640 1 -- 172.21.15.87:0/1211223111 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540159e0 con 0x7fd964106170 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.250+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96419cc20 con 0x7fd964106170 2024-04-22T13:33:06.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.250+0000 7fd95a7fc640 1 -- 172.21.15.87:0/1211223111 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd95400cd30 con 0x7fd964106170 2024-04-22T13:33:06.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.250+0000 7fd95a7fc640 1 -- 172.21.15.87:0/1211223111 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540054e0 con 0x7fd964106170 2024-04-22T13:33:06.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.250+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd964069920 con 0x7fd964106170 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.252+0000 7fd95a7fc640 1 -- 172.21.15.87:0/1211223111 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd95400c880 con 0x7fd964106170 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd95a7fc640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 0x7fd9500a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd969540640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 0x7fd9500a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd95a7fc640 1 -- 172.21.15.87:0/1211223111 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd9540d4b90 con 0x7fd964106170 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd96e39a740 1 Processor -- start 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd96e39a740 1 -- start start 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd969540640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 0x7fd9500a6430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd960005fd0 tx=0x7fd960005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x561281806fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.253+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 0x5612818124d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd96e39a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56128176b8d0 con 0x5612818100e0 2024-04-22T13:33:06.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd96e39a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561281708e80 con 0x561281806b60 2024-04-22T13:33:06.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd96e39a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561281767890 con 0x5612818078d0 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x561281806fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:40920/0 (socket says 172.21.15.87:40920) 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969540640 1 -- 172.21.15.87:0/572778215 learned_addr learned my addr 172.21.15.87:0/572778215 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969d41640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 0x5612818124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 -- 172.21.15.87:0/572778215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 msgr2=0x561281806620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 -- 172.21.15.87:0/572778215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 msgr2=0x5612818124d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 0x5612818124d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 -- 172.21.15.87:0/572778215 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56128172a320 con 0x561281806b60 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969540640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd969d41640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 0x5612818124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.254+0000 7fd968d3f640 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x561281806fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9540d2110 tx=0x7fd9540c1110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42387d4c0c73667e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.255+0000 7fd94f7fe640 1 -- 172.21.15.87:0/572778215 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954023070 con 0x561281806b60 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.255+0000 7fd94f7fe640 1 -- 172.21.15.87:0/572778215 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9540a1060 con 0x561281806b60 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.255+0000 7fd94f7fe640 1 -- 172.21.15.87:0/572778215 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95408b0b0 con 0x561281806b60 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.255+0000 7fd96e39a740 1 -- 172.21.15.87:0/572778215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 msgr2=0x561281806fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.255+0000 7fd96e39a740 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x561281806fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9540d2110 tx=0x7fd9540c1110 comp rx=0 tx=0).stop 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 -- 172.21.15.87:0/572778215 shutdown_connections 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818100e0 0x5612818124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x561281806fe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 --2- 172.21.15.87:0/572778215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x561281806620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 -- 172.21.15.87:0/572778215 >> 172.21.15.87:0/572778215 conn(0x5612817cdc90 msgr2=0x561281801850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 -- 172.21.15.87:0/572778215 shutdown_connections 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 -- 172.21.15.87:0/572778215 wait complete. 2024-04-22T13:33:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 -- start start 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 Processor -- start 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.256+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x56128180ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x56128180d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd96e39a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561281812e10 con 0x561281806b60 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd96e39a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612816f3eb0 con 0x5612818100e0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd96e39a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561281812f90 con 0x5612818078d0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd96e39a740 10 client.0 _ll_register_callbacks cb 0x561281720b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd969540640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd969540640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38644/0 (socket says 172.21.15.87:38644) 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd969540640 1 -- 172.21.15.87:0/4157025183 learned_addr learned my addr 172.21.15.87:0/4157025183 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x56128180ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd969d41640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x56128180d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 -- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 msgr2=0x56128180d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x56128180d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 -- 172.21.15.87:0/4157025183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 msgr2=0x56128180fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 -- 172.21.15.87:0/4157025183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56128147e210 con 0x5612818078d0 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd969540640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd968d3f640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x56128180ceb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd9540a0b20 tx=0x7fd954015db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2a382c08f426e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.257+0000 7fd94cff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954023070 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9540a1060 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd95408b2c0 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd954086050 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 10 client.25480 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56128154a620 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9540a1060 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd954026060 con 0x5612818078d0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 10 client.25480 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 20 client.25480 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x5612818221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 4 client.25480 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56128177f880 con 0x56128181fdf0 2024-04-22T13:33:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.258+0000 7fd969540640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x5612818221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.259+0000 7fd969540640 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x5612818221e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a68901bc3f806c5b server_cookie=83cca1d04ff688dc in_seq=0 out_seq=0 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.259+0000 7fd94cff9640 10 client.25480 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd94cff9640 1 -- 172.21.15.87:0/4157025183 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56128177f880 con 0x56128181fdf0 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd94cff9640 10 client.25480 handle_command_reply: tid=0 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 client.25480 shutdown 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 msgr2=0x56128180ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x56128180ceb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd9540a0b20 tx=0x7fd954015db0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 shutdown_connections 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x5612818221e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x56128180d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd94cff9640 0 client.25480 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818078d0 0x56128180ceb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 --2- 172.21.15.87:0/4157025183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561281806b60 0x56128180fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 >> 172.21.15.87:0/4157025183 conn(0x5612817cdc90 msgr2=0x5612818035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 shutdown_connections 2024-04-22T13:33:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 1 -- 172.21.15.87:0/4157025183 wait complete. 2024-04-22T13:33:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.260+0000 7fd96e39a740 20 client.25480 trim_cache size 0 max 16384 2024-04-22T13:33:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 Processor -- start 2024-04-22T13:33:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 -- start start 2024-04-22T13:33:06.280 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x56128180f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 0x56128180b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641ae0d0 con 0x5612818078d0 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641ab9a0 con 0x561281806b60 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd96e39a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641a9810 con 0x5612818100e0 2024-04-22T13:33:06.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38650/0 (socket says 172.21.15.87:38650) 2024-04-22T13:33:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.275+0000 7fd968d3f640 1 -- 172.21.15.87:0/1468657226 learned_addr learned my addr 172.21.15.87:0/1468657226 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd969540640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x56128180f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd969d41640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 0x56128180b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 -- 172.21.15.87:0/1468657226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 msgr2=0x56128180b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 0x56128180b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 -- 172.21.15.87:0/1468657226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 msgr2=0x56128180f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x56128180f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 -- 172.21.15.87:0/1468657226 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612817fcf30 con 0x5612818078d0 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd969d41640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 0x56128180b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:06.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd968d3f640 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd9540d2200 tx=0x7fd954005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4d496b46ecbc66 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd94effd640 1 -- 172.21.15.87:0/1468657226 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954023050 con 0x5612818078d0 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd94effd640 1 -- 172.21.15.87:0/1468657226 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9540a0960 con 0x5612818078d0 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.276+0000 7fd94effd640 1 -- 172.21.15.87:0/1468657226 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540a95d0 con 0x5612818078d0 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- 172.21.15.87:0/1468657226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 msgr2=0x56128180fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd9540d2200 tx=0x7fd954005880 comp rx=0 tx=0).stop 2024-04-22T13:33:06.284 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- 172.21.15.87:0/1468657226 shutdown_connections 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612818100e0 0x56128180b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180fb20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 --2- 172.21.15.87:0/1468657226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561281806b60 0x56128180f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- 172.21.15.87:0/1468657226 >> 172.21.15.87:0/1468657226 conn(0x5612817cdc90 msgr2=0x561281808080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- 172.21.15.87:0/1468657226 shutdown_connections 2024-04-22T13:33:06.285 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- 172.21.15.87:0/1468657226 wait complete. 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.277+0000 7fd96e39a740 1 -- start start 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 Processor -- start 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 0x561281825750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x561281809120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641a9e60 con 0x5612818078d0 2024-04-22T13:33:06.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641ab2d0 con 0x5612818100e0 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd96e39a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641aa530 con 0x561281806b60 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd968d3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38656/0 (socket says 172.21.15.87:38656) 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd968d3f640 1 -- 172.21.15.87:0/239903777 learned_addr learned my addr 172.21.15.87:0/239903777 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd969d41640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x561281809120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.278+0000 7fd969540640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 0x561281825750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd96e39a740 10 client.0 _ll_register_callbacks cb 0x7fd95c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 -- 172.21.15.87:0/239903777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 msgr2=0x561281825750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 0x561281825750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 -- 172.21.15.87:0/239903777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 msgr2=0x56128180a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 -- 172.21.15.87:0/239903777 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561281809660 con 0x5612818100e0 2024-04-22T13:33:06.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969540640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 0x561281825750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd968d3f640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd969d41640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x561281809120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009bf0 tx=0x7fd95c032c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23bfac056a6ff71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd94cff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c046070 con 0x5612818100e0 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd95c0328d0 con 0x5612818100e0 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.279+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56128170b3d0 con 0x5612818100e0 2024-04-22T13:33:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd95c039040 con 0x5612818100e0 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 10 client.25457 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561281885590 con 0x5612818100e0 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56128170b3d0 con 0x5612818100e0 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd95c03f040 con 0x5612818100e0 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 10 client.25457 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 20 client.25457 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.280+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x561281887d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.281+0000 7fd96e39a740 4 client.25457 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:33:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.281+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612817792f0 con 0x56128181fdf0 2024-04-22T13:33:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.281+0000 7fd969540640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x561281887d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.281+0000 7fd969540640 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x561281887d10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=578e28020d91f6fc server_cookie=16f6602fb670b715 in_seq=0 out_seq=0 2024-04-22T13:33:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.281+0000 7fd94cff9640 10 client.25457 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd94cff9640 1 -- 172.21.15.87:0/239903777 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612817792f0 con 0x56128181fdf0 2024-04-22T13:33:06.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd94cff9640 10 client.25457 handle_command_reply: tid=0 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 client.25457 shutdown 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 msgr2=0x561281809120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x561281809120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009bf0 tx=0x7fd95c032c50 comp rx=0 tx=0).stop 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 shutdown_connections 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56128181fdf0 0x561281887d10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd94cff9640 0 client.25457 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.287+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612818100e0 0x561281809120 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612818078d0 0x56128180a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 1 --2- 172.21.15.87:0/239903777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561281806b60 0x561281825750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 >> 172.21.15.87:0/239903777 conn(0x5612817cdc90 msgr2=0x56128180b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 shutdown_connections 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 1 -- 172.21.15.87:0/239903777 wait complete. 2024-04-22T13:33:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96e39a740 20 client.25457 trim_cache size 0 max 16384 2024-04-22T13:33:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:33:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:33:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:33:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 msgr2=0x7fd9500a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 0x7fd9500a6430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd960005fd0 tx=0x7fd960005850 comp rx=0 tx=0).stop 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 msgr2=0x7fd964197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.288+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd954009fa0 tx=0x7fd95400edd0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 shutdown_connections 2024-04-22T13:33:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9500a3f70 0x7fd9500a6430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96419be40 0x7fd96419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd964106170 0x7fd964197ac0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 --2- 172.21.15.87:0/1211223111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9641010f0 0x7fd964197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 >> 172.21.15.87:0/1211223111 conn(0x7fd964077fd0 msgr2=0x7fd9640fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 shutdown_connections 2024-04-22T13:33:06.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:06.289+0000 7fd96b7a4640 1 -- 172.21.15.87:0/1211223111 wait complete. 2024-04-22T13:33:06.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:06 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:33:06.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:06 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:33:06.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:06 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:33:07.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:07 smithi120 ceph-mon[31101]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-22T13:33:07.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:07 smithi132 ceph-mon[31531]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-22T13:33:07.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:07 smithi087 ceph-mon[27999]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-22T13:33:09.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:09 smithi120 ceph-mon[31101]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:09.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:09 smithi132 ceph-mon[31531]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:09.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:09 smithi087 ceph-mon[27999]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:11.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:11 smithi120 ceph-mon[31101]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:11.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:11 smithi132 ceph-mon[31531]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:11.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:11 smithi087 ceph-mon[27999]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:13.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:13 smithi120 ceph-mon[31101]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:13.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:13 smithi132 ceph-mon[31531]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:13.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:13 smithi087 ceph-mon[27999]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T13:33:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:15 smithi120 ceph-mon[31101]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:15.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:15 smithi132 ceph-mon[31531]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:15.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:15 smithi087 ceph-mon[27999]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T13:33:17.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:17 smithi120 ceph-mon[31101]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-22T13:33:17.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:17 smithi132 ceph-mon[31531]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-22T13:33:17.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:17 smithi087 ceph-mon[27999]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- 172.21.15.87:0/2967894787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a01010f0 msgr2=0x7f63a01014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 --2- 172.21.15.87:0/2967894787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a01010f0 0x7f63a01014d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6390009900 tx=0x7f639002bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- 172.21.15.87:0/2967894787 shutdown_connections 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 --2- 172.21.15.87:0/2967894787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0106170 0x7f63a010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 --2- 172.21.15.87:0/2967894787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a0101aa0 0x7f63a0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 --2- 172.21.15.87:0/2967894787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a01010f0 0x7f63a01014d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- 172.21.15.87:0/2967894787 >> 172.21.15.87:0/2967894787 conn(0x7f63a0077fd0 msgr2=0x7f63a00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- 172.21.15.87:0/2967894787 shutdown_connections 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- 172.21.15.87:0/2967894787 wait complete. 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 Processor -- start 2024-04-22T13:33:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.424+0000 7f63a64d2640 1 -- start start 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 0x7f63a0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0198c40 0x7f63a01990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01110b0 con 0x7f63a0106170 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a0110f30 con 0x7f63a0198c40 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a64d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63a0111230 con 0x7f63a019eb80 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33630/0 (socket says 172.21.15.87:33630) 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 -- 172.21.15.87:0/1507315064 learned_addr learned my addr 172.21.15.87:0/1507315064 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0198c40 0x7f63a01990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639f7fe640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 0x7f63a0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 -- 172.21.15.87:0/1507315064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 msgr2=0x7f63a0198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 0x7f63a0198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 -- 172.21.15.87:0/1507315064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0198c40 msgr2=0x7f63a01990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0198c40 0x7f63a01990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639ffff640 1 -- 172.21.15.87:0/1507315064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a01a53e0 con 0x7f63a0106170 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.425+0000 7f639f7fe640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 0x7f63a0198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.426+0000 7f639ffff640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6390035000 tx=0x7f63900045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.426+0000 7f639d7fa640 1 -- 172.21.15.87:0/1507315064 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390043020 con 0x7f63a0106170 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.426+0000 7f639d7fa640 1 -- 172.21.15.87:0/1507315064 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f639002caf0 con 0x7f63a0106170 2024-04-22T13:33:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.426+0000 7f639d7fa640 1 -- 172.21.15.87:0/1507315064 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390003ab0 con 0x7f63a0106170 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.426+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a01a5670 con 0x7f63a0106170 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.427+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a01a5b40 con 0x7f63a0106170 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.429+0000 7f63a919e740 1 Processor -- start 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 -- start start 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639d7fa640 1 -- 172.21.15.87:0/1507315064 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f6390046020 con 0x7f63a0106170 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c94b6258d0 con 0x55c94b6c0b60 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c94b5c2e80 con 0x55c94b6c18d0 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a919e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c94b621890 con 0x55c94b6ca0e0 2024-04-22T13:33:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33634/0 (socket says 172.21.15.87:33634) 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 -- 172.21.15.87:0/2517639569 learned_addr learned my addr 172.21.15.87:0/2517639569 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639f7fe640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 -- 172.21.15.87:0/2517639569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 msgr2=0x55c94b6cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 -- 172.21.15.87:0/2517639569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 msgr2=0x55c94b6c0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.430+0000 7f639ffff640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f639ffff640 1 -- 172.21.15.87:0/2517639569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c94b5e4320 con 0x55c94b6c0b60 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f639d7fa640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 0x7f63880a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f639ffff640 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f63900efad0 tx=0x7f63900f8550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7bfebc59336e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 0x7f63880a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f639d7fa640 1 -- 172.21.15.87:0/1507315064 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f63900f24a0 con 0x7f63a0106170 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f63867fc640 1 -- 172.21.15.87:0/2517639569 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900ba620 con 0x55c94b6c0b60 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f63867fc640 1 -- 172.21.15.87:0/2517639569 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900bea80 con 0x55c94b6c0b60 2024-04-22T13:33:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.431+0000 7f63867fc640 1 -- 172.21.15.87:0/2517639569 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900ba620 con 0x55c94b6c0b60 2024-04-22T13:33:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- 172.21.15.87:0/2517639569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 msgr2=0x55c94b6c0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 0x7f63880a64b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6398005fd0 tx=0x7f6398009600 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f63900efad0 tx=0x7f63900f8550 comp rx=0 tx=0).stop 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- 172.21.15.87:0/2517639569 shutdown_connections 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c0fe0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 --2- 172.21.15.87:0/2517639569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- 172.21.15.87:0/2517639569 >> 172.21.15.87:0/2517639569 conn(0x55c94b687c90 msgr2=0x55c94b6bb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- 172.21.15.87:0/2517639569 shutdown_connections 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- 172.21.15.87:0/2517639569 wait complete. 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.432+0000 7f63a919e740 1 -- start start 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 Processor -- start 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c94b6258d0 con 0x55c94b6c0b60 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c94b5adeb0 con 0x55c94b6c18d0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c94b6d0fe0 con 0x55c94b6ca0e0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a919e740 10 client.0 _ll_register_callbacks cb 0x55c94b5dab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f639ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33640/0 (socket says 172.21.15.87:33640) 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f639ffff640 1 -- 172.21.15.87:0/796995089 learned_addr learned my addr 172.21.15.87:0/796995089 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f639f7fe640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.433+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 -- 172.21.15.87:0/796995089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 msgr2=0x55c94b6c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 -- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 msgr2=0x55c94b6c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 -- 172.21.15.87:0/796995089 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c94b338210 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639f7fe640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f639ffff640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f63900efc70 tx=0x7f63900f8f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3b83cc3ba682c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f637f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639002cca0 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6390003c50 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63900bea80 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f63900bdce0 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f63a919e740 10 client.16125 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.434+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c94b404620 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900bea80 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f63900be5e0 con 0x55c94b6c0b60 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 10 client.16125 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 10 client.16125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 20 client.16125 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6ddfd0 0x55c94b6e03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 4 client.16125 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c94b639880 con 0x55c94b6ddfd0 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.435+0000 7f639f7fe640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6ddfd0 0x55c94b6e03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.436+0000 7f637f7fe640 10 client.16125 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.436+0000 7f639f7fe640 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6ddfd0 0x55c94b6e03c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1053ecf02a819fc6 server_cookie=837d5a7c278bee43 in_seq=0 out_seq=0 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.436+0000 7f637f7fe640 1 -- 172.21.15.87:0/796995089 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c94b639880 con 0x55c94b6ddfd0 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.436+0000 7f637f7fe640 10 client.16125 handle_command_reply: tid=0 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.436+0000 7f63a919e740 1 client.16125 shutdown 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 msgr2=0x55c94b6c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f63900efc70 tx=0x7f63900f8f80 comp rx=0 tx=0).stop 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 shutdown_connections 2024-04-22T13:33:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6ddfd0 0x55c94b6e03c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c18d0 0x55c94b6c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 --2- 172.21.15.87:0/796995089 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9aa0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f637f7fe640 0 client.16125 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 >> 172.21.15.87:0/796995089 conn(0x55c94b687c90 msgr2=0x55c94b6bd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 shutdown_connections 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 1 -- 172.21.15.87:0/796995089 wait complete. 2024-04-22T13:33:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.437+0000 7f63a919e740 20 client.16125 trim_cache size 0 max 16384 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.466+0000 7f63a919e740 1 Processor -- start 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.466+0000 7f63a919e740 1 -- start start 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.466+0000 7f63a919e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.466+0000 7f63a919e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a919e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a919e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01ae5b0 con 0x55c94b6ca0e0 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a919e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01abe80 con 0x55c94b6c0b60 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a919e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a9cf0 con 0x55c94b6c18d0 2024-04-22T13:33:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:51546/0 (socket says 172.21.15.87:51546) 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a4a6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 -- 172.21.15.87:0/2601153074 learned_addr learned my addr 172.21.15.87:0/2601153074 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639f7fe640 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 -- 172.21.15.87:0/2601153074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 msgr2=0x55c94b6c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 -- 172.21.15.87:0/2601153074 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 msgr2=0x55c94b6e5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6e5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 -- 172.21.15.87:0/2601153074 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c94b6bfe30 con 0x55c94b6c0b60 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6e5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.467+0000 7f639ffff640 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63900bad50 tx=0x7f6390009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73655f3bbb0af9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.468+0000 7f6385ffb640 1 -- 172.21.15.87:0/2601153074 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900bdec0 con 0x55c94b6c0b60 2024-04-22T13:33:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.468+0000 7f6385ffb640 1 -- 172.21.15.87:0/2601153074 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900c6ff0 con 0x55c94b6c0b60 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.468+0000 7f6385ffb640 1 -- 172.21.15.87:0/2601153074 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900be150 con 0x55c94b6c0b60 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- 172.21.15.87:0/2601153074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 msgr2=0x55c94b6c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63900bad50 tx=0x7f6390009680 comp rx=0 tx=0).stop 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- 172.21.15.87:0/2601153074 shutdown_connections 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6e5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 --2- 172.21.15.87:0/2601153074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6c9560 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- 172.21.15.87:0/2601153074 >> 172.21.15.87:0/2601153074 conn(0x55c94b687c90 msgr2=0x55c94b6bb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- 172.21.15.87:0/2601153074 shutdown_connections 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- 172.21.15.87:0/2601153074 wait complete. 2024-04-22T13:33:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.469+0000 7f63a919e740 1 -- start start 2024-04-22T13:33:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 Processor -- start 2024-04-22T13:33:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6d6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01aa340 con 0x55c94b6ca0e0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01ab7b0 con 0x55c94b6c0b60 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01aaa10 con 0x55c94b6c18d0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f63a919e740 10 client.0 _ll_register_callbacks cb 0x7f6398009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.470+0000 7f639ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6d6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:51556/0 (socket says 172.21.15.87:51556) 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 -- 172.21.15.87:0/707903410 learned_addr learned my addr 172.21.15.87:0/707903410 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 -- 172.21.15.87:0/707903410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 msgr2=0x55c94b6d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6d6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 -- 172.21.15.87:0/707903410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 msgr2=0x55c94b6c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 -- 172.21.15.87:0/707903410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c94b6c48e0 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f63a4a6f640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f639ffff640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63900ba7e0 tx=0x7f63900f5b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a508fca24de2a34d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f637f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.471+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900be150 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63900c6ff0 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f63900dd250 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6390003e60 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f63a919e740 10 client.25475 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c94b6c48e0 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.472+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900c6ff0 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f63900c6b60 con 0x55c94b6c0b60 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 10 client.25475 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 10 client.25475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 20 client.25475 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6cb270 0x55c94b6cb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 4 client.25475 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c94b5c53d0 con 0x55c94b6cb270 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f639f7fe640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6cb270 0x55c94b6cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f639f7fe640 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6cb270 0x55c94b6cb650 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1f17d5e82f85c88 server_cookie=928e92b5d24bec7c in_seq=0 out_seq=0 2024-04-22T13:33:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.473+0000 7f637f7fe640 10 client.25475 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.477+0000 7f637f7fe640 1 -- 172.21.15.87:0/707903410 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c94b5c53d0 con 0x55c94b6cb270 2024-04-22T13:33:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.477+0000 7f637f7fe640 10 client.25475 handle_command_reply: tid=0 2024-04-22T13:33:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.477+0000 7f63a919e740 1 client.25475 shutdown 2024-04-22T13:33:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.477+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 msgr2=0x55c94b6d6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.477+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63900ba7e0 tx=0x7f63900f5b90 comp rx=0 tx=0).stop 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 shutdown_connections 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c94b6cb270 0x55c94b6cb650 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c94b6ca0e0 0x55c94b6c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c94b6c18d0 0x55c94b6d6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 --2- 172.21.15.87:0/707903410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c94b6c0b60 0x55c94b6d6200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 >> 172.21.15.87:0/707903410 conn(0x55c94b687c90 msgr2=0x55c94b6c2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f637f7fe640 0 client.25475 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 shutdown_connections 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 1 -- 172.21.15.87:0/707903410 wait complete. 2024-04-22T13:33:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a919e740 20 client.25475 trim_cache size 0 max 16384 2024-04-22T13:33:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:33:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:33:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:33:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 msgr2=0x7f63880a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 0x7f63880a64b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6398005fd0 tx=0x7f6398009600 comp rx=0 tx=0).stop 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 msgr2=0x7f63a019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.478+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6390035000 tx=0x7f63900045e0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 shutdown_connections 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63880a3ff0 0x7f63880a64b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0198c40 0x7f63a01990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63a019eb80 0x7f63a0198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.477 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 --2- 172.21.15.87:0/1507315064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63a0106170 0x7f63a019e640 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 >> 172.21.15.87:0/1507315064 conn(0x7f63a0077fd0 msgr2=0x7f63a0069aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 shutdown_connections 2024-04-22T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:19.479+0000 7f63a64d2640 1 -- 172.21.15.87:0/1507315064 wait complete. 2024-04-22T13:33:19.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:19 smithi120 ceph-mon[31101]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-22T13:33:19.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:19 smithi132 ceph-mon[31531]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-22T13:33:19.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:19 smithi087 ceph-mon[27999]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-22T13:33:21.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:21 smithi120 ceph-mon[31101]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:21.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:21 smithi132 ceph-mon[31531]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:21 smithi087 ceph-mon[27999]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:23.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:23 smithi120 ceph-mon[31101]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:23.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:23 smithi132 ceph-mon[31531]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:23.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:23 smithi087 ceph-mon[27999]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:33:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:25 smithi120 ceph-mon[31101]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:25 smithi120 ceph-mon[31101]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T13:33:25.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:25 smithi132 ceph-mon[31531]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:25.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:25 smithi132 ceph-mon[31531]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T13:33:25.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:25 smithi087 ceph-mon[27999]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:25.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:25 smithi087 ceph-mon[27999]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T13:33:27.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:27 smithi120 ceph-mon[31101]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:27.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:27 smithi132 ceph-mon[31531]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:27.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:27 smithi087 ceph-mon[27999]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:28.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:28.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 -- 172.21.15.87:0/3618591835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc105e30 msgr2=0x7fe3fc106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 --2- 172.21.15.87:0/3618591835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc105e30 0x7fe3fc106210 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009930 tx=0x7fe3f802bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 -- 172.21.15.87:0/3618591835 shutdown_connections 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 --2- 172.21.15.87:0/3618591835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc101b60 0x7fe3fc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 --2- 172.21.15.87:0/3618591835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc1011a0 0x7fe3fc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 --2- 172.21.15.87:0/3618591835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc105e30 0x7fe3fc106210 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 -- 172.21.15.87:0/3618591835 >> 172.21.15.87:0/3618591835 conn(0x7fe3fc077e30 msgr2=0x7fe3fc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.140+0000 7fe404406640 1 -- 172.21.15.87:0/3618591835 shutdown_connections 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.141+0000 7fe404406640 1 -- 172.21.15.87:0/3618591835 wait complete. 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.141+0000 7fe404406640 1 Processor -- start 2024-04-22T13:33:28.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.141+0000 7fe404406640 1 -- start start 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc1011a0 0x7fe3fc197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 0x7fe3fc19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1110e0 con 0x7fe3fc101b60 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc110f60 con 0x7fe3fc105e30 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc111260 con 0x7fe3fc1011a0 2024-04-22T13:33:28.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45360/0 (socket says 172.21.15.87:45360) 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 -- 172.21.15.87:0/2288858883 learned_addr learned my addr 172.21.15.87:0/2288858883 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4021a2640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc1011a0 0x7fe3fc197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4029a3640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 0x7fe3fc19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 -- 172.21.15.87:0/2288858883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc1011a0 msgr2=0x7fe3fc197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc1011a0 0x7fe3fc197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 -- 172.21.15.87:0/2288858883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 msgr2=0x7fe3fc19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 0x7fe3fc19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 -- 172.21.15.87:0/2288858883 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc19c630 con 0x7fe3fc101b60 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4029a3640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 0x7fe3fc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe4019a1640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c890 tx=0x7fe3ec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec004220 con 0x7fe3fc101b60 2024-04-22T13:33:28.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec0043c0 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.142+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc19c920 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.143+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec004a00 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.143+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc1a41b0 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.144+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3fc102570 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.145+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe3ec0079b0 con 0x7fe3fc101b60 2024-04-22T13:33:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.145+0000 7fe3f37fe640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 0x7fe3e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.145+0000 7fe4021a2640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 0x7fe3e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.145+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe3ec0d04e0 con 0x7fe3fc101b60 2024-04-22T13:33:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.146+0000 7fe4021a2640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 0x7fe3e80a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005ab0 tx=0x7fe3f8005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.150+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3ec014030 con 0x7fe3fc101b60 2024-04-22T13:33:28.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.374+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3fc109390 con 0x7fe3fc101b60 2024-04-22T13:33:28.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.375+0000 7fe3f37fe640 1 -- 172.21.15.87:0/2288858883 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fe3ec09cbf0 con 0x7fe3fc101b60 2024-04-22T13:33:28.371 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:28.371 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:28.371 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 msgr2=0x7fe3e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 0x7fe3e80a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005ab0 tx=0x7fe3f8005fd0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 msgr2=0x7fe3fc197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00c890 tx=0x7fe3ec00cd50 comp rx=0 tx=0).stop 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 shutdown_connections 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3e80a3d20 0x7fe3e80a61e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3fc105e30 0x7fe3fc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3fc101b60 0x7fe3fc197b80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 --2- 172.21.15.87:0/2288858883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3fc1011a0 0x7fe3fc197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 >> 172.21.15.87:0/2288858883 conn(0x7fe3fc077e30 msgr2=0x7fe3fc0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 shutdown_connections 2024-04-22T13:33:28.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.377+0000 7fe404406640 1 -- 172.21.15.87:0/2288858883 wait complete. 2024-04-22T13:33:28.384 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 86afa3a0 --release --await' 2024-04-22T13:33:28.384 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 86afa3a0 --release --await 2024-04-22T13:33:28.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 -- 172.21.15.87:0/2459717802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103960 msgr2=0x7f321010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 --2- 172.21.15.87:0/2459717802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103960 0x7f321010be40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009900 tx=0x7f31fc02bcd0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 -- 172.21.15.87:0/2459717802 shutdown_connections 2024-04-22T13:33:28.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 --2- 172.21.15.87:0/2459717802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321010e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 --2- 172.21.15.87:0/2459717802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103960 0x7f321010be40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 --2- 172.21.15.87:0/2459717802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103040 0x7f3210103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 -- 172.21.15.87:0/2459717802 >> 172.21.15.87:0/2459717802 conn(0x7f32100fc7e0 msgr2=0x7f32100fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 -- 172.21.15.87:0/2459717802 shutdown_connections 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.497+0000 7f3215b9f640 1 -- 172.21.15.87:0/2459717802 wait complete. 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 Processor -- start 2024-04-22T13:33:28.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 -- start start 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103960 0x7f3210198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3210112050 con 0x7f3210103040 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3210111ed0 con 0x7f3210103960 2024-04-22T13:33:28.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3215b9f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32101121d0 con 0x7f321010c410 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45382/0 (socket says 172.21.15.87:45382) 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 -- 172.21.15.87:0/3673473577 learned_addr learned my addr 172.21.15.87:0/3673473577 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f320ffff640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103960 0x7f3210198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f321539e640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 -- 172.21.15.87:0/3673473577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 msgr2=0x7f321019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 -- 172.21.15.87:0/3673473577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103960 msgr2=0x7f3210198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103960 0x7f3210198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f3214b9d640 1 -- 172.21.15.87:0/3673473577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321019d5e0 con 0x7f3210103040 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.498+0000 7f321539e640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f320400d950 tx=0x7f320400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f320dffb640 1 -- 172.21.15.87:0/3673473577 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3204017030 con 0x7f3210103040 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f320dffb640 1 -- 172.21.15.87:0/3673473577 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204004020 con 0x7f3210103040 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f320dffb640 1 -- 172.21.15.87:0/3673473577 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32040047a0 con 0x7f3210103040 2024-04-22T13:33:28.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321019d8d0 con 0x7f3210103040 2024-04-22T13:33:28.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.499+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32101a5160 con 0x7f3210103040 2024-04-22T13:33:28.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.501+0000 7f320dffb640 1 -- 172.21.15.87:0/3673473577 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3204009a30 con 0x7f3210103040 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.501+0000 7f320dffb640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 0x7f32000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.502+0000 7f320dffb640 1 -- 172.21.15.87:0/3673473577 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f32040d0160 con 0x7f3210103040 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.502+0000 7f320ffff640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 0x7f32000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.502+0000 7f320ffff640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 0x7f32000a6260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006010 tx=0x7f31fc005e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.502+0000 7f32199f1740 1 Processor -- start 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 -- start start 2024-04-22T13:33:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc274490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 0x5635cc276dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5635cc1d0500 con 0x5635cc26b4d0 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5635cc1766a0 con 0x5635cc26bdf0 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f32199f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5635cc1cc4c0 con 0x5635cc2749d0 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45386/0 (socket says 172.21.15.87:45386) 2024-04-22T13:33:28.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 -- 172.21.15.87:0/818435762 learned_addr learned my addr 172.21.15.87:0/818435762 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f320ffff640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc274490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f321539e640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 0x5635cc276dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 -- 172.21.15.87:0/818435762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 msgr2=0x5635cc276dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 0x5635cc276dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 -- 172.21.15.87:0/818435762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 msgr2=0x5635cc274490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc274490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 -- 172.21.15.87:0/818435762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635cc1955e0 con 0x5635cc26b4d0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f321539e640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 0x5635cc276dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.503+0000 7f3214b9d640 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3204098b40 tx=0x7f3204005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a5797270fbbbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f31faffd640 1 -- 172.21.15.87:0/818435762 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32040a4de0 con 0x5635cc26b4d0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f31faffd640 1 -- 172.21.15.87:0/818435762 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204009d20 con 0x5635cc26b4d0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f31faffd640 1 -- 172.21.15.87:0/818435762 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32040cdd30 con 0x5635cc26b4d0 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 -- 172.21.15.87:0/818435762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 msgr2=0x5635cc26b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3204098b40 tx=0x7f3204005780 comp rx=0 tx=0).stop 2024-04-22T13:33:28.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 -- 172.21.15.87:0/818435762 shutdown_connections 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc2749d0 0x5635cc276dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc274490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 --2- 172.21.15.87:0/818435762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc26b8b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 -- 172.21.15.87:0/818435762 >> 172.21.15.87:0/818435762 conn(0x5635cc265c00 msgr2=0x5635cc265fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 -- 172.21.15.87:0/818435762 shutdown_connections 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.504+0000 7f32199f1740 1 -- 172.21.15.87:0/818435762 wait complete. 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 -- start start 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 Processor -- start 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc274450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc271f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5635cc277760 con 0x5635cc26bdf0 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.505+0000 7f32199f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5635cc189a60 con 0x5635cc2749d0 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f32199f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5635cc2778e0 con 0x5635cc26b4d0 2024-04-22T13:33:28.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f32199f1740 10 client.0 _ll_register_callbacks cb 0x5635cbee2110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:28.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f320ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f320ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45388/0 (socket says 172.21.15.87:45388) 2024-04-22T13:33:28.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f320ffff640 1 -- 172.21.15.87:0/3982571024 learned_addr learned my addr 172.21.15.87:0/3982571024 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc271f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc274450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 -- 172.21.15.87:0/3982571024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 msgr2=0x5635cc274450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc274450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 -- 172.21.15.87:0/3982571024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 msgr2=0x5635cc2717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 -- 172.21.15.87:0/3982571024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635cbee2920 con 0x5635cc2749d0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f320ffff640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f321539e640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc271f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3208005cc0 tx=0x7f320800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f744005545f1d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f31effff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320801c060 con 0x5635cc2749d0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f320800ccf0 con 0x5635cc2749d0 2024-04-22T13:33:28.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.506+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3204009d20 con 0x5635cc2749d0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3208021440 con 0x5635cc2749d0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 10 client.25490 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5635cbfaed30 con 0x5635cc2749d0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204009d20 con 0x5635cc2749d0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f320801f020 con 0x5635cc2749d0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 10 client.25490 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 10 client.25490 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 20 client.25490 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc284b00 0x5635cc286f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 4 client.25490 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635cc1a3a10 con 0x5635cc284b00 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.507+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc284b00 0x5635cc286f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc284b00 0x5635cc286f20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52036eda76b93562 server_cookie=66c6bd03e82d60d3 in_seq=0 out_seq=0 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f31effff640 10 client.25490 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f31effff640 1 -- 172.21.15.87:0/3982571024 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635cc1a3a10 con 0x5635cc284b00 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f31effff640 10 client.25490 handle_command_reply: tid=0 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f32199f1740 1 client.25490 shutdown 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.508+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 msgr2=0x5635cc271f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc271f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3208005cc0 tx=0x7f320800ee30 comp rx=0 tx=0).stop 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 shutdown_connections 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc284b00 0x5635cc286f20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc271f20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26bdf0 0x5635cc2717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 --2- 172.21.15.87:0/3982571024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc274450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 >> 172.21.15.87:0/3982571024 conn(0x5635cc265c00 msgr2=0x5635cc267950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f31effff640 0 client.25490 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:28.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 shutdown_connections 2024-04-22T13:33:28.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 1 -- 172.21.15.87:0/3982571024 wait complete. 2024-04-22T13:33:28.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.509+0000 7f32199f1740 20 client.25490 trim_cache size 0 max 16384 2024-04-22T13:33:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 Processor -- start 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 -- start start 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 0x5635cc271070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc273f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32101b0150 con 0x5635cc26b4d0 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32101ada20 con 0x5635cc2749d0 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f32199f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32101ab870 con 0x5635cc26bdf0 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45404/0 (socket says 172.21.15.87:45404) 2024-04-22T13:33:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 -- 172.21.15.87:0/3193968422 learned_addr learned my addr 172.21.15.87:0/3193968422 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f320ffff640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 0x5635cc271070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f321539e640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc273f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 -- 172.21.15.87:0/3193968422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 msgr2=0x5635cc271070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 0x5635cc271070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 -- 172.21.15.87:0/3193968422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 msgr2=0x5635cc273f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc273f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f3214b9d640 1 -- 172.21.15.87:0/3193968422 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635cc266540 con 0x5635cc26b4d0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f321539e640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc273f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.526+0000 7f320ffff640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 0x5635cc271070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32040045f0 tx=0x7f32040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2e128704cec94e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f31fa7fc640 1 -- 172.21.15.87:0/3193968422 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320409bf40 con 0x5635cc26b4d0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f31fa7fc640 1 -- 172.21.15.87:0/3193968422 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204009d20 con 0x5635cc26b4d0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f31fa7fc640 1 -- 172.21.15.87:0/3193968422 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32040987e0 con 0x5635cc26b4d0 2024-04-22T13:33:28.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 -- 172.21.15.87:0/3193968422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 msgr2=0x5635cc270720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32040045f0 tx=0x7f32040046c0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 -- 172.21.15.87:0/3193968422 shutdown_connections 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc2749d0 0x5635cc273f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26bdf0 0x5635cc271070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 --2- 172.21.15.87:0/3193968422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc26b4d0 0x5635cc270720 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.527+0000 7f32199f1740 1 -- 172.21.15.87:0/3193968422 >> 172.21.15.87:0/3193968422 conn(0x5635cc265c00 msgr2=0x5635cc2678c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 -- 172.21.15.87:0/3193968422 shutdown_connections 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 -- 172.21.15.87:0/3193968422 wait complete. 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 -- start start 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 Processor -- start 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc27cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.528+0000 7f32199f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc27d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32101abee0 con 0x5635cc2749d0 2024-04-22T13:33:28.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32101ad350 con 0x5635cc26bdf0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32101ac5b0 con 0x5635cc26b4d0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 10 client.0 _ll_register_callbacks cb 0x7f3208009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45406/0 (socket says 172.21.15.87:45406) 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 -- 172.21.15.87:0/3391170745 learned_addr learned my addr 172.21.15.87:0/3391170745 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f320ffff640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc27d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc27cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 -- 172.21.15.87:0/3391170745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 msgr2=0x5635cc27cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc27cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 -- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 msgr2=0x5635cc27d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc27d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 -- 172.21.15.87:0/3391170745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635cc276340 con 0x5635cc2749d0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f321539e640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f320800f890 tx=0x7f320800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b38c729ec546db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f31effff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320802e060 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f320800fd20 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3204009d20 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f320801c020 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 10 client.16155 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.529+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5635cc2eb900 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3204009d20 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3208031030 con 0x5635cc2749d0 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 10 client.16155 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 10 client.16155 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 20 client.16155 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc2ec100 0x5635cc2ee520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 4 client.16155 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "86afa3a0", "release": true, "await": true}] 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "86afa3a0", "release": true, "await": true}) -- 0x5635cc1ddf20 con 0x5635cc2ec100 2024-04-22T13:33:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc2ec100 0x5635cc2ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f3214b9d640 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc2ec100 0x5635cc2ee520 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330ff995ef569f09 server_cookie=2c4a1336b53f63b in_seq=0 out_seq=0 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.530+0000 7f31effff640 10 client.16155 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f31effff640 1 -- 172.21.15.87:0/3391170745 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5635cc1ddf20 con 0x5635cc2ec100 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f31effff640 10 client.16155 handle_command_reply: tid=0 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 client.16155 shutdown 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 msgr2=0x5635cc275d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f320800f890 tx=0x7f320800c800 comp rx=0 tx=0).stop 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 shutdown_connections 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5635cc2ec100 0x5635cc2ee520 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5635cc2749d0 0x5635cc275d80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635cc26bdf0 0x5635cc27d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f31effff640 0 client.16155 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:28.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 --2- 172.21.15.87:0/3391170745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5635cc26b4d0 0x5635cc27cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.532+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 >> 172.21.15.87:0/3391170745 conn(0x5635cc265c00 msgr2=0x5635cc2643d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 shutdown_connections 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f32199f1740 1 -- 172.21.15.87:0/3391170745 wait complete. 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f32199f1740 20 client.16155 trim_cache size 0 max 16384 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:33:28.538 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 8, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "86afa3a0": { 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 8, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 83.5, 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:33:28.539 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:28.540 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:28.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 msgr2=0x7f32000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 0x7f32000a6260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006010 tx=0x7f31fc005e40 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 msgr2=0x7f32101985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f320400d950 tx=0x7f320400de10 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 shutdown_connections 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32000a3da0 0x7f32000a6260 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f321010c410 0x7f321019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3210103960 0x7f3210198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 --2- 172.21.15.87:0/3673473577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3210103040 0x7f32101985f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.533+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 >> 172.21.15.87:0/3673473577 conn(0x7f32100fc7e0 msgr2=0x7f32100fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.534+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 shutdown_connections 2024-04-22T13:33:28.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.534+0000 7f3215b9f640 1 -- 172.21.15.87:0/3673473577 wait complete. 2024-04-22T13:33:28.542 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 86afa3a0, seconds elapsed: 24.8 2024-04-22T13:33:28.543 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.55 2024-04-22T13:33:28.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 -- 172.21.15.87:0/1217421526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70107fe0 msgr2=0x7fdf701083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/1217421526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70107fe0 0x7fdf701083c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009900 tx=0x7fdf6c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 -- 172.21.15.87:0/1217421526 shutdown_connections 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/1217421526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf70069a80 0x7fdf7010eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/1217421526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf700690e0 0x7fdf70069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/1217421526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70107fe0 0x7fdf701083c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 -- 172.21.15.87:0/1217421526 >> 172.21.15.87:0/1217421526 conn(0x7fdf70077e30 msgr2=0x7fdf70078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 -- 172.21.15.87:0/1217421526 shutdown_connections 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.648+0000 7fdf77b7e640 1 -- 172.21.15.87:0/1217421526 wait complete. 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 Processor -- start 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 -- start start 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf700690e0 0x7fdf70101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 0x7fdf70101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdf70112160 con 0x7fdf70069a80 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf70111fe0 con 0x7fdf70107fe0 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf77b7e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf701122e0 con 0x7fdf700690e0 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45414/0 (socket says 172.21.15.87:45414) 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf7591a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf700690e0 0x7fdf70101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 -- 172.21.15.87:0/4057001156 learned_addr learned my addr 172.21.15.87:0/4057001156 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf7611b640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 0x7fdf70101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 -- 172.21.15.87:0/4057001156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf700690e0 msgr2=0x7fdf70101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf700690e0 0x7fdf70101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 -- 172.21.15.87:0/4057001156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 msgr2=0x7fdf70101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 0x7fdf70101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf75119640 1 -- 172.21.15.87:0/4057001156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf701a6be0 con 0x7fdf70069a80 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.649+0000 7fdf7611b640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 0x7fdf70101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf75119640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c920 tx=0x7fdf6000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60007c70 con 0x7fdf70069a80 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf60007e10 con 0x7fdf70069a80 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf701a6ed0 con 0x7fdf70069a80 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60018820 con 0x7fdf70069a80 2024-04-22T13:33:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.650+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf701a7390 con 0x7fdf70069a80 2024-04-22T13:33:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.651+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf70068de0 con 0x7fdf70069a80 2024-04-22T13:33:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.653+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdf6000f450 con 0x7fdf70069a80 2024-04-22T13:33:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.653+0000 7fdf66ffd640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 0x7fdf5c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.653+0000 7fdf7591a640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 0x7fdf5c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.654+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdf600d0b50 con 0x7fdf70069a80 2024-04-22T13:33:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.654+0000 7fdf7591a640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 0x7fdf5c0a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c006010 tx=0x7fdf6c02c6c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.656+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf60014030 con 0x7fdf70069a80 2024-04-22T13:33:28.694 INFO:tasks.workunit.client.0.smithi087.stderr:Updating files: 97% (11921/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T13:33:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.878+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf701027a0 con 0x7fdf70069a80 2024-04-22T13:33:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.879+0000 7fdf66ffd640 1 -- 172.21.15.87:0/4057001156 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7fdf6009d2f0 con 0x7fdf70069a80 2024-04-22T13:33:28.875 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:28.875 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 msgr2=0x7fdf5c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 0x7fdf5c0a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c006010 tx=0x7fdf6c02c6c0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 msgr2=0x7fdf701017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c920 tx=0x7fdf6000cde0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 shutdown_connections 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdf5c0a3d20 0x7fdf5c0a61e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf70107fe0 0x7fdf70101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdf70069a80 0x7fdf701017b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 --2- 172.21.15.87:0/4057001156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf700690e0 0x7fdf70101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 >> 172.21.15.87:0/4057001156 conn(0x7fdf70077e30 msgr2=0x7fdf700ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 shutdown_connections 2024-04-22T13:33:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:28.881+0000 7fdf77b7e640 1 -- 172.21.15.87:0/4057001156 wait complete. 2024-04-22T13:33:28.888 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 83.55 --await' 2024-04-22T13:33:28.889 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 83.55 --await 2024-04-22T13:33:28.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- 172.21.15.87:0/960924953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c101a50 msgr2=0x7f904c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 --2- 172.21.15.87:0/960924953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c101a50 0x7f904c105a90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f903c009900 tx=0x7f903c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- 172.21.15.87:0/960924953 shutdown_connections 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 --2- 172.21.15.87:0/960924953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 0x7f904c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 --2- 172.21.15.87:0/960924953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c101a50 0x7f904c105a90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 --2- 172.21.15.87:0/960924953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c1010a0 0x7f904c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- 172.21.15.87:0/960924953 >> 172.21.15.87:0/960924953 conn(0x7f904c077f50 msgr2=0x7f904c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- 172.21.15.87:0/960924953 shutdown_connections 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- 172.21.15.87:0/960924953 wait complete. 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 Processor -- start 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.002+0000 7f90537bb640 1 -- start start 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 0x7f904c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 0x7f904c1a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f904c111090 con 0x7f904c1010a0 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f904c110f10 con 0x7f904c106170 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f904c111210 con 0x7f904c19a030 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45432/0 (socket says 172.21.15.87:45432) 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 -- 172.21.15.87:0/2969395072 learned_addr learned my addr 172.21.15.87:0/2969395072 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9050d56640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 0x7f904c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051d58640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 0x7f904c1a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 -- 172.21.15.87:0/2969395072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 msgr2=0x7f904c1a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 0x7f904c1a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 -- 172.21.15.87:0/2969395072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 msgr2=0x7f904c199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 0x7f904c199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 -- 172.21.15.87:0/2969395072 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c1a4db0 con 0x7f904c1010a0 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051d58640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 0x7f904c1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:29.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f9051557640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f904400c890 tx=0x7f904400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90427fc640 1 -- 172.21.15.87:0/2969395072 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044004220 con 0x7f904c1010a0 2024-04-22T13:33:29.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904c1a50a0 con 0x7f904c1010a0 2024-04-22T13:33:29.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90427fc640 1 -- 172.21.15.87:0/2969395072 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90440043c0 con 0x7f904c1010a0 2024-04-22T13:33:29.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90427fc640 1 -- 172.21.15.87:0/2969395072 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044004a00 con 0x7f904c1010a0 2024-04-22T13:33:29.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.003+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904c1a55c0 con 0x7f904c1010a0 2024-04-22T13:33:29.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.005+0000 7f90427fc640 1 -- 172.21.15.87:0/2969395072 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f90440079b0 con 0x7f904c1010a0 2024-04-22T13:33:29.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 Processor -- start 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 -- start start 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5d3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5e36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b94c5340b0 con 0x55b94c5d4010 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b94c4e5680 con 0x55b94c5d36f0 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90563a8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b94c4f1c60 con 0x55b94c5e12d0 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90427fc640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 0x7f90380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45442/0 (socket says 172.21.15.87:45442) 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9051557640 1 -- 172.21.15.87:0/3794425438 learned_addr learned my addr 172.21.15.87:0/3794425438 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9051d58640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 0x7f90380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9051d58640 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5d3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f9050d56640 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5e36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.006+0000 7f90427fc640 1 -- 172.21.15.87:0/2969395072 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f90440d04e0 con 0x7f904c1010a0 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 -- 172.21.15.87:0/3794425438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 msgr2=0x55b94c5e36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5e36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 -- 172.21.15.87:0/3794425438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 msgr2=0x55b94c5d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5d3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 -- 172.21.15.87:0/3794425438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b94c316c90 con 0x55b94c5d4010 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051557640 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9044098950 tx=0x7f904409c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b583c16ce41377e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f9051d58640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 0x7f90380a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9048009fd0 tx=0x7f9048009290 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f90377fe640 1 -- 172.21.15.87:0/3794425438 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904401d050 con 0x55b94c5d4010 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f90377fe640 1 -- 172.21.15.87:0/3794425438 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90440bb6b0 con 0x55b94c5d4010 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.007+0000 7f90377fe640 1 -- 172.21.15.87:0/3794425438 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90440cdf20 con 0x55b94c5d4010 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- 172.21.15.87:0/3794425438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 msgr2=0x55b94c5e0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9044098950 tx=0x7f904409c380 comp rx=0 tx=0).stop 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- 172.21.15.87:0/3794425438 shutdown_connections 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5e0d90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 --2- 172.21.15.87:0/3794425438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5d3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- 172.21.15.87:0/3794425438 >> 172.21.15.87:0/3794425438 conn(0x55b94c5ce0f0 msgr2=0x55b94c5ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- 172.21.15.87:0/3794425438 shutdown_connections 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- 172.21.15.87:0/3794425438 wait complete. 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 -- start start 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 Processor -- start 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.008+0000 7f90563a8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d4010 0x55b94c5de8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 0x55b94c5dee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b94c537400 con 0x55b94c5d36f0 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b94c4f3840 con 0x55b94c5e12d0 2024-04-22T13:33:29.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b94c53c720 con 0x55b94c5d4010 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 10 client.0 _ll_register_callbacks cb 0x55b94c544ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45448/0 (socket says 172.21.15.87:45448) 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 -- 172.21.15.87:0/1294071052 learned_addr learned my addr 172.21.15.87:0/1294071052 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9050d56640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d4010 0x55b94c5de8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051d58640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 0x55b94c5dee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 -- 172.21.15.87:0/1294071052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d4010 msgr2=0x55b94c5de8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d4010 0x55b94c5de8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 -- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 msgr2=0x55b94c5dee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 0x55b94c5dee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 -- 172.21.15.87:0/1294071052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b94c377420 con 0x55b94c5d36f0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051d58640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 0x55b94c5dee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9051557640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9044098ec0 tx=0x7f90440a51d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0de505ac365428 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904409cbf0 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f90440d00c0 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90440bb6b0 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f904401d070 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 10 client.16173 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.009+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b94c20a580 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90440bb6b0 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9044098240 con 0x55b94c5d36f0 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 10 client.16173 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 10 client.16173 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 20 client.16173 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c5f0f20 0x55b94c5f3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 4 client.16173 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b94c545b10 con 0x55b94c5f0f20 2024-04-22T13:33:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.010+0000 7f9050d56640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c5f0f20 0x55b94c5f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f9034ff9640 10 client.16173 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f9050d56640 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c5f0f20 0x55b94c5f3310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7561d7ceffb90f7c server_cookie=82e330651771fdff in_seq=0 out_seq=0 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f9034ff9640 1 -- 172.21.15.87:0/1294071052 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b94c545b10 con 0x55b94c5f0f20 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f9034ff9640 10 client.16173 handle_command_reply: tid=0 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 client.16173 shutdown 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 msgr2=0x55b94c5de3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9044098ec0 tx=0x7f90440a51d0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 shutdown_connections 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c5f0f20 0x55b94c5f3310 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5e12d0 0x55b94c5dee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f9034ff9640 0 client.16173 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d4010 0x55b94c5de8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 --2- 172.21.15.87:0/1294071052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d36f0 0x55b94c5de3b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.011+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 >> 172.21.15.87:0/1294071052 conn(0x55b94c5ce0f0 msgr2=0x55b94c5cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.012+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 shutdown_connections 2024-04-22T13:33:29.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.012+0000 7f90563a8740 1 -- 172.21.15.87:0/1294071052 wait complete. 2024-04-22T13:33:29.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.012+0000 7f90563a8740 20 client.16173 trim_cache size 0 max 16384 2024-04-22T13:33:29.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 Processor -- start 2024-04-22T13:33:29.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 -- start start 2024-04-22T13:33:29.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5e0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1aba30 con 0x55b94c5d4010 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1a98a0 con 0x55b94c5d36f0 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f90563a8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1ab360 con 0x55b94c5e12d0 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9050d56640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9050d56640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45460/0 (socket says 172.21.15.87:45460) 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9050d56640 1 -- 172.21.15.87:0/4145412801 learned_addr learned my addr 172.21.15.87:0/4145412801 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9051557640 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5e0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9050d56640 1 -- 172.21.15.87:0/4145412801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 msgr2=0x55b94c5cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.029+0000 7f9050d56640 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9050d56640 1 -- 172.21.15.87:0/4145412801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 msgr2=0x55b94c5e0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9050d56640 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5e0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9050d56640 1 -- 172.21.15.87:0/4145412801 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b94c5c85e0 con 0x55b94c5d4010 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9050d56640 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f904c06b6f0 tx=0x7f903c02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18423ef6e98b502 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9036ffd640 1 -- 172.21.15.87:0/4145412801 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c050070 con 0x55b94c5d4010 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9036ffd640 1 -- 172.21.15.87:0/4145412801 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f903c02cc90 con 0x55b94c5d4010 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.030+0000 7f9036ffd640 1 -- 172.21.15.87:0/4145412801 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c005040 con 0x55b94c5d4010 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- 172.21.15.87:0/4145412801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 msgr2=0x55b94c5ecc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f904c06b6f0 tx=0x7f903c02c330 comp rx=0 tx=0).stop 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- 172.21.15.87:0/4145412801 shutdown_connections 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5e12d0 0x55b94c5cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5d4010 0x55b94c5ecc90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- 172.21.15.87:0/4145412801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d36f0 0x55b94c5e0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- 172.21.15.87:0/4145412801 >> 172.21.15.87:0/4145412801 conn(0x55b94c5ce0f0 msgr2=0x55b94c5dd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- 172.21.15.87:0/4145412801 shutdown_connections 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- 172.21.15.87:0/4145412801 wait complete. 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- start start 2024-04-22T13:33:29.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 Processor -- start 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 0x55b94c5db310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 0x55b94c5db850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1aa5c0 con 0x55b94c5e12d0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1a9ef0 con 0x55b94c5d4010 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1aac90 con 0x55b94c5d36f0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.031+0000 7f90563a8740 10 client.0 _ll_register_callbacks cb 0x7f9048009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45462/0 (socket says 172.21.15.87:45462) 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 -- 172.21.15.87:0/2279280714 learned_addr learned my addr 172.21.15.87:0/2279280714 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051557640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 0x55b94c5db310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9050d56640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 0x55b94c5db850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 -- 172.21.15.87:0/2279280714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 msgr2=0x55b94c5db310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 0x55b94c5db310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 -- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 msgr2=0x55b94c5db850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 0x55b94c5db850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 -- 172.21.15.87:0/2279280714 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b94c656660 con 0x55b94c5e12d0 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051557640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 0x55b94c5db310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9050d56640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 0x55b94c5db850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9051d58640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f90480148d0 tx=0x7f90480098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d388bb5c1d6922d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048025070 con 0x55b94c5e12d0 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9048004d20 con 0x55b94c5e12d0 2024-04-22T13:33:29.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f903c02cc90 con 0x55b94c5e12d0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9048005020 con 0x55b94c5e12d0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f90563a8740 10 client.16179 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.032+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b94c657700 con 0x55b94c5e12d0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f903c02cc90 con 0x55b94c5e12d0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9048031050 con 0x55b94c5e12d0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 10 client.16179 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 10 client.16179 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 20 client.16179 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c657ed0 0x55b94c65a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 4 client.16179 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.55, "await": true}] 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.55, "await": true}) -- 0x55b94c53f980 con 0x55b94c657ed0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f9051557640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c657ed0 0x55b94c65a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f9051557640 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c657ed0 0x55b94c65a2f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7686e2e9491ee31 server_cookie=4b3f471721b03c9 in_seq=0 out_seq=0 2024-04-22T13:33:29.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.033+0000 7f9034ff9640 10 client.16179 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.039+0000 7f9034ff9640 1 -- 172.21.15.87:0/2279280714 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b94c53f980 con 0x55b94c657ed0 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.039+0000 7f9034ff9640 10 client.16179 handle_command_reply: tid=0 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 client.16179 shutdown 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 msgr2=0x55b94c656120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f90480148d0 tx=0x7f90480098e0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 shutdown_connections 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b94c657ed0 0x55b94c65a2f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b94c5e12d0 0x55b94c656120 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b94c5d4010 0x55b94c5db850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f9034ff9640 0 client.16179 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:29.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 --2- 172.21.15.87:0/2279280714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b94c5d36f0 0x55b94c5db310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 >> 172.21.15.87:0/2279280714 conn(0x55b94c5ce0f0 msgr2=0x55b94c5cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 shutdown_connections 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 1 -- 172.21.15.87:0/2279280714 wait complete. 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90563a8740 20 client.16179 trim_cache size 0 max 16384 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 10, 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "1e2e33c6": { 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 10, 2024-04-22T13:33:29.037 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 83.5, 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:29.038 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:29.039 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:29.039 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:29.039 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:29.039 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:29.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 msgr2=0x7f90380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 0x7f90380a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9048009fd0 tx=0x7f9048009290 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 msgr2=0x7f904c199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f904400c890 tx=0x7f904400cd50 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 shutdown_connections 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f90380a3d20 0x7f90380a61e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f904c19a030 0x7f904c1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f904c106170 0x7f904c199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 --2- 172.21.15.87:0/2969395072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f904c1010a0 0x7f904c199510 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.040+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 >> 172.21.15.87:0/2969395072 conn(0x7f904c077f50 msgr2=0x7f904c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.041+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 shutdown_connections 2024-04-22T13:33:29.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:29.041+0000 7f90537bb640 1 -- 172.21.15.87:0/2969395072 wait complete. 2024-04-22T13:33:29.050 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 1e2e33c6, quiesce took 0.5 seconds. Will release after: 23.55 2024-04-22T13:33:29.452 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to 'd69f13d827952bb07bb9b7973bca14f97d764d6f'. 2024-04-22T13:33:29.452 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.452 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T13:33:29.452 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:29.453 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at d69f13d8279 mds/cache: request_drop_foreign_locks shouldn't strip local xlocks 2024-04-22T13:33:29.458 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T13:33:29.490 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T13:33:29.492 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T13:33:29.492 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T13:33:29.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:29 smithi120 ceph-mon[31101]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:29.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:29 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2288858883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:29.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:29 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4057001156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:29.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:29 smithi132 ceph-mon[31531]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:29 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2288858883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:29.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:29 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4057001156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:29.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:29 smithi087 ceph-mon[27999]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-22T13:33:29.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:29 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2288858883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:29.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:29 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4057001156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:30.089 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T13:33:30.144 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T13:33:30.182 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T13:33:30.186 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T13:33:30.186 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T13:33:30.222 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T13:33:30.225 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:33:30.225 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T13:33:30.281 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-22T13:33:30.282 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-22T13:33:30.283 DEBUG:teuthology.orchestra.run.smithi087:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d69f13d827952bb07bb9b7973bca14f97d764d6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-22T13:33:31.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:31 smithi120 ceph-mon[31101]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:31.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:31 smithi132 ceph-mon[31531]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:31.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:31 smithi087 ceph-mon[27999]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 -- 172.21.15.87:0/3459810301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 msgr2=0x7f9320101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 --2- 172.21.15.87:0/3459810301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320101620 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9314009900 tx=0x7f931402bc80 comp rx=0 tx=0).stop 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 -- 172.21.15.87:0/3459810301 shutdown_connections 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 --2- 172.21.15.87:0/3459810301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 0x7f932010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 --2- 172.21.15.87:0/3459810301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320101620 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 --2- 172.21.15.87:0/3459810301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 0x7f9320107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 -- 172.21.15.87:0/3459810301 >> 172.21.15.87:0/3459810301 conn(0x7f9320076c50 msgr2=0x7f9320077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 -- 172.21.15.87:0/3459810301 shutdown_connections 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.576+0000 7f93276c7640 1 -- 172.21.15.87:0/3459810301 wait complete. 2024-04-22T13:33:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 Processor -- start 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 -- start start 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 0x7f9320199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 0x7f93201a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111080 con 0x7f93201011a0 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9320110f00 con 0x7f9320101b60 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f93276c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111200 con 0x7f9320106fb0 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43848/0 (socket says 172.21.15.87:43848) 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 -- 172.21.15.87:0/4147999184 learned_addr learned my addr 172.21.15.87:0/4147999184 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9324c62640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 0x7f9320199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325c64640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 0x7f93201a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 -- 172.21.15.87:0/4147999184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 msgr2=0x7f93201a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 0x7f93201a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 -- 172.21.15.87:0/4147999184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 msgr2=0x7f9320199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 0x7f9320199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.577+0000 7f9325463640 1 -- 172.21.15.87:0/4147999184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93201a49e0 con 0x7f93201011a0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f9325463640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f93127fc640 1 -- 172.21.15.87:0/4147999184 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c00cd80 con 0x7f93201011a0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f93127fc640 1 -- 172.21.15.87:0/4147999184 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931c0054d0 con 0x7f93201011a0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93201a4bb0 con 0x7f93201011a0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f93127fc640 1 -- 172.21.15.87:0/4147999184 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c010690 con 0x7f93201011a0 2024-04-22T13:33:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.578+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93201a5080 con 0x7f93201011a0 2024-04-22T13:33:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.580+0000 7f93127fc640 1 -- 172.21.15.87:0/4147999184 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f931c004e90 con 0x7f93201011a0 2024-04-22T13:33:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.580+0000 7f932a38b740 1 Processor -- start 2024-04-22T13:33:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 -- start start 2024-04-22T13:33:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd40620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd4a0e0 0x55daccd4c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55daccca58d0 con 0x55daccd40b60 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55daccc42e80 con 0x55daccd418d0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f932a38b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55daccca1890 con 0x55daccd4a0e0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f93127fc640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 0x7f930c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325463640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325463640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43862/0 (socket says 172.21.15.87:43862) 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325463640 1 -- 172.21.15.87:0/473863771 learned_addr learned my addr 172.21.15.87:0/473863771 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9324c62640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd4a0e0 0x55daccd4c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd40620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 0x7f930c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f93127fc640 1 -- 172.21.15.87:0/4147999184 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f931c0d0bc0 con 0x7f93201011a0 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 -- 172.21.15.87:0/473863771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd4a0e0 msgr2=0x55daccd4c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd4a0e0 0x55daccd4c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 -- 172.21.15.87:0/473863771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 msgr2=0x55daccd40fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325c64640 1 -- 172.21.15.87:0/473863771 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daccc64320 con 0x55daccd418d0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.581+0000 7f9325463640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.582+0000 7f9325c64640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 0x7f930c0a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9318010ca0 tx=0x7f9318015000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.582+0000 7f9325c64640 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd40620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9318013da0 tx=0x7f93180107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a97b0486f0ebd72 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.582+0000 7f930b7fe640 1 -- 172.21.15.87:0/473863771 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318021070 con 0x55daccd418d0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.582+0000 7f930b7fe640 1 -- 172.21.15.87:0/473863771 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9318007b40 con 0x55daccd418d0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.582+0000 7f930b7fe640 1 -- 172.21.15.87:0/473863771 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931801d3f0 con 0x55daccd418d0 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- 172.21.15.87:0/473863771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 msgr2=0x55daccd40620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd40620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9318013da0 tx=0x7f93180107a0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- 172.21.15.87:0/473863771 shutdown_connections 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd4a0e0 0x55daccd4c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd40b60 0x55daccd40fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 --2- 172.21.15.87:0/473863771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd40620 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- 172.21.15.87:0/473863771 >> 172.21.15.87:0/473863771 conn(0x55daccd07c90 msgr2=0x55daccd3b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- 172.21.15.87:0/473863771 shutdown_connections 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- 172.21.15.87:0/473863771 wait complete. 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 -- start start 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.583+0000 7f932a38b740 1 Processor -- start 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd40b60 0x55daccd49a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd418d0 0x55daccd46dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55daccd4cdb0 con 0x55daccd4a0e0 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55daccc2deb0 con 0x55daccd40b60 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55daccd4cf30 con 0x55daccd418d0 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f932a38b740 10 client.0 _ll_register_callbacks cb 0x55daccc5ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43876/0 (socket says 172.21.15.87:43876) 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 -- 172.21.15.87:0/979278590 learned_addr learned my addr 172.21.15.87:0/979278590 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325463640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd40b60 0x55daccd49a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9324c62640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd418d0 0x55daccd46dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 -- 172.21.15.87:0/979278590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd418d0 msgr2=0x55daccd46dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd418d0 0x55daccd46dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 -- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd40b60 msgr2=0x55daccd49a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd40b60 0x55daccd49a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 -- 172.21.15.87:0/979278590 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dacc9b8210 con 0x55daccd4a0e0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9325c64640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f931800ea40 tx=0x7f931800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7acc6dde1b8d5f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9308ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318033060 con 0x55daccd4a0e0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.584+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9318003970 con 0x55daccd4a0e0 2024-04-22T13:33:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dacccb9880 con 0x55daccd4a0e0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9318021040 con 0x55daccd4a0e0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 10 client.16191 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dacca84620 con 0x55daccd4a0e0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55dacccb9880 con 0x55daccd4a0e0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9318036020 con 0x55daccd4a0e0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 10 client.16191 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 10 client.16191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 20 client.16191 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccd5a050 0x55daccd5c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 4 client.16191 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daccc453d0 con 0x55daccd5a050 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.585+0000 7f9325463640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccd5a050 0x55daccd5c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.586+0000 7f9325463640 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccd5a050 0x55daccd5c440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a313ef01d7138f66 server_cookie=e5d28391a8313a6f in_seq=0 out_seq=0 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.586+0000 7f9308ff9640 10 client.16191 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f9308ff9640 1 -- 172.21.15.87:0/979278590 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55daccc453d0 con 0x55daccd5a050 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f9308ff9640 10 client.16191 handle_command_reply: tid=0 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 client.16191 shutdown 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 msgr2=0x55daccd47330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f931800ea40 tx=0x7f931800ee70 comp rx=0 tx=0).stop 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 shutdown_connections 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccd5a050 0x55daccd5c440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd47330 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd418d0 0x55daccd46dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 --2- 172.21.15.87:0/979278590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd40b60 0x55daccd49a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f9308ff9640 0 client.16191 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 >> 172.21.15.87:0/979278590 conn(0x55daccd07c90 msgr2=0x55daccd3d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 shutdown_connections 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 1 -- 172.21.15.87:0/979278590 wait complete. 2024-04-22T13:33:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.587+0000 7f932a38b740 20 client.16191 trim_cache size 0 max 16384 2024-04-22T13:33:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 Processor -- start 2024-04-22T13:33:32.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 -- start start 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd55cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 0x55daccd620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93201b0150 con 0x55daccd418d0 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93201ada20 con 0x55daccd4a0e0 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f932a38b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93201ab890 con 0x55daccd40b60 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9324c62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9324c62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43888/0 (socket says 172.21.15.87:43888) 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9324c62640 1 -- 172.21.15.87:0/1835198508 learned_addr learned my addr 172.21.15.87:0/1835198508 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9325c64640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 0x55daccd620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9324c62640 1 -- 172.21.15.87:0/1835198508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 msgr2=0x55daccd55cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9325463640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd55cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.602+0000 7f9324c62640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd55cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9324c62640 1 -- 172.21.15.87:0/1835198508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 msgr2=0x55daccd620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9324c62640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 0x55daccd620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9324c62640 1 -- 172.21.15.87:0/1835198508 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daccd40990 con 0x55daccd418d0 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9325c64640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 0x55daccd620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9325463640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd55cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f9324c62640 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9314009900 tx=0x7f93140040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2527659a6661d5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f930affd640 1 -- 172.21.15.87:0/1835198508 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314004240 con 0x55daccd418d0 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f930affd640 1 -- 172.21.15.87:0/1835198508 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931402cc80 con 0x55daccd418d0 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f930affd640 1 -- 172.21.15.87:0/1835198508 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314004900 con 0x55daccd418d0 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f932a38b740 1 -- 172.21.15.87:0/1835198508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 msgr2=0x55daccd61bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.603+0000 7f932a38b740 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9314009900 tx=0x7f93140040d0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- 172.21.15.87:0/1835198508 shutdown_connections 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd4a0e0 0x55daccd620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd418d0 0x55daccd61bb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- 172.21.15.87:0/1835198508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd55cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- 172.21.15.87:0/1835198508 >> 172.21.15.87:0/1835198508 conn(0x55daccd07c90 msgr2=0x55daccd3b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- 172.21.15.87:0/1835198508 shutdown_connections 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- 172.21.15.87:0/1835198508 wait complete. 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- start start 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 Processor -- start 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd52700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93201abee0 con 0x55daccd4a0e0 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93201ad350 con 0x55daccd418d0 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93201ac5b0 con 0x55daccd40b60 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f932a38b740 10 client.0 _ll_register_callbacks cb 0x7f9318013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.604+0000 7f9325c64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43894/0 (socket says 172.21.15.87:43894) 2024-04-22T13:33:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 -- 172.21.15.87:0/1422425096 learned_addr learned my addr 172.21.15.87:0/1422425096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9324c62640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd52700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325463640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 -- 172.21.15.87:0/1422425096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 msgr2=0x55daccd521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 -- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 msgr2=0x55daccd52700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd52700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 -- 172.21.15.87:0/1422425096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daccd435d0 con 0x55daccd4a0e0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9325c64640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9318005730 tx=0x7f931800e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e68504eab4211de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9308ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318033070 con 0x55daccd4a0e0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9318003970 con 0x55daccd4a0e0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931402cc80 con 0x55daccd4a0e0 2024-04-22T13:33:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9318013900 con 0x55daccd4a0e0 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f932a38b740 10 client.16197 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55daccdbfc20 con 0x55daccd4a0e0 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.605+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931402cc80 con 0x55daccd4a0e0 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9318049020 con 0x55daccd4a0e0 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 10 client.16197 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 20 client.16197 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccdc0290 0x55daccdc26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 4 client.16197 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dacccb32f0 con 0x55daccdc0290 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f9325463640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccdc0290 0x55daccdc26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f9308ff9640 10 client.16197 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.606+0000 7f9325463640 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccdc0290 0x55daccdc26b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4c3a54fbe4736 server_cookie=9e73aafd7df193da in_seq=0 out_seq=0 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.612+0000 7f9308ff9640 1 -- 172.21.15.87:0/1422425096 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dacccb32f0 con 0x55daccdc0290 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.612+0000 7f9308ff9640 10 client.16197 handle_command_reply: tid=0 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.612+0000 7f932a38b740 1 client.16197 shutdown 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 msgr2=0x55daccd43040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9318005730 tx=0x7f931800e880 comp rx=0 tx=0).stop 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 shutdown_connections 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daccdc0290 0x55daccdc26b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daccd4a0e0 0x55daccd43040 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daccd418d0 0x55daccd52700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f9308ff9640 0 client.16197 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 --2- 172.21.15.87:0/1422425096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daccd40b60 0x55daccd521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 >> 172.21.15.87:0/1422425096 conn(0x55daccd07c90 msgr2=0x55daccd456a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.609 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 shutdown_connections 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 1 -- 172.21.15.87:0/1422425096 wait complete. 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f932a38b740 20 client.16197 trim_cache size 0 max 16384 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:33:32.610 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 msgr2=0x7f930c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 0x7f930c0a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9318010ca0 tx=0x7f9318015000 comp rx=0 tx=0).stop 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 msgr2=0x7f9320199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00edf0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 shutdown_connections 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f930c0a3dc0 0x7f930c0a6280 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9320106fb0 0x7f93201a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9320101b60 0x7f9320199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 --2- 172.21.15.87:0/4147999184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93201011a0 0x7f9320199420 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 >> 172.21.15.87:0/4147999184 conn(0x7f9320076c50 msgr2=0x7f93200ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 shutdown_connections 2024-04-22T13:33:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:32.613+0000 7f93276c7640 1 -- 172.21.15.87:0/4147999184 wait complete. 2024-04-22T13:33:33.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:33 smithi120 ceph-mon[31101]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:33.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:33 smithi132 ceph-mon[31531]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:33.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:33 smithi087 ceph-mon[27999]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T13:33:34.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:33:34.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.340+0000 7fcf718b3640 1 -- 172.21.15.87:0/2953268215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c101080 msgr2=0x7fcf6c101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.340+0000 7fcf718b3640 1 --2- 172.21.15.87:0/2953268215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c101080 0x7fcf6c101460 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009ca0 tx=0x7fcf5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 -- 172.21.15.87:0/2953268215 shutdown_connections 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 --2- 172.21.15.87:0/2953268215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c106150 0x7fcf6c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 --2- 172.21.15.87:0/2953268215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101a30 0x7fcf6c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 --2- 172.21.15.87:0/2953268215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c101080 0x7fcf6c101460 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 -- 172.21.15.87:0/2953268215 >> 172.21.15.87:0/2953268215 conn(0x7fcf6c078000 msgr2=0x7fcf6c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 -- 172.21.15.87:0/2953268215 shutdown_connections 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 -- 172.21.15.87:0/2953268215 wait complete. 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 Processor -- start 2024-04-22T13:33:34.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.341+0000 7fcf718b3640 1 -- start start 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 0x7fcf6c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c19cff0 0x7fcf6c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1110c0 con 0x7fcf6c106150 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c110f40 con 0x7fcf6c101080 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c111240 con 0x7fcf6c19cff0 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43908/0 (socket says 172.21.15.87:43908) 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/1791405961 learned_addr learned my addr 172.21.15.87:0/1791405961 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c19cff0 0x7fcf6c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6affd640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 0x7fcf6c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/1791405961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c19cff0 msgr2=0x7fcf6c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c19cff0 0x7fcf6c19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/1791405961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 msgr2=0x7fcf6c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 0x7fcf6c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/1791405961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6c19dae0 con 0x7fcf6c106150 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6affd640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 0x7fcf6c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000c970 tx=0x7fcf6000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf708b1640 1 -- 172.21.15.87:0/1791405961 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60014070 con 0x7fcf6c106150 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6c19ddd0 con 0x7fcf6c106150 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf708b1640 1 -- 172.21.15.87:0/1791405961 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf60004440 con 0x7fcf6c106150 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf708b1640 1 -- 172.21.15.87:0/1791405961 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60003e50 con 0x7fcf6c106150 2024-04-22T13:33:34.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.342+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6c1a5610 con 0x7fcf6c106150 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf708b1640 1 -- 172.21.15.87:0/1791405961 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcf60017070 con 0x7fcf6c106150 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 Processor -- start 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 -- start start 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a6047d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a60488e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558a603e21e0 con 0x558a6047dea0 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf708b1640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 0x7fcf540a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a6037f390 con 0x558a6047d580 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf7457f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a603de1a0 con 0x558a60486a80 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf6affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf6affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43914/0 (socket says 172.21.15.87:43914) 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf6affd640 1 -- 172.21.15.87:0/559767623 learned_addr learned my addr 172.21.15.87:0/559767623 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.345+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 0x7fcf540a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf708b1640 1 -- 172.21.15.87:0/1791405961 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf600d03d0 con 0x7fcf6c106150 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a60488e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a6047d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 -- 172.21.15.87:0/559767623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 msgr2=0x558a60488e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a60488e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 -- 172.21.15.87:0/559767623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 msgr2=0x558a6047d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a6047d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 -- 172.21.15.87:0/559767623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a601be7d0 con 0x558a6047dea0 2024-04-22T13:33:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a6047d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a60488e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6affd640 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c006000 tx=0x7fcf5c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8a65ba52119f67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 0x7fcf540a6380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009d20 tx=0x7fcf64009820 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf51ffb640 1 -- 172.21.15.87:0/559767623 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c039050 con 0x558a6047dea0 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf51ffb640 1 -- 172.21.15.87:0/559767623 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf5c045070 con 0x558a6047dea0 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.346+0000 7fcf51ffb640 1 -- 172.21.15.87:0/559767623 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c004040 con 0x558a6047dea0 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- 172.21.15.87:0/559767623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 msgr2=0x558a60486540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c006000 tx=0x7fcf5c009800 comp rx=0 tx=0).stop 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- 172.21.15.87:0/559767623 shutdown_connections 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a60488e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60486540 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 --2- 172.21.15.87:0/559767623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a6047d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- 172.21.15.87:0/559767623 >> 172.21.15.87:0/559767623 conn(0x558a60477cb0 msgr2=0x558a60478090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- 172.21.15.87:0/559767623 shutdown_connections 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- 172.21.15.87:0/559767623 wait complete. 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 -- start start 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.347+0000 7fcf7457f740 1 Processor -- start 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60483f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a60486a80 0x558a60484470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558a60489810 con 0x558a6047d580 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a6039b2f0 con 0x558a60486a80 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a60489990 con 0x558a6047dea0 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 10 client.0 _ll_register_callbacks cb 0x558a603a8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43916/0 (socket says 172.21.15.87:43916) 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 -- 172.21.15.87:0/710419789 learned_addr learned my addr 172.21.15.87:0/710419789 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a60486a80 0x558a60484470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 -- 172.21.15.87:0/710419789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 msgr2=0x558a60483f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60483f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60483f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 -- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a60486a80 msgr2=0x558a60484470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a60486a80 0x558a60484470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 -- 172.21.15.87:0/710419789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a600f43f0 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf6affd640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c02c700 tx=0x7fcf5c003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90755f3fd1ada839 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf4affd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c050070 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf5c02c770 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.348+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf5c045070 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf5c05b050 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 10 client.16209 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558a600b40f0 con 0x558a6047d580 2024-04-22T13:33:34.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf5c045070 con 0x558a6047d580 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf5c0347a0 con 0x558a6047d580 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 10 client.16209 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 10 client.16209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 20 client.16209 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a60496770 0x558a60498b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 4 client.16209 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a603efc00 con 0x558a60496770 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.349+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a60496770 0x558a60498b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.350+0000 7fcf4affd640 10 client.16209 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.350+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a60496770 0x558a60498b60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ddb20fe433a2558 server_cookie=79968bc92f0e49b6 in_seq=0 out_seq=0 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.350+0000 7fcf4affd640 1 -- 172.21.15.87:0/710419789 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a603efc00 con 0x558a60496770 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.350+0000 7fcf4affd640 10 client.16209 handle_command_reply: tid=0 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.350+0000 7fcf7457f740 1 client.16209 shutdown 2024-04-22T13:33:34.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 msgr2=0x558a604839f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c02c700 tx=0x7fcf5c003b10 comp rx=0 tx=0).stop 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 shutdown_connections 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a60496770 0x558a60498b60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a60486a80 0x558a60484470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf4affd640 0 client.16209 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60483f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 --2- 172.21.15.87:0/710419789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047d580 0x558a604839f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 >> 172.21.15.87:0/710419789 conn(0x558a60477cb0 msgr2=0x558a60479a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 shutdown_connections 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 1 -- 172.21.15.87:0/710419789 wait complete. 2024-04-22T13:33:34.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.351+0000 7fcf7457f740 20 client.16209 trim_cache size 0 max 16384 2024-04-22T13:33:34.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 Processor -- start 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 -- start start 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a60486120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a6049e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1b0240 con 0x558a6047dea0 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1adb10 con 0x558a6047d580 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf7457f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1ab960 con 0x558a60486a80 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43922/0 (socket says 172.21.15.87:43922) 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/631911155 learned_addr learned my addr 172.21.15.87:0/631911155 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/631911155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 msgr2=0x558a6049e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a6049e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6affd640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a60486120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a6049e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/631911155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 msgr2=0x558a60486120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a60486120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 -- 172.21.15.87:0/631911155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a6047dcd0 con 0x558a6047dea0 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a6049e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcf60098db0 tx=0x7fcf60091870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddce75b643edde43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.369+0000 7fcf517fa640 1 -- 172.21.15.87:0/631911155 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6009c3e0 con 0x558a6047dea0 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf517fa640 1 -- 172.21.15.87:0/631911155 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf600a0790 con 0x558a6047dea0 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf517fa640 1 -- 172.21.15.87:0/631911155 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60098a50 con 0x558a6047dea0 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 -- 172.21.15.87:0/631911155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 msgr2=0x558a60492700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcf60098db0 tx=0x7fcf60091870 comp rx=0 tx=0).stop 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 -- 172.21.15.87:0/631911155 shutdown_connections 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a60486a80 0x558a6049e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a6047dea0 0x558a60492700 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 --2- 172.21.15.87:0/631911155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a60486120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 -- 172.21.15.87:0/631911155 >> 172.21.15.87:0/631911155 conn(0x558a60477cb0 msgr2=0x558a60482a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 -- 172.21.15.87:0/631911155 shutdown_connections 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.370+0000 7fcf7457f740 1 -- 172.21.15.87:0/631911155 wait complete. 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 -- start start 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 Processor -- start 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a604805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60480b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1abfd0 con 0x558a60486a80 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1ad440 con 0x558a6047d580 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1ac6a0 con 0x558a6047dea0 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf7457f740 10 client.0 _ll_register_callbacks cb 0x7fcf64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43928/0 (socket says 172.21.15.87:43928) 2024-04-22T13:33:34.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 -- 172.21.15.87:0/2778706560 learned_addr learned my addr 172.21.15.87:0/2778706560 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6affd640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a604805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60480b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 -- 172.21.15.87:0/2778706560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 msgr2=0x558a60480b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60480b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 -- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 msgr2=0x558a604805e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a604805e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6b7fe640 1 -- 172.21.15.87:0/2778706560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a604fba80 con 0x558a60486a80 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6a7fc640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60480b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.371+0000 7fcf6affd640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a604805e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf6b7fe640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009460 tx=0x7fcf640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaf2abe61832de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf64004050 con 0x558a60486a80 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf640041f0 con 0x558a60486a80 2024-04-22T13:33:34.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf600a0790 con 0x558a60486a80 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf640043f0 con 0x558a60486a80 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 10 client.16215 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558a604fcc90 con 0x558a60486a80 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf600a0790 con 0x558a60486a80 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf6402f050 con 0x558a60486a80 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 10 client.16215 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 20 client.16215 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a604fd460 0x558a604ff880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 4 client.16215 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.372+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a603e9a70 con 0x558a604fd460 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf6affd640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a604fd460 0x558a604ff880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf4affd640 10 client.16215 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf6affd640 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a604fd460 0x558a604ff880 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25e6ec8a9fcbe9b6 server_cookie=f3505ce874caa71 in_seq=0 out_seq=0 2024-04-22T13:33:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf4affd640 1 -- 172.21.15.87:0/2778706560 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558a603e9a70 con 0x558a604fd460 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf4affd640 10 client.16215 handle_command_reply: tid=0 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.373+0000 7fcf7457f740 1 client.16215 shutdown 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 msgr2=0x558a60481060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009460 tx=0x7fcf640046c0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 shutdown_connections 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558a604fd460 0x558a604ff880 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558a60486a80 0x558a60481060 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a6047dea0 0x558a60480b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf4affd640 0 client.16215 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 --2- 172.21.15.87:0/2778706560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a6047d580 0x558a604805e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 >> 172.21.15.87:0/2778706560 conn(0x558a60477cb0 msgr2=0x558a60482190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 shutdown_connections 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 1 -- 172.21.15.87:0/2778706560 wait complete. 2024-04-22T13:33:34.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf7457f740 20 client.16215 trim_cache size 0 max 16384 2024-04-22T13:33:34.377 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:34.377 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:33:34.377 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:33:34.377 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 msgr2=0x7fcf540a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 0x7fcf540a6380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009d20 tx=0x7fcf64009820 comp rx=0 tx=0).stop 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 msgr2=0x7fcf6c198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000c970 tx=0x7fcf6000ce30 comp rx=0 tx=0).stop 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 shutdown_connections 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf540a3ec0 0x7fcf540a6380 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c19cff0 0x7fcf6c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf6c106150 0x7fcf6c198c70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 --2- 172.21.15.87:0/1791405961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf6c101080 0x7fcf6c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 >> 172.21.15.87:0/1791405961 conn(0x7fcf6c078000 msgr2=0x7fcf6c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 shutdown_connections 2024-04-22T13:33:34.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.374+0000 7fcf718b3640 1 -- 172.21.15.87:0/1791405961 wait complete. 2024-04-22T13:33:34.381 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:33:34.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.484+0000 7f4880406640 1 -- 172.21.15.87:0/2068089307 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 msgr2=0x7f48781083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- 172.21.15.87:0/2068089307 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f48781083b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4874009930 tx=0x7f487402bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- 172.21.15.87:0/2068089307 shutdown_connections 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- 172.21.15.87:0/2068089307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f487810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- 172.21.15.87:0/2068089307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- 172.21.15.87:0/2068089307 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f48781083b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- 172.21.15.87:0/2068089307 >> 172.21.15.87:0/2068089307 conn(0x7f4878077e30 msgr2=0x7f4878078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- 172.21.15.87:0/2068089307 shutdown_connections 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- 172.21.15.87:0/2068089307 wait complete. 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 Processor -- start 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- start start 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f4878198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4878112120 con 0x7f4878107fd0 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4878111fa0 con 0x7f4878069aa0 2024-04-22T13:33:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.485+0000 7f4880406640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f48781122a0 con 0x7f48780690e0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43940/0 (socket says 172.21.15.87:43940) 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 -- 172.21.15.87:0/4030874095 learned_addr learned my addr 172.21.15.87:0/4030874095 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487d9a1640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f4878198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e1a2640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 -- 172.21.15.87:0/4030874095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 msgr2=0x7f4878198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 -- 172.21.15.87:0/4030874095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 msgr2=0x7f4878198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f4878198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 -- 172.21.15.87:0/4030874095 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487819d5f0 con 0x7f4878107fd0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e1a2640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487d9a1640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f4878198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f487e9a3640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f487000cce0 tx=0x7f487000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487000f450 con 0x7f4878107fd0 2024-04-22T13:33:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f487000faa0 con 0x7f4878107fd0 2024-04-22T13:33:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487819d8e0 con 0x7f4878107fd0 2024-04-22T13:33:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870018b70 con 0x7f4878107fd0 2024-04-22T13:33:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.486+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48781a5170 con 0x7f4878107fd0 2024-04-22T13:33:34.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.487+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4878199950 con 0x7f4878107fd0 2024-04-22T13:33:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.489+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4870018d10 con 0x7f4878107fd0 2024-04-22T13:33:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.489+0000 7f486f7fe640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 0x7f48640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.489+0000 7f487e1a2640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 0x7f48640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.489+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f48700d0d40 con 0x7f4878107fd0 2024-04-22T13:33:34.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.490+0000 7f487e1a2640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 0x7f48640a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4874006010 tx=0x7f4874009610 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.493+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4870014030 con 0x7f4878107fd0 2024-04-22T13:33:34.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.708+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4878109ea0 con 0x7f4878107fd0 2024-04-22T13:33:34.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.709+0000 7f486f7fe640 1 -- 172.21.15.87:0/4030874095 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f487009d4e0 con 0x7f4878107fd0 2024-04-22T13:33:34.705 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:34.705 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:34.705 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 msgr2=0x7f48640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 0x7f48640a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4874006010 tx=0x7f4874009610 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 msgr2=0x7f487819cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f487000cce0 tx=0x7f487000a620 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 shutdown_connections 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48640a3d20 0x7f48640a61e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4878107fd0 0x7f487819cec0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4878069aa0 0x7f4878198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 --2- 172.21.15.87:0/4030874095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48780690e0 0x7f4878198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 >> 172.21.15.87:0/4030874095 conn(0x7f4878077e30 msgr2=0x7f4878100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 shutdown_connections 2024-04-22T13:33:34.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.711+0000 7f4880406640 1 -- 172.21.15.87:0/4030874095 wait complete. 2024-04-22T13:33:34.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:33:34.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- 172.21.15.87:0/3287737230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb360103b00 msgr2=0x7fb36010bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 --2- 172.21.15.87:0/3287737230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb360103b00 0x7fb36010bfe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb354009900 tx=0x7fb35402bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- 172.21.15.87:0/3287737230 shutdown_connections 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 --2- 172.21.15.87:0/3287737230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 0x7fb36010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 --2- 172.21.15.87:0/3287737230 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb360103b00 0x7fb36010bfe0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 --2- 172.21.15.87:0/3287737230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3601031e0 0x7fb3601035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- 172.21.15.87:0/3287737230 >> 172.21.15.87:0/3287737230 conn(0x7fb3600771d0 msgr2=0x7fb3600775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- 172.21.15.87:0/3287737230 shutdown_connections 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- 172.21.15.87:0/3287737230 wait complete. 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 Processor -- start 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.823+0000 7fb36534b640 1 -- start start 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb360103b00 0x7fb3601a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 0x7fb36019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb360112190 con 0x7fb3601031e0 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb360112010 con 0x7fb360103b00 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb360112310 con 0x7fb36010c520 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43950/0 (socket says 172.21.15.87:43950) 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 -- 172.21.15.87:0/4079128440 learned_addr learned my addr 172.21.15.87:0/4079128440 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb360103b00 0x7fb3601a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 0x7fb36019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 -- 172.21.15.87:0/4079128440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 msgr2=0x7fb36019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 0x7fb36019a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 -- 172.21.15.87:0/4079128440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb360103b00 msgr2=0x7fb3601a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb360103b00 0x7fb3601a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 -- 172.21.15.87:0/4079128440 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36019b070 con 0x7fb3601031e0 2024-04-22T13:33:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb35effd640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb35000d950 tx=0x7fb35000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb347fff640 1 -- 172.21.15.87:0/4079128440 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350017030 con 0x7fb3601031e0 2024-04-22T13:33:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb347fff640 1 -- 172.21.15.87:0/4079128440 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb350004020 con 0x7fb3601031e0 2024-04-22T13:33:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36019b360 con 0x7fb3601031e0 2024-04-22T13:33:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb347fff640 1 -- 172.21.15.87:0/4079128440 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500047a0 con 0x7fb3601031e0 2024-04-22T13:33:34.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.824+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3601a7620 con 0x7fb3601031e0 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.826+0000 7fb347fff640 1 -- 172.21.15.87:0/4079128440 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb350009a30 con 0x7fb3601031e0 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 Processor -- start 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 -- start start 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56229ea521f0 con 0x56229eaed4d0 2024-04-22T13:33:34.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56229e9ef4b0 con 0x56229eaf6a00 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb368015740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56229ea4e1b0 con 0x56229eaede80 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43956/0 (socket says 172.21.15.87:43956) 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 -- 172.21.15.87:0/2458485455 learned_addr learned my addr 172.21.15.87:0/2458485455 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35effd640 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb347fff640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 0x7fb3480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 0x7fb3480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 -- 172.21.15.87:0/2458485455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 msgr2=0x56229eaf6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 -- 172.21.15.87:0/2458485455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 msgr2=0x56229eaf8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.827+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb35f7fe640 1 -- 172.21.15.87:0/2458485455 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229e9e11b0 con 0x56229eaed4d0 2024-04-22T13:33:34.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb347fff640 1 -- 172.21.15.87:0/4079128440 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb3500d0300 con 0x7fb3601031e0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb358013840 tx=0x7fb358013d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50abce8abc5bece1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb3457fa640 1 -- 172.21.15.87:0/2458485455 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358013250 con 0x56229eaed4d0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb3457fa640 1 -- 172.21.15.87:0/2458485455 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb358004030 con 0x56229eaed4d0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb3457fa640 1 -- 172.21.15.87:0/2458485455 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35801d640 con 0x56229eaed4d0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.828+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 0x7fb3480a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb358020000 tx=0x7fb35801e000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- 172.21.15.87:0/2458485455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 msgr2=0x56229eaed8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb358013840 tx=0x7fb358013d00 comp rx=0 tx=0).stop 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- 172.21.15.87:0/2458485455 shutdown_connections 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- 172.21.15.87:0/2458485455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaed8b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- 172.21.15.87:0/2458485455 >> 172.21.15.87:0/2458485455 conn(0x56229eae7ba0 msgr2=0x56229eae7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- 172.21.15.87:0/2458485455 shutdown_connections 2024-04-22T13:33:34.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- 172.21.15.87:0/2458485455 wait complete. 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- start start 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 Processor -- start 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56229eaf9970 con 0x56229eaed4d0 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56229ea0b500 con 0x56229eaf6a00 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56229eaf9af0 con 0x56229eaede80 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb368015740 10 client.0 _ll_register_callbacks cb 0x56229e9d9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb35effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb35effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43960/0 (socket says 172.21.15.87:43960) 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb35effd640 1 -- 172.21.15.87:0/3824721297 learned_addr learned my addr 172.21.15.87:0/3824721297 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb35effd640 1 -- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 msgr2=0x56229eaf3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.829+0000 7fb35effd640 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb35effd640 1 -- 172.21.15.87:0/3824721297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 msgr2=0x56229eaf3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb35effd640 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb35effd640 1 -- 172.21.15.87:0/3824721297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229e7643f0 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb35effd640 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb350098710 tx=0x7fb350097f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ea47f0028c0850 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500a0660 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb350098920 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb358004030 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb3500a4f20 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 10 client.16233 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56229e82e7d0 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb358004030 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb350098000 con 0x56229eaed4d0 2024-04-22T13:33:34.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 10 client.16233 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 10 client.16233 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 20 client.16233 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb06bd0 0x56229eb08fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 4 client.16233 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.830+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56229e9f1930 con 0x56229eb06bd0 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.831+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb06bd0 0x56229eb08fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.831+0000 7fb342ff5640 10 client.16233 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.831+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb06bd0 0x56229eb08fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c19b0447ca032a3f server_cookie=98ec17c8846ce1dc in_seq=0 out_seq=0 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb342ff5640 1 -- 172.21.15.87:0/3824721297 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56229e9f1930 con 0x56229eb06bd0 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb342ff5640 10 client.16233 handle_command_reply: tid=0 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 client.16233 shutdown 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 msgr2=0x56229eaf6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb350098710 tx=0x7fb350097f40 comp rx=0 tx=0).stop 2024-04-22T13:33:34.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 shutdown_connections 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb06bd0 0x56229eb08fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaf6a00 0x56229eaf3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaede80 0x56229eaf3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb342ff5640 0 client.16233 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 --2- 172.21.15.87:0/3824721297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaed4d0 0x56229eaf6290 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 >> 172.21.15.87:0/3824721297 conn(0x56229eae7ba0 msgr2=0x56229eae98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 shutdown_connections 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 1 -- 172.21.15.87:0/3824721297 wait complete. 2024-04-22T13:33:34.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.832+0000 7fb368015740 20 client.16233 trim_cache size 0 max 16384 2024-04-22T13:33:34.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 Processor -- start 2024-04-22T13:33:34.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 -- start start 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaed4d0 0x56229eb02830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 0x56229eaf5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601b2580 con 0x56229eaf6a00 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601afe50 con 0x56229eaede80 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb368015740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601adca0 con 0x56229eaed4d0 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.845+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43962/0 (socket says 172.21.15.87:43962) 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 -- 172.21.15.87:0/462620155 learned_addr learned my addr 172.21.15.87:0/462620155 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35effd640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaed4d0 0x56229eb02830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 -- 172.21.15.87:0/462620155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaed4d0 msgr2=0x56229eb02830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 0x56229eaf5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaed4d0 0x56229eb02830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 -- 172.21.15.87:0/462620155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 msgr2=0x56229eaf5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 0x56229eaf5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 -- 172.21.15.87:0/462620155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229eaed0b0 con 0x56229eaf6a00 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 0x56229eaf5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb358011750 tx=0x7fb358009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b11e9e07ddce068 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb344ff9640 1 -- 172.21.15.87:0/462620155 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358013050 con 0x56229eaf6a00 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb344ff9640 1 -- 172.21.15.87:0/462620155 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3580131f0 con 0x56229eaf6a00 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.846+0000 7fb344ff9640 1 -- 172.21.15.87:0/462620155 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35801d840 con 0x56229eaf6a00 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- 172.21.15.87:0/462620155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 msgr2=0x56229eaf6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb358011750 tx=0x7fb358009de0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- 172.21.15.87:0/462620155 shutdown_connections 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaf6a00 0x56229eaf6290 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaede80 0x56229eaf5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- 172.21.15.87:0/462620155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaed4d0 0x56229eb02830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- 172.21.15.87:0/462620155 >> 172.21.15.87:0/462620155 conn(0x56229eae7ba0 msgr2=0x56229eae6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- 172.21.15.87:0/462620155 shutdown_connections 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- 172.21.15.87:0/462620155 wait complete. 2024-04-22T13:33:34.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- start start 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 Processor -- start 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaed4d0 0x56229eafef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 0x56229eafa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601ae310 con 0x56229eaede80 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601af780 con 0x56229eaed4d0 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601ae9e0 con 0x56229eaf6a00 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb368015740 10 client.0 _ll_register_callbacks cb 0x7fb35009bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb35e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb35e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43966/0 (socket says 172.21.15.87:43966) 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb35e7fc640 1 -- 172.21.15.87:0/3485080679 learned_addr learned my addr 172.21.15.87:0/3485080679 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.847+0000 7fb35effd640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaed4d0 0x56229eafef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 0x56229eafa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 -- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 msgr2=0x56229eafa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 0x56229eafa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 -- 172.21.15.87:0/3485080679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaed4d0 msgr2=0x56229eafef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaed4d0 0x56229eafef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 -- 172.21.15.87:0/3485080679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229eafadf0 con 0x56229eaede80 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35f7fe640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 0x56229eafa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb35e7fc640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb35402c8b0 tx=0x7fb354004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f3c7dc6dc15839 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35404e060 con 0x56229eaede80 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb35402c910 con 0x56229eaede80 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3580131f0 con 0x56229eaede80 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb35402cb10 con 0x56229eaede80 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 10 client.16239 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56229eb6cea0 con 0x56229eaede80 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3580131f0 con 0x56229eaede80 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb354051030 con 0x56229eaede80 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 10 client.16239 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 10 client.16239 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 20 client.16239 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb6d600 0x56229eb6fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 4 client.16239 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.848+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56229ea5fc10 con 0x56229eb6d600 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb35effd640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb6d600 0x56229eb6fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb35effd640 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb6d600 0x56229eb6fa20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d26caaf66ee8f716 server_cookie=f67dbdc9187a8613 in_seq=0 out_seq=0 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb342ff5640 10 client.16239 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb342ff5640 1 -- 172.21.15.87:0/3485080679 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56229ea5fc10 con 0x56229eb6d600 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb342ff5640 10 client.16239 handle_command_reply: tid=0 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 client.16239 shutdown 2024-04-22T13:33:34.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 msgr2=0x56229eafa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb35402c8b0 tx=0x7fb354004320 comp rx=0 tx=0).stop 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 shutdown_connections 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56229eb6d600 0x56229eb6fa20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56229eaf6a00 0x56229eafa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb342ff5640 0 client.16239 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56229eaede80 0x56229eafa2c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 --2- 172.21.15.87:0/3485080679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56229eaed4d0 0x56229eafef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 >> 172.21.15.87:0/3485080679 conn(0x56229eae7ba0 msgr2=0x56229eaf2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 shutdown_connections 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.849+0000 7fb368015740 1 -- 172.21.15.87:0/3485080679 wait complete. 2024-04-22T13:33:34.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb368015740 20 client.16239 trim_cache size 0 max 16384 2024-04-22T13:33:34.856 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 msgr2=0x7fb3480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 0x7fb3480a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb358020000 tx=0x7fb35801e000 comp rx=0 tx=0).stop 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 msgr2=0x7fb3601a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb35000d950 tx=0x7fb35000de10 comp rx=0 tx=0).stop 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 shutdown_connections 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3480a3d20 0x7fb3480a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb36010c520 0x7fb36019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb360103b00 0x7fb3601a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 --2- 172.21.15.87:0/4079128440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3601031e0 0x7fb3601a06a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 >> 172.21.15.87:0/4079128440 conn(0x7fb3600771d0 msgr2=0x7fb360100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 shutdown_connections 2024-04-22T13:33:34.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.850+0000 7fb36534b640 1 -- 172.21.15.87:0/4079128440 wait complete. 2024-04-22T13:33:34.858 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:33:34.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 -- 172.21.15.87:0/2447768682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc107270 msgr2=0x7f14bc10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 --2- 172.21.15.87:0/2447768682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc107270 0x7f14bc10ee40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009ca0 tx=0x7f14b402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 -- 172.21.15.87:0/2447768682 shutdown_connections 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 --2- 172.21.15.87:0/2447768682 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc107270 0x7f14bc10ee40 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 --2- 172.21.15.87:0/2447768682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc06b1f0 0x7f14bc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 --2- 172.21.15.87:0/2447768682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc06a840 0x7f14bc06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 -- 172.21.15.87:0/2447768682 >> 172.21.15.87:0/2447768682 conn(0x7f14bc077eb0 msgr2=0x7f14bc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 -- 172.21.15.87:0/2447768682 shutdown_connections 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.963+0000 7f14c0f52640 1 -- 172.21.15.87:0/2447768682 wait complete. 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 Processor -- start 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 -- start start 2024-04-22T13:33:34.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc107270 0x7f14bc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 0x7f14bc19d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1121b0 con 0x7f14bc06a840 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc112030 con 0x7f14bc107270 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14c0f52640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc112330 con 0x7f14bc19ceb0 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43976/0 (socket says 172.21.15.87:43976) 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 -- 172.21.15.87:0/92443415 learned_addr learned my addr 172.21.15.87:0/92443415 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 -- 172.21.15.87:0/92443415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 msgr2=0x7f14bc19d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14bad9d640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 0x7f14bc19d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14b3fff640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc107270 0x7f14bc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 0x7f14bc19d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 -- 172.21.15.87:0/92443415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc107270 msgr2=0x7f14bc198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc107270 0x7f14bc198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14ba59c640 1 -- 172.21.15.87:0/92443415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc19d9a0 con 0x7f14bc06a840 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.964+0000 7f14bad9d640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 0x7f14bc19d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14ba59c640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00c920 tx=0x7f14ac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14b37fe640 1 -- 172.21.15.87:0/92443415 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac007c70 con 0x7f14bc06a840 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14b37fe640 1 -- 172.21.15.87:0/92443415 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14ac007e10 con 0x7f14bc06a840 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc19dc90 con 0x7f14bc06a840 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14b37fe640 1 -- 172.21.15.87:0/92443415 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac018820 con 0x7f14bc06a840 2024-04-22T13:33:34.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.965+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc1a54d0 con 0x7f14bc06a840 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.967+0000 7f14b37fe640 1 -- 172.21.15.87:0/92443415 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f14ac00f450 con 0x7f14bc06a840 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.967+0000 7f14c3c22740 1 Processor -- start 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 -- start start 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14b37fe640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 0x7f14a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a0e80 0x55f1f72a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f1f72051f0 con 0x55f1f72a9a00 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1f71a24b0 con 0x55f1f72a0e80 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14c3c22740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1f72011b0 con 0x55f1f72a04d0 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14ba59c640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 0x7f14a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43980/0 (socket says 172.21.15.87:43980) 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 -- 172.21.15.87:0/2202373621 learned_addr learned my addr 172.21.15.87:0/2202373621 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a0e80 0x55f1f72a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14b37fe640 1 -- 172.21.15.87:0/92443415 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f14ac0d0930 con 0x7f14bc06a840 2024-04-22T13:33:34.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 -- 172.21.15.87:0/2202373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 msgr2=0x55f1f72a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 -- 172.21.15.87:0/2202373621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a0e80 msgr2=0x55f1f72a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a0e80 0x55f1f72a9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 -- 172.21.15.87:0/2202373621 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f71941b0 con 0x55f1f72a9a00 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009a60 tx=0x7f14b4009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256b99a3e0b9bd71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14ba59c640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 0x7f14a80a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0ce6f0 tx=0x7f14ac087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.968+0000 7f14b0ff9640 1 -- 172.21.15.87:0/2202373621 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4048070 con 0x55f1f72a9a00 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14b0ff9640 1 -- 172.21.15.87:0/2202373621 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b403a040 con 0x55f1f72a9a00 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14b0ff9640 1 -- 172.21.15.87:0/2202373621 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4039030 con 0x55f1f72a9a00 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 -- 172.21.15.87:0/2202373621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 msgr2=0x55f1f72abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009a60 tx=0x7f14b4009380 comp rx=0 tx=0).stop 2024-04-22T13:33:34.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 -- 172.21.15.87:0/2202373621 shutdown_connections 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72abe50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a0e80 0x55f1f72a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2202373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 -- 172.21.15.87:0/2202373621 >> 172.21.15.87:0/2202373621 conn(0x55f1f729aba0 msgr2=0x55f1f729af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 -- 172.21.15.87:0/2202373621 shutdown_connections 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.969+0000 7f14c3c22740 1 -- 172.21.15.87:0/2202373621 wait complete. 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 -- start start 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 Processor -- start 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f72a9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f72a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f1f72ac910 con 0x55f1f72a9a00 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1f71be500 con 0x55f1f72a04d0 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1f72aca90 con 0x55f1f72a0e80 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14c3c22740 10 client.0 _ll_register_callbacks cb 0x55f1f718cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43982/0 (socket says 172.21.15.87:43982) 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 -- 172.21.15.87:0/478311247 learned_addr learned my addr 172.21.15.87:0/478311247 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f72a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14b3fff640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f72a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 -- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 msgr2=0x55f1f72a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f72a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 -- 172.21.15.87:0/478311247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 msgr2=0x55f1f72a9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f72a9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.970+0000 7f14bad9d640 1 -- 172.21.15.87:0/478311247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f6f173f0 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14bad9d640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f14b4002230 tx=0x7f14b4032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548067c9a23b7a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4039030 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f14b4004310 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b403a040 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f14b4048040 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 10 client.16251 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f1f6fe17d0 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b403a040 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f14b4058020 con 0x55f1f72a9a00 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 10 client.16251 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 10 client.16251 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:33:34.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 20 client.16251 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f72b9a60 0x55f1f72bbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 4 client.16251 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1f71a4930 con 0x55f1f72b9a60 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.971+0000 7f14b3fff640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f72b9a60 0x55f1f72bbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.972+0000 7f14a1ffb640 10 client.16251 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.972+0000 7f14b3fff640 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f72b9a60 0x55f1f72bbe50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6894e3646b9ebce server_cookie=eb445fb2fec00634 in_seq=0 out_seq=0 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14a1ffb640 1 -- 172.21.15.87:0/478311247 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1f71a4930 con 0x55f1f72b9a60 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14a1ffb640 10 client.16251 handle_command_reply: tid=0 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 client.16251 shutdown 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 msgr2=0x55f1f72a6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f14b4002230 tx=0x7f14b4032960 comp rx=0 tx=0).stop 2024-04-22T13:33:34.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 shutdown_connections 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f72b9a60 0x55f1f72bbe50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f72a6d00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f72a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14a1ffb640 0 client.16251 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 --2- 172.21.15.87:0/478311247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f72a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 >> 172.21.15.87:0/478311247 conn(0x55f1f729aba0 msgr2=0x55f1f729c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 shutdown_connections 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 1 -- 172.21.15.87:0/478311247 wait complete. 2024-04-22T13:33:34.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.973+0000 7f14c3c22740 20 client.16251 trim_cache size 0 max 16384 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 Processor -- start 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 -- start start 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72b5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a9a00 0x55f1f72a9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1b01e0 con 0x55f1f72a0e80 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1adab0 con 0x55f1f72a9a00 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14c3c22740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1ab920 con 0x55f1f72a04d0 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43996/0 (socket says 172.21.15.87:43996) 2024-04-22T13:33:34.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 -- 172.21.15.87:0/2980132809 learned_addr learned my addr 172.21.15.87:0/2980132809 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a9a00 0x55f1f72a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 -- 172.21.15.87:0/2980132809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 msgr2=0x55f1f72b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 -- 172.21.15.87:0/2980132809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a9a00 msgr2=0x55f1f72a9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a9a00 0x55f1f72a9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14b3fff640 1 -- 172.21.15.87:0/2980132809 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f72aca30 con 0x55f1f72a0e80 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.986+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f14bc06a420 tx=0x7f14a4009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e424fb78df7486a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14a3fff640 1 -- 172.21.15.87:0/2980132809 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a401f030 con 0x55f1f72a0e80 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14a3fff640 1 -- 172.21.15.87:0/2980132809 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a40054d0 con 0x55f1f72a0e80 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14a3fff640 1 -- 172.21.15.87:0/2980132809 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a401b8b0 con 0x55f1f72a0e80 2024-04-22T13:33:34.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 -- 172.21.15.87:0/2980132809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 msgr2=0x55f1f72a8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f14bc06a420 tx=0x7f14a4009350 comp rx=0 tx=0).stop 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 -- 172.21.15.87:0/2980132809 shutdown_connections 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a9a00 0x55f1f72a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a0e80 0x55f1f72a8e20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2980132809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a04d0 0x55f1f72b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 -- 172.21.15.87:0/2980132809 >> 172.21.15.87:0/2980132809 conn(0x55f1f729aba0 msgr2=0x55f1f72a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 -- 172.21.15.87:0/2980132809 shutdown_connections 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.987+0000 7f14c3c22740 1 -- 172.21.15.87:0/2980132809 wait complete. 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 -- start start 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 Processor -- start 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f731c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f731ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1abf70 con 0x55f1f72a9a00 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1ad3e0 con 0x55f1f72a04d0 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1ac640 con 0x55f1f72a0e80 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14c3c22740 10 client.0 _ll_register_callbacks cb 0x7f14b40324c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44012/0 (socket says 172.21.15.87:44012) 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 -- 172.21.15.87:0/2183509574 learned_addr learned my addr 172.21.15.87:0/2183509574 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f731c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f731ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 -- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 msgr2=0x55f1f731ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f731ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 -- 172.21.15.87:0/2183509574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 msgr2=0x55f1f731c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f731c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.988+0000 7f14bad9d640 1 -- 172.21.15.87:0/2183509574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f7317440 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14bad9d640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f80 tx=0x7f14b4032e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ad7776f44be44 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4039040 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f14b40344e0 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14a40054d0 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f14b40346e0 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 10 client.16257 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f1f7324140 con 0x55f1f72a9a00 2024-04-22T13:33:34.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a40054d0 con 0x55f1f72a9a00 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f14b4058040 con 0x55f1f72a9a00 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 10 client.16257 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 10 client.16257 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 20 client.16257 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f7324610 0x55f1f7326a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 4 client.16257 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1f7212c10 con 0x55f1f7324610 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.989+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f7324610 0x55f1f7326a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14b3fff640 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f7324610 0x55f1f7326a00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b7a2b29b568bf0b server_cookie=abed7a9eb8674adc in_seq=0 out_seq=0 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14a1ffb640 10 client.16257 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14a1ffb640 1 -- 172.21.15.87:0/2183509574 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1f7212c10 con 0x55f1f7324610 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14a1ffb640 10 client.16257 handle_command_reply: tid=0 2024-04-22T13:33:34.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 client.16257 shutdown 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 msgr2=0x55f1f7316d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f80 tx=0x7f14b4032e50 comp rx=0 tx=0).stop 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 shutdown_connections 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f1f7324610 0x55f1f7326a00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1f72a9a00 0x55f1f7316d50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14a1ffb640 0 client.16257 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1f72a0e80 0x55f1f731ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 --2- 172.21.15.87:0/2183509574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1f72a04d0 0x55f1f731c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 >> 172.21.15.87:0/2183509574 conn(0x55f1f729aba0 msgr2=0x55f1f72b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 shutdown_connections 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 1 -- 172.21.15.87:0/2183509574 wait complete. 2024-04-22T13:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.990+0000 7f14c3c22740 20 client.16257 trim_cache size 0 max 16384 2024-04-22T13:33:34.995 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 msgr2=0x7f14a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 0x7f14a80a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0ce6f0 tx=0x7f14ac087040 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 msgr2=0x7f14bc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00c920 tx=0x7f14ac00cde0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 shutdown_connections 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f14a80a3da0 0x7f14a80a6260 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14bc19ceb0 0x7f14bc19d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14bc107270 0x7f14bc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 --2- 172.21.15.87:0/92443415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14bc06a840 0x7f14bc1985f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:34.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 >> 172.21.15.87:0/92443415 conn(0x7f14bc077eb0 msgr2=0x7f14bc0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:34.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 shutdown_connections 2024-04-22T13:33:34.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:34.991+0000 7f14c0f52640 1 -- 172.21.15.87:0/92443415 wait complete. 2024-04-22T13:33:34.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:33:35.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.101+0000 7f2d6e72e640 1 -- 172.21.15.87:0/1728426559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d681031d0 msgr2=0x7f2d681035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.101+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/1728426559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d681031d0 0x7f2d681035b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009900 tx=0x7f2d5802bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.101+0000 7f2d6e72e640 1 -- 172.21.15.87:0/1728426559 shutdown_connections 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.101+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/1728426559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d6810c510 0x7f2d6810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/1728426559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d68103af0 0x7f2d6810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/1728426559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d681031d0 0x7f2d681035b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- 172.21.15.87:0/1728426559 >> 172.21.15.87:0/1728426559 conn(0x7f2d68077f50 msgr2=0x7f2d68078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- 172.21.15.87:0/1728426559 shutdown_connections 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- 172.21.15.87:0/1728426559 wait complete. 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 Processor -- start 2024-04-22T13:33:35.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- start start 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d681031d0 0x7f2d68198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d68103af0 0x7f2d68198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681121a0 con 0x7f2d6810c510 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68112020 con 0x7f2d681031d0 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.102+0000 7f2d6e72e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68112320 con 0x7f2d68103af0 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44030/0 (socket says 172.21.15.87:44030) 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 -- 172.21.15.87:0/3060535926 learned_addr learned my addr 172.21.15.87:0/3060535926 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d677fe640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d68103af0 0x7f2d68198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d67fff640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d681031d0 0x7f2d68198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 -- 172.21.15.87:0/3060535926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d68103af0 msgr2=0x7f2d68198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d68103af0 0x7f2d68198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 -- 172.21.15.87:0/3060535926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d681031d0 msgr2=0x7f2d68198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d681031d0 0x7f2d68198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 -- 172.21.15.87:0/3060535926 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6819d730 con 0x7f2d6810c510 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6cccb640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000eaa0 tx=0x7f2d6000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d657fa640 1 -- 172.21.15.87:0/3060535926 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000cdb0 con 0x7f2d6810c510 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d657fa640 1 -- 172.21.15.87:0/3060535926 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d60005020 con 0x7f2d6810c510 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6819da20 con 0x7f2d6810c510 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d657fa640 1 -- 172.21.15.87:0/3060535926 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d600116d0 con 0x7f2d6810c510 2024-04-22T13:33:35.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.103+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681a52b0 con 0x7f2d6810c510 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.105+0000 7f2d657fa640 1 -- 172.21.15.87:0/3060535926 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2d600051d0 con 0x7f2d6810c510 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 Processor -- start 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 -- start start 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 0x56394d718710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d7222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56394d67d1f0 con 0x56394d718ce0 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56394d61a4b0 con 0x56394d718330 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d713f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56394d6791b0 con 0x56394d721e40 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d657fa640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 0x7f2d500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44040/0 (socket says 172.21.15.87:44040) 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 -- 172.21.15.87:0/2727292192 learned_addr learned my addr 172.21.15.87:0/2727292192 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 0x56394d718710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d6cccb640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 0x7f2d500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d7222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 -- 172.21.15.87:0/2727292192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 msgr2=0x56394d7222c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d7222c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 -- 172.21.15.87:0/2727292192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 msgr2=0x56394d718710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 0x56394d718710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d657fa640 1 -- 172.21.15.87:0/3060535926 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2d600d0900 con 0x7f2d6810c510 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d67fff640 1 -- 172.21.15.87:0/2727292192 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56394d60c1b0 con 0x56394d718ce0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.106+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 0x56394d718710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009900 tx=0x7f2d58005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec5b0ffaeb9a239 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d4e7fc640 1 -- 172.21.15.87:0/2727292192 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58004240 con 0x56394d718ce0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d6cccb640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 0x7f2d500a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d600ce170 tx=0x7f2d60087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d4e7fc640 1 -- 172.21.15.87:0/2727292192 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5802cdc0 con 0x56394d718ce0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d4e7fc640 1 -- 172.21.15.87:0/2727292192 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58004920 con 0x56394d718ce0 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 -- 172.21.15.87:0/2727292192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 msgr2=0x56394d721820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009900 tx=0x7f2d58005e70 comp rx=0 tx=0).stop 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 -- 172.21.15.87:0/2727292192 shutdown_connections 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d7222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718ce0 0x56394d721820 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2727292192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718330 0x56394d718710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 -- 172.21.15.87:0/2727292192 >> 172.21.15.87:0/2727292192 conn(0x56394d712ba0 msgr2=0x56394d712f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.107+0000 7f2d713f5740 1 -- 172.21.15.87:0/2727292192 shutdown_connections 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 -- 172.21.15.87:0/2727292192 wait complete. 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 -- start start 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 Processor -- start 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d725540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d725a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56394d728d60 con 0x56394d718330 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56394d636500 con 0x56394d718ce0 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56394d728ee0 con 0x56394d721e40 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d713f5740 10 client.0 _ll_register_callbacks cb 0x56394d604b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44044/0 (socket says 172.21.15.87:44044) 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.108+0000 7f2d677fe640 1 -- 172.21.15.87:0/1884811860 learned_addr learned my addr 172.21.15.87:0/1884811860 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d67fff640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d725540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 -- 172.21.15.87:0/1884811860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 msgr2=0x56394d725a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d725a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 -- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 msgr2=0x56394d725540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d725540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 -- 172.21.15.87:0/1884811860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56394d38f3f0 con 0x56394d718330 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d67fff640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d725540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d677fe640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c012480 tx=0x7f2d5c012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440fddac4b066421 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c004220 con 0x56394d718330 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2d5c0043c0 con 0x56394d718330 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d5802cdc0 con 0x56394d718330 2024-04-22T13:33:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2d5c0117f0 con 0x56394d718330 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 10 client.16269 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56394d4597d0 con 0x56394d718330 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5802cdc0 con 0x56394d718330 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2d5c023030 con 0x56394d718330 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 10 client.16269 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.109+0000 7f2d713f5740 20 client.16269 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d736000 0x56394d7384a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d713f5740 4 client.16269 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56394d61c930 con 0x56394d736000 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d67fff640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d736000 0x56394d7384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d477fe640 10 client.16269 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.110+0000 7f2d67fff640 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d736000 0x56394d7384a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=653c89503dc55136 server_cookie=964f9724ad5c056c in_seq=0 out_seq=0 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d477fe640 1 -- 172.21.15.87:0/1884811860 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56394d61c930 con 0x56394d736000 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d477fe640 10 client.16269 handle_command_reply: tid=0 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 client.16269 shutdown 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 msgr2=0x56394d7281b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c012480 tx=0x7f2d5c012940 comp rx=0 tx=0).stop 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 shutdown_connections 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d736000 0x56394d7384a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d721e40 0x56394d725a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d477fe640 0 client.16269 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d725540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 --2- 172.21.15.87:0/1884811860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d7281b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 >> 172.21.15.87:0/1884811860 conn(0x56394d712ba0 msgr2=0x56394d7148f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 shutdown_connections 2024-04-22T13:33:35.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 1 -- 172.21.15.87:0/1884811860 wait complete. 2024-04-22T13:33:35.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.111+0000 7f2d713f5740 20 client.16269 trim_cache size 0 max 16384 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.124+0000 7f2d713f5740 1 Processor -- start 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 -- start start 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 0x56394d711a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d721e40 0x56394d727c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681ae180 con 0x56394d718330 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681aba50 con 0x56394d721e40 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d713f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681a98e0 con 0x56394d718ce0 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44060/0 (socket says 172.21.15.87:44060) 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 -- 172.21.15.87:0/2520813701 learned_addr learned my addr 172.21.15.87:0/2520813701 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 0x56394d711a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 -- 172.21.15.87:0/2520813701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 msgr2=0x56394d711a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 0x56394d711a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 -- 172.21.15.87:0/2520813701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d721e40 msgr2=0x56394d727c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d721e40 0x56394d727c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 -- 172.21.15.87:0/2520813701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56394d728ca0 con 0x56394d718330 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 0x56394d711a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c012450 tx=0x7f2d5c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f826797a6b4c9d96 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d4dffb640 1 -- 172.21.15.87:0/2520813701 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c011b90 con 0x56394d718330 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d4dffb640 1 -- 172.21.15.87:0/2520813701 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c003aa0 con 0x56394d718330 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.125+0000 7f2d4dffb640 1 -- 172.21.15.87:0/2520813701 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c01b690 con 0x56394d718330 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- 172.21.15.87:0/2520813701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 msgr2=0x56394d731c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c012450 tx=0x7f2d5c00b8a0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- 172.21.15.87:0/2520813701 shutdown_connections 2024-04-22T13:33:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d721e40 0x56394d727c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718ce0 0x56394d711a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2520813701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d718330 0x56394d731c90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- 172.21.15.87:0/2520813701 >> 172.21.15.87:0/2520813701 conn(0x56394d712ba0 msgr2=0x56394d711370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- 172.21.15.87:0/2520813701 shutdown_connections 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- 172.21.15.87:0/2520813701 wait complete. 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 -- start start 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.126+0000 7f2d713f5740 1 Processor -- start 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718330 0x56394d7272b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d7277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681a9f30 con 0x56394d721e40 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681ab380 con 0x56394d718ce0 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681aa5e0 con 0x56394d718330 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 10 client.0 _ll_register_callbacks cb 0x7f2d5c00bb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44062/0 (socket says 172.21.15.87:44062) 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 -- 172.21.15.87:0/2537694174 learned_addr learned my addr 172.21.15.87:0/2537694174 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718330 0x56394d7272b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d7277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 -- 172.21.15.87:0/2537694174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718330 msgr2=0x56394d7272b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718330 0x56394d7272b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 -- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 msgr2=0x56394d7277f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d7277f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 -- 172.21.15.87:0/2537694174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56394d79b710 con 0x56394d721e40 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d7277f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d677fe640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005cd0 tx=0x7f2d5c012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff719ac6790e2909 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d477fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c040060 con 0x56394d721e40 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2d5c011b90 con 0x56394d721e40 2024-04-22T13:33:35.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.127+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d5c003aa0 con 0x56394d721e40 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2d5c011d90 con 0x56394d721e40 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 10 client.16275 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56394d79c960 con 0x56394d721e40 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c003aa0 con 0x56394d721e40 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2d5c043070 con 0x56394d721e40 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 10 client.16275 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 20 client.16275 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d79cec0 0x56394d79f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 4 client.16275 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56394d68ac10 con 0x56394d79cec0 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.128+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d79cec0 0x56394d79f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d477fe640 10 client.16275 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d67fff640 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d79cec0 0x56394d79f2b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=230c012d46130541 server_cookie=128d3977213ad614 in_seq=0 out_seq=0 2024-04-22T13:33:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d477fe640 1 -- 172.21.15.87:0/2537694174 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56394d68ac10 con 0x56394d79cec0 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d477fe640 10 client.16275 handle_command_reply: tid=0 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 client.16275 shutdown 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 msgr2=0x56394d727d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005cd0 tx=0x7f2d5c012940 comp rx=0 tx=0).stop 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 shutdown_connections 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56394d79cec0 0x56394d79f2b0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56394d721e40 0x56394d727d30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d477fe640 0 client.16275 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56394d718ce0 0x56394d7277f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 --2- 172.21.15.87:0/2537694174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56394d718330 0x56394d7272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 >> 172.21.15.87:0/2537694174 conn(0x56394d712ba0 msgr2=0x56394d711c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 shutdown_connections 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 1 -- 172.21.15.87:0/2537694174 wait complete. 2024-04-22T13:33:35.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.129+0000 7f2d713f5740 20 client.16275 trim_cache size 0 max 16384 2024-04-22T13:33:35.133 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:33:35.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 msgr2=0x7f2d500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 0x7f2d500a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d600ce170 tx=0x7f2d60087040 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 msgr2=0x7f2d6819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000eaa0 tx=0x7f2d6000ef60 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 shutdown_connections 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2d500a3d20 0x7f2d500a61e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d6810c510 0x7f2d6819d000 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d68103af0 0x7f2d68198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 --2- 172.21.15.87:0/3060535926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d681031d0 0x7f2d68198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 >> 172.21.15.87:0/3060535926 conn(0x7f2d68077f50 msgr2=0x7f2d680fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 shutdown_connections 2024-04-22T13:33:35.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:35.130+0000 7f2d6e72e640 1 -- 172.21.15.87:0/3060535926 wait complete. 2024-04-22T13:33:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:35 smithi120 ceph-mon[31101]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T13:33:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:35 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4030874095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:35.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:35 smithi132 ceph-mon[31531]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T13:33:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:35 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4030874095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:35.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:35 smithi087 ceph-mon[27999]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T13:33:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:35 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4030874095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:36.137 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:33:36.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:33:36.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:33:36.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- 172.21.15.87:0/574102674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec106160 msgr2=0x7f34ec10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 --2- 172.21.15.87:0/574102674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec106160 0x7f34ec10dd30 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009900 tx=0x7f34e402bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- 172.21.15.87:0/574102674 shutdown_connections 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 --2- 172.21.15.87:0/574102674 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec106160 0x7f34ec10dd30 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 --2- 172.21.15.87:0/574102674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec101a40 0x7f34ec105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 --2- 172.21.15.87:0/574102674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec101090 0x7f34ec101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- 172.21.15.87:0/574102674 >> 172.21.15.87:0/574102674 conn(0x7f34ec077f50 msgr2=0x7f34ec078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- 172.21.15.87:0/574102674 shutdown_connections 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- 172.21.15.87:0/574102674 wait complete. 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 Processor -- start 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.257+0000 7f34f19f4640 1 -- start start 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec106160 0x7f34ec19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec198660 0x7f34ec198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1110d0 con 0x7f34ec101090 2024-04-22T13:33:36.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec110f50 con 0x7f34ec106160 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec111250 con 0x7f34ec198660 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44082/0 (socket says 172.21.15.87:44082) 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 -- 172.21.15.87:0/2588136138 learned_addr learned my addr 172.21.15.87:0/2588136138 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec106160 0x7f34ec19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec198660 0x7f34ec198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 -- 172.21.15.87:0/2588136138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec198660 msgr2=0x7f34ec198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec198660 0x7f34ec198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 -- 172.21.15.87:0/2588136138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec106160 msgr2=0x7f34ec19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec106160 0x7f34ec19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 -- 172.21.15.87:0/2588136138 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ec199260 con 0x7f34ec101090 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00ebc0 tx=0x7f34dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f09f2640 1 -- 172.21.15.87:0/2588136138 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc010630 con 0x7f34ec101090 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f09f2640 1 -- 172.21.15.87:0/2588136138 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34dc010c80 con 0x7f34ec101090 2024-04-22T13:33:36.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.258+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ec1a5340 con 0x7f34ec101090 2024-04-22T13:33:36.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.259+0000 7f34f09f2640 1 -- 172.21.15.87:0/2588136138 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc018b60 con 0x7f34ec101090 2024-04-22T13:33:36.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.259+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ec1a5740 con 0x7f34ec101090 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f09f2640 1 -- 172.21.15.87:0/2588136138 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f34dc018d00 con 0x7f34ec101090 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 Processor -- start 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 -- start start 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 0x555f54d13cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d14290 0x555f54d1c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555f54c77f00 con 0x555f54d1ce10 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555f54c15590 con 0x555f54d138e0 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f45e9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555f54c73ec0 con 0x555f54d14290 2024-04-22T13:33:36.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44090/0 (socket says 172.21.15.87:44090) 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2077567125 learned_addr learned my addr 172.21.15.87:0/2077567125 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34f09f2640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 0x7f34d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 0x555f54d13cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d14290 0x555f54d1c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.261+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 0x7f34d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2077567125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d14290 msgr2=0x555f54d1c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d14290 0x555f54d1c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34f09f2640 1 -- 172.21.15.87:0/2588136138 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34dc0d0ed0 con 0x7f34ec101090 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2077567125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 msgr2=0x555f54d13cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 0x555f54d13cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2077567125 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f54a527e0 con 0x555f54d1ce10 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 0x555f54d13cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009b70 tx=0x7f34e00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3821e27590ae5357 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 0x7f34d40a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34e4006010 tx=0x7f34e4009260 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34d1ffb640 1 -- 172.21.15.87:0/2077567125 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0009e60 con 0x555f54d1ce10 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34d1ffb640 1 -- 172.21.15.87:0/2077567125 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e0012900 con 0x555f54d1ce10 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.262+0000 7f34d1ffb640 1 -- 172.21.15.87:0/2077567125 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e000fae0 con 0x555f54d1ce10 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- 172.21.15.87:0/2077567125 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 msgr2=0x555f54d1f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009b70 tx=0x7f34e00117b0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- 172.21.15.87:0/2077567125 shutdown_connections 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d1f260 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d14290 0x555f54d1c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2077567125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d138e0 0x555f54d13cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- 172.21.15.87:0/2077567125 >> 172.21.15.87:0/2077567125 conn(0x555f54d0dfb0 msgr2=0x555f54d0e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- 172.21.15.87:0/2077567125 shutdown_connections 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- 172.21.15.87:0/2077567125 wait complete. 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 -- start start 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.263+0000 7f34f45e9740 1 Processor -- start 2024-04-22T13:33:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d19de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 0x555f54d1a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555f54d1fd80 con 0x555f54d14290 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555f54c31580 con 0x555f54d1ce10 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555f54d1ff00 con 0x555f54d138e0 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 10 client.0 _ll_register_callbacks cb 0x555f54c37650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44102/0 (socket says 172.21.15.87:44102) 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2768019154 learned_addr learned my addr 172.21.15.87:0/2768019154 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d19de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 0x555f54d1a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2768019154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 msgr2=0x555f54d19de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d19de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 msgr2=0x555f54d1a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 0x555f54d1a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 -- 172.21.15.87:0/2768019154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f54ab2f70 con 0x555f54d14290 2024-04-22T13:33:36.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 0x555f54d1a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:36.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f34e0012680 tx=0x7f34e00126b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c663409fa2c5a897 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34caffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0028070 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34e000f7e0 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.264+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e0012900 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f34e0034050 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 10 client.16287 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555f5494a170 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e0012900 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f34e00046c0 con 0x555f54d14290 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 10 client.16287 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 20 client.16287 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d2d230 0x555f54d2f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 4 client.16287 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:36.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f54c85920 con 0x555f54d2d230 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.265+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d2d230 0x555f54d2f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.266+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d2d230 0x555f54d2f620 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acb99e20e99556e7 server_cookie=e78d5658420dbd16 in_seq=0 out_seq=0 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.266+0000 7f34caffd640 10 client.16287 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.266+0000 7f34caffd640 1 -- 172.21.15.87:0/2768019154 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f54c85920 con 0x555f54d2d230 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.266+0000 7f34caffd640 10 client.16287 handle_command_reply: tid=0 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.266+0000 7f34f45e9740 1 client.16287 shutdown 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 msgr2=0x555f54d1a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f34e0012680 tx=0x7f34e00126b0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 shutdown_connections 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d2d230 0x555f54d2f620 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d1ce10 0x555f54d1a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d14290 0x555f54d1a320 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34caffd640 0 client.16287 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2768019154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d19de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 >> 172.21.15.87:0/2768019154 conn(0x555f54d0dfb0 msgr2=0x555f54d0fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 shutdown_connections 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 1 -- 172.21.15.87:0/2768019154 wait complete. 2024-04-22T13:33:36.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.267+0000 7f34f45e9740 20 client.16287 trim_cache size 0 max 16384 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 Processor -- start 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 -- start start 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d1c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d28c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1b0390 con 0x555f54d1ce10 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1adc60 con 0x555f54d14290 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34f45e9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1abad0 con 0x555f54d138e0 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34eb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34eb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44104/0 (socket says 172.21.15.87:44104) 2024-04-22T13:33:36.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34eb7fe640 1 -- 172.21.15.87:0/357687810 learned_addr learned my addr 172.21.15.87:0/357687810 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d28c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.284+0000 7f34eaffd640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d1c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 -- 172.21.15.87:0/357687810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 msgr2=0x555f54d1c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d1c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 -- 172.21.15.87:0/357687810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 msgr2=0x555f54d28c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d28c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 -- 172.21.15.87:0/357687810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f54d099f0 con 0x555f54d1ce10 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d28c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f34e4005ab0 tx=0x7f34e4041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b4944e1a54ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34d17fa640 1 -- 172.21.15.87:0/357687810 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4050070 con 0x555f54d1ce10 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34d17fa640 1 -- 172.21.15.87:0/357687810 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e40046c0 con 0x555f54d1ce10 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34d17fa640 1 -- 172.21.15.87:0/357687810 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4045040 con 0x555f54d1ce10 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34f45e9740 1 -- 172.21.15.87:0/357687810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 msgr2=0x555f54d0faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34f45e9740 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f34e4005ab0 tx=0x7f34e4041000 comp rx=0 tx=0).stop 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34f45e9740 1 -- 172.21.15.87:0/357687810 shutdown_connections 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34f45e9740 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d0faf0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.285+0000 7f34f45e9740 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d28c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 --2- 172.21.15.87:0/357687810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d1c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- 172.21.15.87:0/357687810 >> 172.21.15.87:0/357687810 conn(0x555f54d0dfb0 msgr2=0x555f54d0f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- 172.21.15.87:0/357687810 shutdown_connections 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- 172.21.15.87:0/357687810 wait complete. 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- start start 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 Processor -- start 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d24db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d252f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1ac120 con 0x555f54d1ce10 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1ad590 con 0x555f54d14290 2024-04-22T13:33:36.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34ec1ac7f0 con 0x555f54d138e0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34f45e9740 10 client.0 _ll_register_callbacks cb 0x7f34e00099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34eb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.286+0000 7f34eb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44118/0 (socket says 172.21.15.87:44118) 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 -- 172.21.15.87:0/2977816567 learned_addr learned my addr 172.21.15.87:0/2977816567 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d24db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d252f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 -- 172.21.15.87:0/2977816567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 msgr2=0x555f54d24db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d24db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 -- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 msgr2=0x555f54d252f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d252f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 -- 172.21.15.87:0/2977816567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f54d219c0 con 0x555f54d1ce10 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34ea7fc640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d252f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34eb7fe640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009a20 tx=0x7f34e40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c04022f867775 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34caffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4059030 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34e4004870 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e40046c0 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f34e4066050 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34f45e9740 10 client.16293 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555f54d93f70 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.287+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e40046c0 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f34e4040040 con 0x555f54d1ce10 2024-04-22T13:33:36.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 10 client.16293 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 20 client.16293 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d94660 0x555f54d96a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 4 client.16293 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555f54c7f790 con 0x555f54d94660 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d94660 0x555f54d96a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34caffd640 10 client.16293 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.288+0000 7f34eaffd640 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d94660 0x555f54d96a80 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9110d1958a1df2d server_cookie=f8a1ccc12a90059b in_seq=0 out_seq=0 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34caffd640 1 -- 172.21.15.87:0/2977816567 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555f54c7f790 con 0x555f54d94660 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34caffd640 10 client.16293 handle_command_reply: tid=0 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 client.16293 shutdown 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 msgr2=0x555f54d21280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009a20 tx=0x7f34e40095c0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 shutdown_connections 2024-04-22T13:33:36.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555f54d94660 0x555f54d96a80 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555f54d1ce10 0x555f54d21280 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f54d14290 0x555f54d252f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 --2- 172.21.15.87:0/2977816567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f54d138e0 0x555f54d24db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 >> 172.21.15.87:0/2977816567 conn(0x555f54d0dfb0 msgr2=0x555f54d0fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34caffd640 0 client.16293 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 shutdown_connections 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 1 -- 172.21.15.87:0/2977816567 wait complete. 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.290+0000 7f34f45e9740 20 client.16293 trim_cache size 0 max 16384 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:36.292 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:33:36.293 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "e53819e7-d773-4e26-9757-3e39a7e5b506", 2024-04-22T13:33:36.293 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:33:36.293 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:36.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 msgr2=0x7f34d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 0x7f34d40a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34e4006010 tx=0x7f34e4009260 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 msgr2=0x7f34ec19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00ebc0 tx=0x7f34dc00c6a0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 shutdown_connections 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34d40a3d20 0x7f34d40a61e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34ec198660 0x7f34ec198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34ec106160 0x7f34ec19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 --2- 172.21.15.87:0/2588136138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34ec101090 0x7f34ec19e5a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 >> 172.21.15.87:0/2588136138 conn(0x7f34ec077f50 msgr2=0x7f34ec0695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 shutdown_connections 2024-04-22T13:33:36.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.291+0000 7f34f19f4640 1 -- 172.21.15.87:0/2588136138 wait complete. 2024-04-22T13:33:36.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:33:36.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:33:36.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 -- 172.21.15.87:0/3127738980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0102b50 msgr2=0x7f45a0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 --2- 172.21.15.87:0/3127738980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0102b50 0x7f45a0106b90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4588009900 tx=0x7f458802bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:36.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 -- 172.21.15.87:0/3127738980 shutdown_connections 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 --2- 172.21.15.87:0/3127738980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a0107270 0x7f45a010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 --2- 172.21.15.87:0/3127738980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0102b50 0x7f45a0106b90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 --2- 172.21.15.87:0/3127738980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 0x7f45a0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.404+0000 7f45a5a43640 1 -- 172.21.15.87:0/3127738980 >> 172.21.15.87:0/3127738980 conn(0x7f45a00fda40 msgr2=0x7f45a00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- 172.21.15.87:0/3127738980 shutdown_connections 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- 172.21.15.87:0/3127738980 wait complete. 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 Processor -- start 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- start start 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 0x7f45a019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a01985e0 0x7f45a0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112190 con 0x7f45a0107270 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112010 con 0x7f45a01985e0 2024-04-22T13:33:36.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f45a5a43640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112310 con 0x7f45a01021a0 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44132/0 (socket says 172.21.15.87:44132) 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.405+0000 7f4597fff640 1 -- 172.21.15.87:0/2129413308 learned_addr learned my addr 172.21.15.87:0/2129413308 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f459effd640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 0x7f45a019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f459f7fe640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a01985e0 0x7f45a0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 -- 172.21.15.87:0/2129413308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 msgr2=0x7f45a019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 0x7f45a019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 -- 172.21.15.87:0/2129413308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a01985e0 msgr2=0x7f45a0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a01985e0 0x7f45a0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 -- 172.21.15.87:0/2129413308 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a0199100 con 0x7f45a0107270 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f4597fff640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4588036040 tx=0x7f458802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f45a4a41640 1 -- 172.21.15.87:0/2129413308 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588043020 con 0x7f45a0107270 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a0199390 con 0x7f45a0107270 2024-04-22T13:33:36.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f45a4a41640 1 -- 172.21.15.87:0/2129413308 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458802cbe0 con 0x7f45a0107270 2024-04-22T13:33:36.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f45a4a41640 1 -- 172.21.15.87:0/2129413308 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588003bf0 con 0x7f45a0107270 2024-04-22T13:33:36.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.406+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a55e0 con 0x7f45a0107270 2024-04-22T13:33:36.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.408+0000 7f45a863b740 1 Processor -- start 2024-04-22T13:33:36.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.408+0000 7f45a4a41640 1 -- 172.21.15.87:0/2129413308 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4588046020 con 0x7f45a0107270 2024-04-22T13:33:36.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 -- start start 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55978492c1e0 con 0x5597849d0a80 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5597848c9390 con 0x5597849c7ea0 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a863b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5597849281a0 con 0x5597849c7580 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44136/0 (socket says 172.21.15.87:44136) 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 -- 172.21.15.87:0/858326457 learned_addr learned my addr 172.21.15.87:0/858326457 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f459effd640 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f459f7fe640 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 -- 172.21.15.87:0/858326457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 msgr2=0x5597849c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 -- 172.21.15.87:0/858326457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 msgr2=0x5597849d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 -- 172.21.15.87:0/858326457 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597847087d0 con 0x5597849d0a80 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f45a4a41640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 0x7f458c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f459f7fe640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 0x7f458c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.409+0000 7f4597fff640 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f45880f0050 tx=0x7f45880f8b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838da64283086935 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a4a41640 1 -- 172.21.15.87:0/2129413308 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45880f2810 con 0x7f45a0107270 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f4595ffb640 1 -- 172.21.15.87:0/858326457 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880be700 con 0x5597849d0a80 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f4595ffb640 1 -- 172.21.15.87:0/858326457 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45880bf030 con 0x5597849d0a80 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f4595ffb640 1 -- 172.21.15.87:0/858326457 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880be290 con 0x5597849d0a80 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 -- 172.21.15.87:0/858326457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 msgr2=0x5597849d2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f45880f0050 tx=0x7f45880f8b40 comp rx=0 tx=0).stop 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f459f7fe640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 0x7f458c0a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4598005fd0 tx=0x7f4598009600 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 -- 172.21.15.87:0/858326457 shutdown_connections 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849d2e70 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 --2- 172.21.15.87:0/858326457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 -- 172.21.15.87:0/858326457 >> 172.21.15.87:0/858326457 conn(0x5597849c1cb0 msgr2=0x5597849c2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 -- 172.21.15.87:0/858326457 shutdown_connections 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.410+0000 7f45a863b740 1 -- 172.21.15.87:0/858326457 wait complete. 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 -- start start 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 Processor -- start 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5597849d3630 con 0x5597849c7ea0 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5597848e52f0 con 0x5597849d0a80 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5597849d37b0 con 0x5597849c7580 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f45a863b740 10 client.0 _ll_register_callbacks cb 0x5597848f2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:36.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f4597fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44142/0 (socket says 172.21.15.87:44142) 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.411+0000 7f4597fff640 1 -- 172.21.15.87:0/3915271500 learned_addr learned my addr 172.21.15.87:0/3915271500 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f459f7fe640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 -- 172.21.15.87:0/3915271500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 msgr2=0x5597849cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 -- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 msgr2=0x5597849ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 -- 172.21.15.87:0/3915271500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55978463e3f0 con 0x5597849c7ea0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f459f7fe640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f4597fff640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45880f0190 tx=0x7f4588000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b38a8db3ccc82f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880be470 con 0x5597849c7ea0 2024-04-22T13:33:36.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45880bafa0 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45880bf030 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f45880be290 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 10 client.16305 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5597845fe0f0 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45880bf030 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f45880f8c70 con 0x5597849c7ea0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 10 client.16305 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 20 client.16305 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5597849e0690 0x5597849e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 4 client.16305 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.412+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559784939c00 con 0x5597849e0690 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.413+0000 7f459effd640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5597849e0690 0x5597849e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.413+0000 7f457effd640 10 client.16305 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.413+0000 7f459effd640 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5597849e0690 0x5597849e2a80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=427f7dc0f6356799 server_cookie=fe19c535236138bf in_seq=0 out_seq=0 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f457effd640 1 -- 172.21.15.87:0/3915271500 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559784939c00 con 0x5597849e0690 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f457effd640 10 client.16305 handle_command_reply: tid=0 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 client.16305 shutdown 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 msgr2=0x5597849ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45880f0190 tx=0x7f4588000bc0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 shutdown_connections 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5597849e0690 0x5597849e2a80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7ea0 0x5597849ce050 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f457effd640 0 client.16305 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 --2- 172.21.15.87:0/3915271500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 >> 172.21.15.87:0/3915271500 conn(0x5597849c1cb0 msgr2=0x5597849c3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 shutdown_connections 2024-04-22T13:33:36.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 1 -- 172.21.15.87:0/3915271500 wait complete. 2024-04-22T13:33:36.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.414+0000 7f45a863b740 20 client.16305 trim_cache size 0 max 16384 2024-04-22T13:33:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 Processor -- start 2024-04-22T13:33:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 -- start start 2024-04-22T13:33:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7ea0 0x5597849dc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b03e0 con 0x5597849c7580 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01adcb0 con 0x5597849d0a80 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f45a863b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01abba0 con 0x5597849c7ea0 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.431+0000 7f459effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44156/0 (socket says 172.21.15.87:44156) 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 -- 172.21.15.87:0/4234225880 learned_addr learned my addr 172.21.15.87:0/4234225880 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 -- 172.21.15.87:0/4234225880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7ea0 msgr2=0x5597849dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7ea0 0x5597849dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 -- 172.21.15.87:0/4234225880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 msgr2=0x5597849d8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459f7fe640 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849d8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 -- 172.21.15.87:0/4234225880 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597849c7cd0 con 0x5597849c7580 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459f7fe640 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f459effd640 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f45900044d0 tx=0x7f459000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d3c176d3394153 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f45957fa640 1 -- 172.21.15.87:0/4234225880 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590003cc0 con 0x5597849c7580 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f45957fa640 1 -- 172.21.15.87:0/4234225880 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4590003e60 con 0x5597849c7580 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.432+0000 7f45957fa640 1 -- 172.21.15.87:0/4234225880 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590011940 con 0x5597849c7580 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- 172.21.15.87:0/4234225880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 msgr2=0x5597849d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f45900044d0 tx=0x7f459000bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- 172.21.15.87:0/4234225880 shutdown_connections 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849d0a80 0x5597849d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7ea0 0x5597849dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- 172.21.15.87:0/4234225880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849c7580 0x5597849d0240 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- 172.21.15.87:0/4234225880 >> 172.21.15.87:0/4234225880 conn(0x5597849c1cb0 msgr2=0x5597849c04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- 172.21.15.87:0/4234225880 shutdown_connections 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- 172.21.15.87:0/4234225880 wait complete. 2024-04-22T13:33:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- start start 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 Processor -- start 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01ac1f0 con 0x5597849d0a80 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01ad5e0 con 0x5597849c7ea0 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.433+0000 7f45a863b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01ac840 con 0x5597849c7580 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f45a863b740 10 client.0 _ll_register_callbacks cb 0x7f459000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44162/0 (socket says 172.21.15.87:44162) 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f4597fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 -- 172.21.15.87:0/508275531 learned_addr learned my addr 172.21.15.87:0/508275531 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459effd640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 -- 172.21.15.87:0/508275531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 msgr2=0x5597849cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 -- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 msgr2=0x5597849cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 -- 172.21.15.87:0/508275531 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597849ca440 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f459f7fe640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4598011e40 tx=0x7f4598004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df878b67f987f9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f457effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598004b20 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4598004cc0 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4590003e60 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4598029470 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f45a863b740 10 client.16311 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559784a4b5a0 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.434+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4590003e60 con 0x5597849d0a80 2024-04-22T13:33:36.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4598031020 con 0x5597849d0a80 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 10 client.16311 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 20 client.16311 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559784a4bcf0 0x559784a4e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 4 client.16311 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559784933a70 con 0x559784a4bcf0 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f459effd640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559784a4bcf0 0x559784a4e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f459effd640 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559784a4bcf0 0x559784a4e110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8ccbd6b501fe063 server_cookie=e8beb7df5f3c45fb in_seq=0 out_seq=0 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f457effd640 10 client.16311 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f457effd640 1 -- 172.21.15.87:0/508275531 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559784933a70 con 0x559784a4bcf0 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f457effd640 10 client.16311 handle_command_reply: tid=0 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.435+0000 7f45a863b740 1 client.16311 shutdown 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 msgr2=0x5597849c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4598011e40 tx=0x7f4598004a40 comp rx=0 tx=0).stop 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 shutdown_connections 2024-04-22T13:33:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559784a4bcf0 0x559784a4e110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5597849d0a80 0x5597849c9d80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f457effd640 0 client.16311 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597849c7ea0 0x5597849cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 --2- 172.21.15.87:0/508275531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597849c7580 0x5597849cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 >> 172.21.15.87:0/508275531 conn(0x5597849c1cb0 msgr2=0x5597849ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 shutdown_connections 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 1 -- 172.21.15.87:0/508275531 wait complete. 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a863b740 20 client.16311 trim_cache size 0 max 16384 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:36.438 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: "e53819e7-d773-4e26-9757-3e39a7e5b506": { 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "e53819e7-d773-4e26-9757-3e39a7e5b506", 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:36.439 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 msgr2=0x7f458c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 0x7f458c0a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4598005fd0 tx=0x7f4598009600 comp rx=0 tx=0).stop 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 msgr2=0x7f45a019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4588036040 tx=0x7f458802c7e0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 shutdown_connections 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f458c0a3d20 0x7f458c0a61e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a01985e0 0x7f45a0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45a0107270 0x7f45a019e920 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 --2- 172.21.15.87:0/2129413308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a01021a0 0x7f45a019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:36.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 >> 172.21.15.87:0/2129413308 conn(0x7f45a00fda40 msgr2=0x7f45a00fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:36.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 shutdown_connections 2024-04-22T13:33:36.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:36.436+0000 7f45a5a43640 1 -- 172.21.15.87:0/2129413308 wait complete. 2024-04-22T13:33:36.443 INFO:tasks.cephfs.filesystem:scrub status for tag:e53819e7-d773-4e26-9757-3e39a7e5b506 - {'path': '/', 'tag': 'e53819e7-d773-4e26-9757-3e39a7e5b506', 'options': 'recursive,force'} 2024-04-22T13:33:37.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:37 smithi120 ceph-mon[31101]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:33:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:37 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:33:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:37 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:37 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:33:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:37 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:37.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:37 smithi132 ceph-mon[31531]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:33:37.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:37 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:33:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:37 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:37 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:33:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:37 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:37.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:37 smithi087 ceph-mon[27999]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:33:37.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:37 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:33:37.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:37 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:37.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:37 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:33:37.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:37 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:33:39.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:39 smithi120 ceph-mon[31101]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:39.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:39 smithi132 ceph-mon[31531]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:39.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:39 smithi087 ceph-mon[27999]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:41.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:41 smithi120 ceph-mon[31101]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:41.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:41 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:33:41.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:41 smithi132 ceph-mon[31531]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:41.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:41 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:33:41.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:41 smithi087 ceph-mon[27999]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:41.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:41 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:33:43.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:43 smithi120 ceph-mon[31101]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:43.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:43 smithi132 ceph-mon[31531]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:43.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:43 smithi087 ceph-mon[27999]: pgmap v58: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T13:33:45.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:45 smithi120 ceph-mon[31101]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:45.708 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 -- 172.21.15.87:0/216604845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c810c340 msgr2=0x7ff8c810e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 --2- 172.21.15.87:0/216604845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c810c340 0x7ff8c810e7c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009ca0 tx=0x7ff8c002c180 comp rx=0 tx=0).stop 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 -- 172.21.15.87:0/216604845 shutdown_connections 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 --2- 172.21.15.87:0/216604845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c810c340 0x7ff8c810e7c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 --2- 172.21.15.87:0/216604845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8103890 0x7ff8c810bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 --2- 172.21.15.87:0/216604845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c8102f70 0x7ff8c8103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 -- 172.21.15.87:0/216604845 >> 172.21.15.87:0/216604845 conn(0x7ff8c80fc770 msgr2=0x7ff8c80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 -- 172.21.15.87:0/216604845 shutdown_connections 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.710+0000 7ff8cc800640 1 -- 172.21.15.87:0/216604845 wait complete. 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 Processor -- start 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 -- start start 2024-04-22T13:33:45.709 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 0x7ff8c8100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c810c340 0x7ff8c8101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8111fb0 con 0x7ff8c8103890 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8111e30 con 0x7ff8c8102f70 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8cc800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8112130 con 0x7ff8c810c340 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42046/0 (socket says 172.21.15.87:42046) 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 -- 172.21.15.87:0/1750642932 learned_addr learned my addr 172.21.15.87:0/1750642932 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 0x7ff8c8100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8c759e640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c810c340 0x7ff8c8101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 -- 172.21.15.87:0/1750642932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c810c340 msgr2=0x7ff8c8101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c810c340 0x7ff8c8101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 -- 172.21.15.87:0/1750642932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 msgr2=0x7ff8c8100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 0x7ff8c8100bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8bffff640 1 -- 172.21.15.87:0/1750642932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c81a6a50 con 0x7ff8c8103890 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.711+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 0x7ff8c8100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8bffff640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000ea60 tx=0x7ff8b000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8c4d99640 1 -- 172.21.15.87:0/1750642932 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b000cde0 con 0x7ff8c8103890 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8c4d99640 1 -- 172.21.15.87:0/1750642932 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b00035f0 con 0x7ff8c8103890 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c81a6d40 con 0x7ff8c8103890 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8c4d99640 1 -- 172.21.15.87:0/1750642932 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0010690 con 0x7ff8c8103890 2024-04-22T13:33:45.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.712+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c81a7230 con 0x7ff8c8103890 2024-04-22T13:33:45.717 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:45 smithi132 ceph-mon[31531]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:45.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.714+0000 7ff8c4d99640 1 -- 172.21.15.87:0/1750642932 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff8b00037a0 con 0x7ff8c8103890 2024-04-22T13:33:45.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.714+0000 7ff8cfbea740 1 Processor -- start 2024-04-22T13:33:45.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 -- start start 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5bb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56040d5148d0 con 0x56040d5afd50 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56040d4b1e80 con 0x56040d5b0ac0 2024-04-22T13:33:45.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56040d510890 con 0x56040d5b92d0 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c4d99640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 0x7ff8b40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42056/0 (socket says 172.21.15.87:42056) 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3998977902 learned_addr learned my addr 172.21.15.87:0/3998977902 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c759e640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c759e640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 0x7ff8b40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8bffff640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5bb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3998977902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 msgr2=0x56040d5bb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5bb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3998977902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 msgr2=0x56040d5af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c4d99640 1 -- 172.21.15.87:0/1750642932 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff8b00d0a40 con 0x7ff8c8103890 2024-04-22T13:33:45.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3998977902 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040d4d3320 con 0x56040d5afd50 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.715+0000 7ff8bffff640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5bb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.716+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80115c0 tx=0x7ff8b8011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2a9d2b5cc16fe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.716+0000 7ff8c759e640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 0x7ff8b40a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009430 tx=0x7ff8c00095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.716+0000 7ff8bd7fa640 1 -- 172.21.15.87:0/3998977902 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800f640 con 0x56040d5afd50 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.716+0000 7ff8bd7fa640 1 -- 172.21.15.87:0/3998977902 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b800fc90 con 0x56040d5afd50 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.716+0000 7ff8bd7fa640 1 -- 172.21.15.87:0/3998977902 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8019c80 con 0x56040d5afd50 2024-04-22T13:33:45.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3998977902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 msgr2=0x56040d5b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff8b80115c0 tx=0x7ff8b8011a80 comp rx=0 tx=0).stop 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3998977902 shutdown_connections 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5bb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b01d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3998977902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3998977902 >> 172.21.15.87:0/3998977902 conn(0x56040d576c90 msgr2=0x56040d5aaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3998977902 shutdown_connections 2024-04-22T13:33:45.722 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3998977902 wait complete. 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- start start 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 Processor -- start 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b0ac0 0x56040d5b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b92d0 0x56040d5b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56040d5bc000 con 0x56040d5afd50 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56040d49ceb0 con 0x56040d5b92d0 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56040d5bc180 con 0x56040d5b0ac0 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.717+0000 7ff8cfbea740 10 client.0 _ll_register_callbacks cb 0x56040d4c9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42068/0 (socket says 172.21.15.87:42068) 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3918539287 learned_addr learned my addr 172.21.15.87:0/3918539287 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8bffff640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b0ac0 0x56040d5b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.723 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c759e640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b92d0 0x56040d5b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3918539287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b0ac0 msgr2=0x56040d5b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b0ac0 0x56040d5b6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b92d0 msgr2=0x56040d5b65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b92d0 0x56040d5b65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/3918539287 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56040d227210 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009350 tx=0x7ff8b800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290f87d6d100b4eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8026020 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff8b8019db0 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b800fc90 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff8b8031050 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8cfbea740 10 client.16323 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.718+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56040d2f3620 con 0x56040d5afd50 2024-04-22T13:33:45.724 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b800fc90 con 0x56040d5afd50 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff8b8003710 con 0x56040d5afd50 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 10 client.16323 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 10 client.16323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 20 client.16323 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d5c91f0 0x56040d5cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 4 client.16323 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56040d528880 con 0x56040d5c91f0 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8bffff640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d5c91f0 0x56040d5cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8bffff640 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d5c91f0 0x56040d5cb5e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29972fa163141725 server_cookie=c922cda5734e8125 in_seq=0 out_seq=0 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.719+0000 7ff8ae7fc640 10 client.16323 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/3918539287 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56040d528880 con 0x56040d5c91f0 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8ae7fc640 10 client.16323 handle_command_reply: tid=0 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 client.16323 shutdown 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 msgr2=0x56040d5b8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009350 tx=0x7ff8b800ff30 comp rx=0 tx=0).stop 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 shutdown_connections 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d5c91f0 0x56040d5cb5e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b92d0 0x56040d5b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b0ac0 0x56040d5b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8ae7fc640 0 client.16323 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/3918539287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b8c70 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.720+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 >> 172.21.15.87:0/3918539287 conn(0x56040d576c90 msgr2=0x56040d5ac790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.721+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 shutdown_connections 2024-04-22T13:33:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.721+0000 7ff8cfbea740 1 -- 172.21.15.87:0/3918539287 wait complete. 2024-04-22T13:33:45.727 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.721+0000 7ff8cfbea740 20 client.16323 trim_cache size 0 max 16384 2024-04-22T13:33:45.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 Processor -- start 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 -- start start 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 0x56040d5c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81b2350 con 0x56040d5b92d0 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81afc20 con 0x56040d5b0ac0 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81ada90 con 0x56040d5afd50 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42080/0 (socket says 172.21.15.87:42080) 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 -- 172.21.15.87:0/2745581960 learned_addr learned my addr 172.21.15.87:0/2745581960 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8bffff640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 0x56040d5c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 -- 172.21.15.87:0/2745581960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 msgr2=0x56040d5c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 0x56040d5c4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 -- 172.21.15.87:0/2745581960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 msgr2=0x56040d5aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c759e640 1 -- 172.21.15.87:0/2745581960 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040d5af020 con 0x56040d5b92d0 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.736+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 0x56040d5c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8c759e640 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009ca0 tx=0x7ff8c003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961001ca918d8981 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8bcff9640 1 -- 172.21.15.87:0/2745581960 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0039070 con 0x56040d5b92d0 2024-04-22T13:33:45.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8bcff9640 1 -- 172.21.15.87:0/2745581960 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8c0055070 con 0x56040d5b92d0 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8bcff9640 1 -- 172.21.15.87:0/2745581960 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0032a30 con 0x56040d5b92d0 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 -- 172.21.15.87:0/2745581960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 msgr2=0x56040d5b8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009ca0 tx=0x7ff8c003f000 comp rx=0 tx=0).stop 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 -- 172.21.15.87:0/2745581960 shutdown_connections 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5b92d0 0x56040d5b8730 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/2745581960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5afd50 0x56040d5c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.737+0000 7ff8cfbea740 1 -- 172.21.15.87:0/2745581960 >> 172.21.15.87:0/2745581960 conn(0x56040d576c90 msgr2=0x56040d5b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- 172.21.15.87:0/2745581960 shutdown_connections 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- 172.21.15.87:0/2745581960 wait complete. 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- start start 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 Processor -- start 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5ba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81ae0e0 con 0x56040d5afd50 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81af550 con 0x56040d5b0ac0 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c81ae7b0 con 0x56040d5b92d0 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8cfbea740 10 client.0 _ll_register_callbacks cb 0x7ff8c0037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42092/0 (socket says 172.21.15.87:42092) 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/4268038256 learned_addr learned my addr 172.21.15.87:0/4268038256 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:45.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8bffff640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5ba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c759e640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/4268038256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 msgr2=0x56040d5c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 msgr2=0x56040d5ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5ba1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.738+0000 7ff8c6d9d640 1 -- 172.21.15.87:0/4268038256 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56040d5b5dd0 con 0x56040d5afd50 2024-04-22T13:33:45.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8c6d9d640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8011ac0 tx=0x7ff8b800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ba4761e8607d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b803a060 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff8b80035f0 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8c0055070 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff8b8026050 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 10 client.16329 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56040d5bbb30 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8c0055070 con 0x56040d5afd50 2024-04-22T13:33:45.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff8b803d020 con 0x56040d5afd50 2024-04-22T13:33:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 10 client.16329 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:45.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 20 client.16329 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d62f630 0x56040d631a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 4 client.16329 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.739+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56040d4b43d0 con 0x56040d62f630 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.740+0000 7ff8bffff640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d62f630 0x56040d631a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.740+0000 7ff8ae7fc640 10 client.16329 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:45.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.740+0000 7ff8bffff640 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d62f630 0x56040d631a50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed06563ec21d8f1 server_cookie=ed8b1d1dda208a78 in_seq=0 out_seq=0 2024-04-22T13:33:45.747 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:45 smithi087 ceph-mon[27999]: pgmap v59: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.743+0000 7ff8ae7fc640 1 -- 172.21.15.87:0/4268038256 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56040d4b43d0 con 0x56040d62f630 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.743+0000 7ff8ae7fc640 10 client.16329 handle_command_reply: tid=0 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.743+0000 7ff8cfbea740 1 client.16329 shutdown 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.743+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 msgr2=0x56040d5b9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.743+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8011ac0 tx=0x7ff8b800f3c0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 shutdown_connections 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56040d62f630 0x56040d631a50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040d5b92d0 0x56040d5c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56040d5b0ac0 0x56040d5ba1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 --2- 172.21.15.87:0/4268038256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56040d5afd50 0x56040d5b9cb0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8ae7fc640 0 client.16329 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 >> 172.21.15.87:0/4268038256 conn(0x56040d576c90 msgr2=0x56040d5b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 shutdown_connections 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 1 -- 172.21.15.87:0/4268038256 wait complete. 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cfbea740 20 client.16329 trim_cache size 0 max 16384 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:33:45.749 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 msgr2=0x7ff8b40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 0x7ff8b40a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009430 tx=0x7ff8c00095c0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 msgr2=0x7ff8c8101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000ea60 tx=0x7ff8b000ef20 comp rx=0 tx=0).stop 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 shutdown_connections 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8b40a3d10 0x7ff8b40a61d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8c810c340 0x7ff8c8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c8103890 0x7ff8c8101100 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 --2- 172.21.15.87:0/1750642932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8c8102f70 0x7ff8c8100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:45.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 >> 172.21.15.87:0/1750642932 conn(0x7ff8c80fc770 msgr2=0x7ff8c80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:45.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 shutdown_connections 2024-04-22T13:33:45.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:45.744+0000 7ff8cc800640 1 -- 172.21.15.87:0/1750642932 wait complete. 2024-04-22T13:33:47.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:47 smithi120 ceph-mon[31101]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T13:33:47.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:47 smithi132 ceph-mon[31531]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T13:33:47.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:47 smithi087 ceph-mon[27999]: pgmap v60: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T13:33:49.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:49 smithi120 ceph-mon[31101]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:33:49.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:49 smithi132 ceph-mon[31531]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:33:49.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:49 smithi087 ceph-mon[27999]: pgmap v61: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:33:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:51 smithi120 ceph-mon[31101]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:51.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:51 smithi132 ceph-mon[31531]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:51.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:51 smithi087 ceph-mon[27999]: pgmap v62: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:52.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 -- 172.21.15.87:0/1834714569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2208105e30 msgr2=0x7f2208106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1834714569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2208105e30 0x7f2208106210 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009930 tx=0x7f21f802bc30 comp rx=0 tx=0).stop 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 -- 172.21.15.87:0/1834714569 shutdown_connections 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1834714569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208101b60 0x7f220810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1834714569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22081011a0 0x7f2208101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1834714569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2208105e30 0x7f2208106210 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 -- 172.21.15.87:0/1834714569 >> 172.21.15.87:0/1834714569 conn(0x7f2208077e30 msgr2=0x7f2208078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 -- 172.21.15.87:0/1834714569 shutdown_connections 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.716+0000 7f220ccd5640 1 -- 172.21.15.87:0/1834714569 wait complete. 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 Processor -- start 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 -- start start 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 0x7f2208197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208105e30 0x7f220819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f22081110c0 con 0x7f22081011a0 2024-04-22T13:33:52.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2208110f40 con 0x7f2208105e30 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220ccd5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111240 con 0x7f2208101b60 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57548/0 (socket says 172.21.15.87:57548) 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 -- 172.21.15.87:0/1671495094 learned_addr learned my addr 172.21.15.87:0/1671495094 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f2206d9d640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208105e30 0x7f220819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f2205d9b640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 0x7f2208197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 -- 172.21.15.87:0/1671495094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 msgr2=0x7f2208197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 0x7f2208197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 -- 172.21.15.87:0/1671495094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208105e30 msgr2=0x7f220819be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208105e30 0x7f220819be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 -- 172.21.15.87:0/1671495094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220819c5b0 con 0x7f22081011a0 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f2205d9b640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 0x7f2208197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:52.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.717+0000 7f220659c640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009230 tx=0x7f21f8004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.718+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8043020 con 0x7f22081011a0 2024-04-22T13:33:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.718+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f8004350 con 0x7f22081011a0 2024-04-22T13:33:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.718+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220819c840 con 0x7f22081011a0 2024-04-22T13:33:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.718+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8003b50 con 0x7f22081011a0 2024-04-22T13:33:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.718+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22081a4130 con 0x7f22081011a0 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.719+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2208198910 con 0x7f22081011a0 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.720+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f21f8046020 con 0x7f22081011a0 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.721+0000 7f21f77fe640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 0x7f21f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.721+0000 7f2205d9b640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 0x7f21f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.721+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f21f80f2840 con 0x7f22081011a0 2024-04-22T13:33:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.721+0000 7f2205d9b640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 0x7f21f00a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f22080fe5c0 tx=0x7f21fc00a5c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:52.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.726+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21f804d020 con 0x7f22081011a0 2024-04-22T13:33:52.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.939+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2208109330 con 0x7f22081011a0 2024-04-22T13:33:52.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.940+0000 7f21f77fe640 1 -- 172.21.15.87:0/1671495094 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f21f80bf060 con 0x7f22081011a0 2024-04-22T13:33:52.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:33:52.937 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:33:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 msgr2=0x7f21f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 0x7f21f00a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f22080fe5c0 tx=0x7f21fc00a5c0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 msgr2=0x7f22081975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009230 tx=0x7f21f8004bb0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 shutdown_connections 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f21f00a3d20 0x7f21f00a61e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2208105e30 0x7f220819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2208101b60 0x7f2208197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 --2- 172.21.15.87:0/1671495094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22081011a0 0x7f22081975c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 >> 172.21.15.87:0/1671495094 conn(0x7f2208077e30 msgr2=0x7f22080feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 shutdown_connections 2024-04-22T13:33:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:52.942+0000 7f220ccd5640 1 -- 172.21.15.87:0/1671495094 wait complete. 2024-04-22T13:33:52.950 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 1e2e33c6 --release --await' 2024-04-22T13:33:52.951 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 1e2e33c6 --release --await 2024-04-22T13:33:53.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- 172.21.15.87:0/281223823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 msgr2=0x7fd82810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- 172.21.15.87:0/281223823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd82810bfd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009900 tx=0x7fd81c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- 172.21.15.87:0/281223823 shutdown_connections 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- 172.21.15.87:0/281223823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 0x7fd82810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- 172.21.15.87:0/281223823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd82810bfd0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- 172.21.15.87:0/281223823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 0x7fd8281035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- 172.21.15.87:0/281223823 >> 172.21.15.87:0/281223823 conn(0x7fd828077f50 msgr2=0x7fd828078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- 172.21.15.87:0/281223823 shutdown_connections 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- 172.21.15.87:0/281223823 wait complete. 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 Processor -- start 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- start start 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 0x7fd828100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 0x7fd8281018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281121a0 con 0x7fd828103af0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112020 con 0x7fd8281031d0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82f50d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112320 con 0x7fd82810c510 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.061+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57556/0 (socket says 172.21.15.87:57556) 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 -- 172.21.15.87:0/3931571776 learned_addr learned my addr 172.21.15.87:0/3931571776 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 0x7fd828100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 0x7fd8281018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 -- 172.21.15.87:0/3931571776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 msgr2=0x7fd8281018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 0x7fd8281018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 -- 172.21.15.87:0/3931571776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 msgr2=0x7fd828100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 0x7fd828100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 -- 172.21.15.87:0/3931571776 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8281a6c20 con 0x7fd828103af0 2024-04-22T13:33:53.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82caa8640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005940 tx=0x7fd81c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd8267fc640 1 -- 172.21.15.87:0/3931571776 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c043020 con 0x7fd828103af0 2024-04-22T13:33:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd8267fc640 1 -- 172.21.15.87:0/3931571776 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c02dc30 con 0x7fd828103af0 2024-04-22T13:33:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8281a6eb0 con 0x7fd828103af0 2024-04-22T13:33:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd8267fc640 1 -- 172.21.15.87:0/3931571776 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c003a10 con 0x7fd828103af0 2024-04-22T13:33:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.062+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8281a7370 con 0x7fd828103af0 2024-04-22T13:33:53.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.064+0000 7fd8267fc640 1 -- 172.21.15.87:0/3931571776 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd81c046020 con 0x7fd828103af0 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 Processor -- start 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 -- start start 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562bef612500 con 0x562bef6b6810 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562bef5b86a0 con 0x562bef6ad310 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8321cf740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562bef60e4c0 con 0x562bef6adc30 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57560/0 (socket says 172.21.15.87:57560) 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 -- 172.21.15.87:0/1986171137 learned_addr learned my addr 172.21.15.87:0/1986171137 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8267fc640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82daaa640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd8267fc640 1 -- 172.21.15.87:0/3931571776 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd81c0f2790 con 0x7fd828103af0 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 -- 172.21.15.87:0/1986171137 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 msgr2=0x562bef6b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 -- 172.21.15.87:0/1986171137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 msgr2=0x562bef6ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82caa8640 1 -- 172.21.15.87:0/1986171137 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bef5d75e0 con 0x562bef6b6810 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.065+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd82caa8640 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd81c0be5b0 tx=0x7fd81c02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70400a7f37e3c118 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 0x7fd8100a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd818009fd0 tx=0x7fd818004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd80f7fe640 1 -- 172.21.15.87:0/1986171137 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c0befb0 con 0x562bef6b6810 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd80f7fe640 1 -- 172.21.15.87:0/1986171137 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c0be8b0 con 0x562bef6b6810 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd80f7fe640 1 -- 172.21.15.87:0/1986171137 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c0bcc40 con 0x562bef6b6810 2024-04-22T13:33:53.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 -- 172.21.15.87:0/1986171137 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 msgr2=0x562bef6b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd81c0be5b0 tx=0x7fd81c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 -- 172.21.15.87:0/1986171137 shutdown_connections 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b8c00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 --2- 172.21.15.87:0/1986171137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 -- 172.21.15.87:0/1986171137 >> 172.21.15.87:0/1986171137 conn(0x562bef6a7a40 msgr2=0x562bef6a7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 -- 172.21.15.87:0/1986171137 shutdown_connections 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.066+0000 7fd8321cf740 1 -- 172.21.15.87:0/1986171137 wait complete. 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 -- start start 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 Processor -- start 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 0x562bef6b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6adc30 0x562bef6b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562bef612500 con 0x562bef6b6810 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562bef5cba60 con 0x562bef6adc30 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562bef6b93c0 con 0x562bef6ad310 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd8321cf740 10 client.0 _ll_register_callbacks cb 0x562bef324110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd82daaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd82daaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57572/0 (socket says 172.21.15.87:57572) 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd82daaa640 1 -- 172.21.15.87:0/191073146 learned_addr learned my addr 172.21.15.87:0/191073146 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd82caa8640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6adc30 0x562bef6b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.067+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 0x562bef6b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 -- 172.21.15.87:0/191073146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 msgr2=0x562bef6b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 0x562bef6b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 -- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6adc30 msgr2=0x562bef6b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6adc30 0x562bef6b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 -- 172.21.15.87:0/191073146 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bef324920 con 0x562bef6b6810 2024-04-22T13:33:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 0x562bef6b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd82daaa640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd818011a70 tx=0x7fd818004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03021400ef12502 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818026020 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd818022460 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd81c0be8b0 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd818031050 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd8321cf740 10 client.16347 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562bef3f0d30 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd81c0be8b0 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.068+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd818020530 con 0x562bef6b6810 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 10 client.16347 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 10 client.16347 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 20 client.16347 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:53.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef6c6730 0x562bef6c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 4 client.16347 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bef5e5a10 con 0x562bef6c6730 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef6c6730 0x562bef6c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd80cff9640 10 client.16347 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.069+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef6c6730 0x562bef6c8ba0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caef29490b45aac0 server_cookie=6a5d40997704ee38 in_seq=0 out_seq=0 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd80cff9640 1 -- 172.21.15.87:0/191073146 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bef5e5a10 con 0x562bef6c6730 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd80cff9640 10 client.16347 handle_command_reply: tid=0 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 client.16347 shutdown 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 msgr2=0x562bef6b40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd818011a70 tx=0x7fd818004ab0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 shutdown_connections 2024-04-22T13:33:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef6c6730 0x562bef6c8ba0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b40f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6adc30 0x562bef6b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd80cff9640 0 client.16347 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 --2- 172.21.15.87:0/191073146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6ad310 0x562bef6b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 >> 172.21.15.87:0/191073146 conn(0x562bef6a7a40 msgr2=0x562bef6a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 shutdown_connections 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 1 -- 172.21.15.87:0/191073146 wait complete. 2024-04-22T13:33:53.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.070+0000 7fd8321cf740 20 client.16347 trim_cache size 0 max 16384 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 Processor -- start 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 -- start start 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6ce470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6b6810 0x562bef6ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281b24c0 con 0x562bef6ad310 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281afd90 con 0x562bef6b6810 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd8321cf740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281adc00 con 0x562bef6adc30 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd82d2a9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd82d2a9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57578/0 (socket says 172.21.15.87:57578) 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.087+0000 7fd82d2a9640 1 -- 172.21.15.87:0/3296254218 learned_addr learned my addr 172.21.15.87:0/3296254218 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6b6810 0x562bef6ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82caa8640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 -- 172.21.15.87:0/3296254218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 msgr2=0x562bef6ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 -- 172.21.15.87:0/3296254218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6b6810 msgr2=0x562bef6ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6b6810 0x562bef6ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 -- 172.21.15.87:0/3296254218 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bef6ada60 con 0x562bef6ad310 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82caa8640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd820009d30 tx=0x7fd820005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364747882a3641ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd80effd640 1 -- 172.21.15.87:0/3296254218 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820004230 con 0x562bef6ad310 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd80effd640 1 -- 172.21.15.87:0/3296254218 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8200043d0 con 0x562bef6ad310 2024-04-22T13:33:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.088+0000 7fd80effd640 1 -- 172.21.15.87:0/3296254218 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200201d0 con 0x562bef6ad310 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- 172.21.15.87:0/3296254218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 msgr2=0x562bef6c2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd820009d30 tx=0x7fd820005f90 comp rx=0 tx=0).stop 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- 172.21.15.87:0/3296254218 shutdown_connections 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6b6810 0x562bef6ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3296254218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6ad310 0x562bef6c2130 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- 172.21.15.87:0/3296254218 >> 172.21.15.87:0/3296254218 conn(0x562bef6a7a40 msgr2=0x562bef6a8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- 172.21.15.87:0/3296254218 shutdown_connections 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- 172.21.15.87:0/3296254218 wait complete. 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 -- start start 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 Processor -- start 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.089+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6be8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281ae250 con 0x562bef6b6810 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281af6c0 con 0x562bef6ad310 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281ae920 con 0x562bef6adc30 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 10 client.0 _ll_register_callbacks cb 0x7fd8180145e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82caa8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:47402/0 (socket says 172.21.15.87:47402) 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82caa8640 1 -- 172.21.15.87:0/3871267778 learned_addr learned my addr 172.21.15.87:0/3871267778 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82d2a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6be8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 -- 172.21.15.87:0/3871267778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 msgr2=0x562bef6b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 -- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 msgr2=0x562bef6be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6be8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 -- 172.21.15.87:0/3871267778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bef6ba1e0 con 0x562bef6b6810 2024-04-22T13:33:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6be8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd82daaa640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b9b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd818005c40 tx=0x7fd81800f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02613a60712f592 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd80cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818026040 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd818014070 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.090+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8200043d0 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd818043050 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 10 client.16353 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562bef72db40 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8200043d0 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd818003cc0 con 0x562bef6b6810 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 10 client.16353 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 10 client.16353 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 20 client.16353 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef72e190 0x562bef730580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 4 client.16353 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1e2e33c6", "release": true, "await": true}] 2024-04-22T13:33:53.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1e2e33c6", "release": true, "await": true}) -- 0x562bef61ff20 con 0x562bef72e190 2024-04-22T13:33:53.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef72e190 0x562bef730580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:53.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.091+0000 7fd82d2a9640 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef72e190 0x562bef730580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5db67a2cf745bebb server_cookie=505725a2d174a320 in_seq=0 out_seq=0 2024-04-22T13:33:53.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.092+0000 7fd80cff9640 10 client.16353 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd80cff9640 1 -- 172.21.15.87:0/3871267778 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x562bef61ff20 con 0x562bef72e190 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd80cff9640 10 client.16353 handle_command_reply: tid=0 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 client.16353 shutdown 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 msgr2=0x562bef6b9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b9b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd818005c40 tx=0x7fd81800f870 comp rx=0 tx=0).stop 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 shutdown_connections 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562bef72e190 0x562bef730580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562bef6b6810 0x562bef6b9b40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd80cff9640 0 client.16353 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562bef6adc30 0x562bef6b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 --2- 172.21.15.87:0/3871267778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bef6ad310 0x562bef6be8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 >> 172.21.15.87:0/3871267778 conn(0x562bef6a7a40 msgr2=0x562bef6a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 shutdown_connections 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 1 -- 172.21.15.87:0/3871267778 wait complete. 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.095+0000 7fd8321cf740 20 client.16353 trim_cache size 0 max 16384 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:33:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 12, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "1e2e33c6": { 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 12, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 83.5, 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:33:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:33:53.098 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 msgr2=0x7fd8100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 0x7fd8100a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd818009fd0 tx=0x7fd818004a40 comp rx=0 tx=0).stop 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 msgr2=0x7fd828101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd81c005940 tx=0x7fd81c0045e0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 shutdown_connections 2024-04-22T13:33:53.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd82810c510 0x7fd8281018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd828103af0 0x7fd828101360 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 --2- 172.21.15.87:0/3931571776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8281031d0 0x7fd828100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 >> 172.21.15.87:0/3931571776 conn(0x7fd828077f50 msgr2=0x7fd8280fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 shutdown_connections 2024-04-22T13:33:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:33:53.096+0000 7fd82f50d640 1 -- 172.21.15.87:0/3931571776 wait complete. 2024-04-22T13:33:53.100 INFO:tasks.workunit.client.0.smithi087.stdout:getting iogen 2024-04-22T13:33:53.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'getting iogen' 2024-04-22T13:33:53.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T13:33:53.103 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 1e2e33c6, seconds elapsed: 24.6 2024-04-22T13:33:53.112 INFO:tasks.workunit.client.0.smithi087.stderr:--2024-04-22 13:33:53-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T13:33:53.112 INFO:tasks.workunit.client.0.smithi087.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-22T13:33:53.160 INFO:tasks.workunit.client.0.smithi087.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-22T13:33:53.204 INFO:tasks.workunit.client.0.smithi087.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T13:33:53.204 INFO:tasks.workunit.client.0.smithi087.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-22T13:33:53.205 INFO:tasks.workunit.client.0.smithi087.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-22T13:33:53.208 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:53.209 INFO:tasks.workunit.client.0.smithi087.stderr: 0K ....... 100% 329M=0s 2024-04-22T13:33:53.209 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:53.209 INFO:tasks.workunit.client.0.smithi087.stderr:2024-04-22 13:33:53 (329 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-22T13:33:53.209 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-04-22T13:33:53.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-22T13:33:53.219 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0 2024-04-22T13:33:53.220 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/src 2024-04-22T13:33:53.221 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/src/iogen.8 2024-04-22T13:33:53.222 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/src/iogen.c 2024-04-22T13:33:53.223 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/README 2024-04-22T13:33:53.224 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/Makefile.bsd 2024-04-22T13:33:53.225 INFO:tasks.workunit.client.0.smithi087.stdout:iogen_3.1p0/Makefile 2024-04-22T13:33:53.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ cd iogen_3.1p0 2024-04-22T13:33:53.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'making iogen' 2024-04-22T13:33:53.227 INFO:tasks.workunit.client.0.smithi087.stdout:making iogen 2024-04-22T13:33:53.228 INFO:tasks.workunit.client.0.smithi087.stderr:+ make 2024-04-22T13:33:53.235 INFO:tasks.workunit.client.0.smithi087.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-22T13:33:53.477 INFO:tasks.workunit.client.0.smithi087.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-22T13:33:53.575 INFO:tasks.workunit.client.0.smithi087.stdout:running iogen 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'running iogen' 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:+ ./iogen -n 5 -s 2g 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:child spawned 84109 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:child spawned 84110 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:child spawned 84111 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:child spawned 84112 2024-04-22T13:33:53.576 INFO:tasks.workunit.client.0.smithi087.stderr:child spawned 84113 2024-04-22T13:33:53.577 INFO:tasks.workunit.client.0.smithi087.stdout:sleep for 10 min 2024-04-22T13:33:53.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'sleep for 10 min' 2024-04-22T13:33:53.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ sleep 600 2024-04-22T13:33:53.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:53 smithi120 ceph-mon[31101]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:53 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1671495094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:53.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:53 smithi132 ceph-mon[31531]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:53.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:53 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1671495094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:53.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:53 smithi087 ceph-mon[27999]: pgmap v63: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:53 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1671495094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:33:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:55 smithi120 ceph-mon[31101]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:33:55.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:55 smithi132 ceph-mon[31531]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:33:55.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:55 smithi087 ceph-mon[27999]: pgmap v64: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:33:57.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:57 smithi120 ceph-mon[31101]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:57.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:57 smithi132 ceph-mon[31531]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:57 smithi087 ceph-mon[27999]: pgmap v65: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:33:58.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:58 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:33:58.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:58 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:33:58.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:58 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:33:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 -- 172.21.15.87:0/299301586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 msgr2=0x7f6a601108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 --2- 172.21.15.87:0/299301586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601108a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009860 tx=0x7f6a5002bb70 comp rx=0 tx=0).stop 2024-04-22T13:33:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 -- 172.21.15.87:0/299301586 shutdown_connections 2024-04-22T13:33:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 --2- 172.21.15.87:0/299301586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a6010ecc0 0x7f6a60108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 --2- 172.21.15.87:0/299301586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a6010e220 0x7f6a6010e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 --2- 172.21.15.87:0/299301586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601108a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 -- 172.21.15.87:0/299301586 >> 172.21.15.87:0/299301586 conn(0x7f6a6006cf00 msgr2=0x7f6a6006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 -- 172.21.15.87:0/299301586 shutdown_connections 2024-04-22T13:33:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.974+0000 7f6a668b2640 1 -- 172.21.15.87:0/299301586 wait complete. 2024-04-22T13:33:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.975+0000 7f6a668b2640 1 Processor -- start 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.976+0000 7f6a668b2640 1 -- start start 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.976+0000 7f6a668b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a6010e220 0x7f6a601a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a668b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a6010ecc0 0x7f6a601a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a668b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a668b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60120e80 con 0x7f6a601104c0 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a668b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60120d00 con 0x7f6a6010e220 2024-04-22T13:33:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a668b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60121000 con 0x7f6a6010ecc0 2024-04-22T13:33:58.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:58.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57608/0 (socket says 172.21.15.87:57608) 2024-04-22T13:33:58.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 -- 172.21.15.87:0/1598887347 learned_addr learned my addr 172.21.15.87:0/1598887347 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 -- 172.21.15.87:0/1598887347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a6010ecc0 msgr2=0x7f6a601a7980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a6010ecc0 0x7f6a601a7980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 -- 172.21.15.87:0/1598887347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a6010e220 msgr2=0x7f6a601a7440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.977+0000 7f6a64e4f640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a6010e220 0x7f6a601a7440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a64e4f640 1 -- 172.21.15.87:0/1598887347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a601ac3e0 con 0x7f6a601104c0 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a64e4f640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800b9e0 tx=0x7f6a5800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a5d7fa640 1 -- 172.21.15.87:0/1598887347 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58019070 con 0x7f6a601104c0 2024-04-22T13:33:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a5d7fa640 1 -- 172.21.15.87:0/1598887347 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a58014440 con 0x7f6a601104c0 2024-04-22T13:33:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a5d7fa640 1 -- 172.21.15.87:0/1598887347 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580125a0 con 0x7f6a601104c0 2024-04-22T13:33:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.978+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a601ac6d0 con 0x7f6a601104c0 2024-04-22T13:33:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.979+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a601b3f60 con 0x7f6a601104c0 2024-04-22T13:33:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.983+0000 7f6a69576740 1 Processor -- start 2024-04-22T13:33:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.984+0000 7f6a5d7fa640 1 -- 172.21.15.87:0/1598887347 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6a5801c070 con 0x7f6a601104c0 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.984+0000 7f6a69576740 1 -- start start 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57612/0 (socket says 172.21.15.87:57612) 2024-04-22T13:33:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5583b540e8d0 con 0x5583b54a9d50 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1081728637 learned_addr learned my addr 172.21.15.87:0/1081728637 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5583b53abe80 con 0x5583b54aaac0 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.985+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5583b540a890 con 0x5583b54b32d0 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5d7fa640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 0x7f6a480a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1081728637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 msgr2=0x5583b54b56c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b56c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1081728637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 msgr2=0x5583b54a9810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54a9810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1081728637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b53cd320 con 0x5583b54a9d50 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a5d7fa640 1 -- 172.21.15.87:0/1598887347 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6a580cfbf0 con 0x7f6a601104c0 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.986+0000 7f6a64e4f640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 0x7f6a480a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.987+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009980 tx=0x7f6a500094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7ec320ae80b8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.987+0000 7f6a64e4f640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 0x7f6a480a62a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a580cd520 tx=0x7f6a58004b50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.987+0000 7f6a467fc640 1 -- 172.21.15.87:0/1081728637 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50039040 con 0x5583b54a9d50 2024-04-22T13:33:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.987+0000 7f6a467fc640 1 -- 172.21.15.87:0/1081728637 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a50034700 con 0x5583b54a9d50 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a467fc640 1 -- 172.21.15.87:0/1081728637 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500052b0 con 0x5583b54a9d50 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 msgr2=0x5583b54aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009980 tx=0x7f6a500094c0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 shutdown_connections 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54a9d50 0x5583b54aa1d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 --2- 172.21.15.87:0/1081728637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.988+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 >> 172.21.15.87:0/1081728637 conn(0x5583b5470c90 msgr2=0x5583b54a4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 shutdown_connections 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 -- 172.21.15.87:0/1081728637 wait complete. 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 -- start start 2024-04-22T13:33:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 Processor -- start 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54afff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.989+0000 7f6a69576740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5583b54b6040 con 0x5583b54b32d0 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a69576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5583b5396eb0 con 0x5583b54aaac0 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a69576740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5583b54b61c0 con 0x5583b54a9d50 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a69576740 10 client.0 _ll_register_callbacks cb 0x5583b53c3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57624/0 (socket says 172.21.15.87:57624) 2024-04-22T13:33:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 -- 172.21.15.87:0/1095114722 learned_addr learned my addr 172.21.15.87:0/1095114722 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 -- 172.21.15.87:0/1095114722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 msgr2=0x5583b54b2c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54b2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 -- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 msgr2=0x5583b54afff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54afff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 -- 172.21.15.87:0/1095114722 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b5121210 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400bb20 tx=0x7f6a5400d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e41fc5a8336a8f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54016070 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.990+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a50034700 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6a54010460 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6a54020050 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 10 client.16365 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5583b51ed620 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a54010b70 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6a54023020 con 0x5583b54b32d0 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 10 client.16365 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 20 client.16365 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b54c5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 4 client.16365 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:33:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.991+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6a50034700 con 0x5583b54c3240 2024-04-22T13:33:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.993+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b54c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.997+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b54c5630 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb69f187c33beb47 server_cookie=ecd16a271dde6953 in_seq=0 out_seq=0 2024-04-22T13:33:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:58.997+0000 7f6a3f7fe640 10 client.16365 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.001+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1095114722 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f6a50034700 con 0x5583b54c3240 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a3f7fe640 10 client.16365 handle_command_reply: tid=0 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 client.16365 shutdown 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 msgr2=0x5583b54b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400bb20 tx=0x7f6a5400d2e0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 shutdown_connections 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b54c5630 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a3f7fe640 0 client.16365 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54b32d0 0x5583b54b0570 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54aaac0 0x5583b54afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 --2- 172.21.15.87:0/1095114722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 >> 172.21.15.87:0/1095114722 conn(0x5583b5470c90 msgr2=0x5583b54a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 shutdown_connections 2024-04-22T13:33:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 1 -- 172.21.15.87:0/1095114722 wait complete. 2024-04-22T13:33:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.002+0000 7f6a69576740 20 client.16365 trim_cache size 0 max 16384 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 Processor -- start 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 -- start start 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54a9d50 0x5583b54bef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601befb0 con 0x5583b54aaac0 2024-04-22T13:33:59.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601bc880 con 0x5583b54a9d50 2024-04-22T13:33:59.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.024+0000 7f6a69576740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601ba6d0 con 0x5583b54b32d0 2024-04-22T13:33:59.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57626/0 (socket says 172.21.15.87:57626) 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 -- 172.21.15.87:0/2341870175 learned_addr learned my addr 172.21.15.87:0/2341870175 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 -- 172.21.15.87:0/2341870175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 msgr2=0x5583b54b2c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 -- 172.21.15.87:0/2341870175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54a9d50 msgr2=0x5583b54bef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54a9d50 0x5583b54bef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 -- 172.21.15.87:0/2341870175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b54c1d60 con 0x5583b54aaac0 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.025+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a5002bbb0 tx=0x7f6a500094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c02019f1f6a7b95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.027+0000 7f6a45ffb640 1 -- 172.21.15.87:0/2341870175 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5004f070 con 0x5583b54aaac0 2024-04-22T13:33:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.027+0000 7f6a45ffb640 1 -- 172.21.15.87:0/2341870175 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a50005500 con 0x5583b54aaac0 2024-04-22T13:33:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.027+0000 7f6a45ffb640 1 -- 172.21.15.87:0/2341870175 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5004a070 con 0x5583b54aaac0 2024-04-22T13:33:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 -- 172.21.15.87:0/2341870175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 msgr2=0x5583b54b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a5002bbb0 tx=0x7f6a500094c0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 -- 172.21.15.87:0/2341870175 shutdown_connections 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54b32d0 0x5583b54b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54b2720 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.028+0000 7f6a69576740 1 --2- 172.21.15.87:0/2341870175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54a9d50 0x5583b54bef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.029+0000 7f6a69576740 1 -- 172.21.15.87:0/2341870175 >> 172.21.15.87:0/2341870175 conn(0x5583b5470c90 msgr2=0x5583b54ab250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.029+0000 7f6a69576740 1 -- 172.21.15.87:0/2341870175 shutdown_connections 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.029+0000 7f6a69576740 1 -- 172.21.15.87:0/2341870175 wait complete. 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.029+0000 7f6a69576740 1 -- start start 2024-04-22T13:33:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.030+0000 7f6a69576740 1 Processor -- start 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.030+0000 7f6a69576740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54bb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.030+0000 7f6a69576740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.030+0000 7f6a69576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 0x5583b54ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a69576740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601bad40 con 0x5583b54aaac0 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a69576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601bc1b0 con 0x5583b54b32d0 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a69576740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6a601bb410 con 0x5583b54a9d50 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57632/0 (socket says 172.21.15.87:57632) 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1735500302 learned_addr learned my addr 172.21.15.87:0/1735500302 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:33:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a69576740 10 client.0 _ll_register_callbacks cb 0x7f6a5400b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:33:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1735500302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 msgr2=0x5583b54bb690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:33:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.031+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54bb690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.036+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 0x5583b54ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.036+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 msgr2=0x5583b54ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:33:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.036+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 0x5583b54ad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.036+0000 7f6a5ffff640 1 -- 172.21.15.87:0/1735500302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b54addf0 con 0x5583b54aaac0 2024-04-22T13:33:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.036+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 0x5583b54ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:33:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.037+0000 7f6a5ffff640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009950 tx=0x7f6a50005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee32adfac9c6cae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:33:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.037+0000 7f6a3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:33:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.037+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5004f070 con 0x5583b54aaac0 2024-04-22T13:33:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.037+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6a50005050 con 0x5583b54aaac0 2024-04-22T13:33:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.037+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a50005500 con 0x5583b54aaac0 2024-04-22T13:33:59.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6a5005a050 con 0x5583b54aaac0 2024-04-22T13:33:59.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 10 client.16371 fetch_fsmap learned FSMap version 21 2024-04-22T13:33:59.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5583b55297b0 con 0x5583b54aaac0 2024-04-22T13:33:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a50005500 con 0x5583b54aaac0 2024-04-22T13:33:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6a5003c070 con 0x5583b54aaac0 2024-04-22T13:33:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 10 client.16371 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:33:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:33:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 20 client.16371 populate_metadata read hostname 'smithi087' 2024-04-22T13:33:59.036 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b552b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:33:59.036 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 4 client.16371 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:33:59.036 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.039+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583b5422880 con 0x5583b54c3240 2024-04-22T13:33:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.045+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b552b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:33:59.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.046+0000 7f6a5f7fe640 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b552b990 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad470ea77330da5 server_cookie=3999488a966e95ea in_seq=0 out_seq=0 2024-04-22T13:33:59.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.046+0000 7f6a3f7fe640 10 client.16371 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:59.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:59 smithi132 ceph-mon[31531]: pgmap v66: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 292 MiB/s wr, 239 op/s 2024-04-22T13:33:59.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:33:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:59 smithi087 ceph-mon[27999]: pgmap v66: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 292 MiB/s wr, 239 op/s 2024-04-22T13:33:59.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:33:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.877+0000 7f6a3f7fe640 1 -- 172.21.15.87:0/1735500302 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583b5422880 con 0x5583b54c3240 2024-04-22T13:33:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.877+0000 7f6a3f7fe640 10 client.16371 handle_command_reply: tid=0 2024-04-22T13:33:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.877+0000 7f6a69576740 1 client.16371 shutdown 2024-04-22T13:33:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 msgr2=0x5583b54ad270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009950 tx=0x7f6a50005d80 comp rx=0 tx=0).stop 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 shutdown_connections 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5583b54c3240 0x5583b552b990 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b54b32d0 0x5583b54ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5583b54aaac0 0x5583b54ad270 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 --2- 172.21.15.87:0/1735500302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5583b54a9d50 0x5583b54bb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 >> 172.21.15.87:0/1735500302 conn(0x5583b5470c90 msgr2=0x5583b54ae490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a3f7fe640 0 client.16371 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 shutdown_connections 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.878+0000 7f6a69576740 1 -- 172.21.15.87:0/1735500302 wait complete. 2024-04-22T13:33:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.879+0000 7f6a69576740 20 client.16371 trim_cache size 0 max 16384 2024-04-22T13:33:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:33:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:33:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:33:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 msgr2=0x7f6a480a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 0x7f6a480a62a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a580cd520 tx=0x7f6a58004b50 comp rx=0 tx=0).stop 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 msgr2=0x7f6a601abcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800b9e0 tx=0x7f6a5800bea0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 shutdown_connections 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6a480a3de0 0x7f6a480a62a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a601104c0 0x7f6a601abcb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a6010ecc0 0x7f6a601a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 --2- 172.21.15.87:0/1598887347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a6010e220 0x7f6a601a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:33:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 >> 172.21.15.87:0/1598887347 conn(0x7f6a6006cf00 msgr2=0x7f6a6010a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:33:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 shutdown_connections 2024-04-22T13:33:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:33:59.880+0000 7f6a668b2640 1 -- 172.21.15.87:0/1598887347 wait complete. 2024-04-22T13:33:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:59 smithi120 ceph-mon[31101]: pgmap v66: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 292 MiB/s wr, 239 op/s 2024-04-22T13:33:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:33:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:33:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:34:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:01 smithi132 ceph-mon[31531]: pgmap v67: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:01 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:34:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:01 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:34:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:01 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:34:01.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:01 smithi087 ceph-mon[27999]: pgmap v67: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:01.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:01 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:34:01.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:01 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:34:01.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:01 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:34:01.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:01 smithi120 ceph-mon[31101]: pgmap v67: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:01.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:01 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:34:01.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:01 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:34:01.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:01 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:34:03.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:03 smithi132 ceph-mon[31531]: pgmap v68: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:03.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:03 smithi087 ceph-mon[27999]: pgmap v68: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:03.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:03 smithi120 ceph-mon[31101]: pgmap v68: 97 pgs: 97 active+clean; 3.4 GiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 292 MiB/s wr, 236 op/s 2024-04-22T13:34:05.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:05 smithi132 ceph-mon[31531]: pgmap v69: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 505 op/s 2024-04-22T13:34:05.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:05 smithi087 ceph-mon[27999]: pgmap v69: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 505 op/s 2024-04-22T13:34:05.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:05 smithi120 ceph-mon[31101]: pgmap v69: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 505 op/s 2024-04-22T13:34:06.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:34:06.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:34:06.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.724+0000 7ff646ffd640 1 -- 172.21.15.87:0/1563040519 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c02cc30 con 0x7ff648104930 2024-04-22T13:34:06.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 -- 172.21.15.87:0/1563040519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff648104930 msgr2=0x7ff648104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 --2- 172.21.15.87:0/1563040519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff648104930 0x7ff648104db0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009900 tx=0x7ff63c02bc80 comp rx=0 tx=0).stop 2024-04-22T13:34:06.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 -- 172.21.15.87:0/1563040519 shutdown_connections 2024-04-22T13:34:06.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 --2- 172.21.15.87:0/1563040519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff648104930 0x7ff648104db0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 --2- 172.21.15.87:0/1563040519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff64806aaf0 0x7ff64806af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 --2- 172.21.15.87:0/1563040519 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648106f40 0x7ff64806a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.725+0000 7ff64e754640 1 -- 172.21.15.87:0/1563040519 >> 172.21.15.87:0/1563040519 conn(0x7ff648077c40 msgr2=0x7ff648078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.728+0000 7ff64e754640 1 -- 172.21.15.87:0/1563040519 shutdown_connections 2024-04-22T13:34:06.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.728+0000 7ff64e754640 1 -- 172.21.15.87:0/1563040519 wait complete. 2024-04-22T13:34:06.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.729+0000 7ff64e754640 1 Processor -- start 2024-04-22T13:34:06.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.729+0000 7ff64e754640 1 -- start start 2024-04-22T13:34:06.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648104930 0x7ff6481a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff648106f40 0x7ff64819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff648118840 con 0x7ff64806aaf0 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481186c0 con 0x7ff648106f40 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff64e754640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481189c0 con 0x7ff648104930 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57322/0 (socket says 172.21.15.87:57322) 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.730+0000 7ff647fff640 1 -- 172.21.15.87:0/3545986012 learned_addr learned my addr 172.21.15.87:0/3545986012 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff6477fe640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648104930 0x7ff6481a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff647fff640 1 -- 172.21.15.87:0/3545986012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648104930 msgr2=0x7ff6481a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff647fff640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648104930 0x7ff6481a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff647fff640 1 -- 172.21.15.87:0/3545986012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff648106f40 msgr2=0x7ff64819eb00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:06.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff647fff640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff648106f40 0x7ff64819eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.731+0000 7ff647fff640 1 -- 172.21.15.87:0/3545986012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64819f280 con 0x7ff64806aaf0 2024-04-22T13:34:06.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.732+0000 7ff647fff640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff64000ba50 tx=0x7ff64000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.732+0000 7ff6457fa640 1 -- 172.21.15.87:0/3545986012 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640019070 con 0x7ff64806aaf0 2024-04-22T13:34:06.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.732+0000 7ff6457fa640 1 -- 172.21.15.87:0/3545986012 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff64000cc10 con 0x7ff64806aaf0 2024-04-22T13:34:06.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.733+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64819f570 con 0x7ff64806aaf0 2024-04-22T13:34:06.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.733+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6481ab890 con 0x7ff64806aaf0 2024-04-22T13:34:06.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.734+0000 7ff6457fa640 1 -- 172.21.15.87:0/3545986012 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640019070 con 0x7ff64806aaf0 2024-04-22T13:34:06.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.738+0000 7ff6457fa640 1 -- 172.21.15.87:0/3545986012 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff64001c070 con 0x7ff64806aaf0 2024-04-22T13:34:06.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 Processor -- start 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 -- start start 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711af960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5560711141e0 con 0x5560711b8a80 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5560710b1390 con 0x5560711afea0 2024-04-22T13:34:06.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.739+0000 7ff651417740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5560711101a0 con 0x5560711af580 2024-04-22T13:34:06.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6457fa640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 0x7ff6300a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57328/0 (socket says 172.21.15.87:57328) 2024-04-22T13:34:06.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6477fe640 1 -- 172.21.15.87:0/832855283 learned_addr learned my addr 172.21.15.87:0/832855283 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:06.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff64ccf1640 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff64ccf1640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 0x7ff6300a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6457fa640 1 -- 172.21.15.87:0/3545986012 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff6400cf9c0 con 0x7ff64806aaf0 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff647fff640 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6477fe640 1 -- 172.21.15.87:0/832855283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 msgr2=0x5560711af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.740+0000 7ff6477fe640 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.741+0000 7ff6477fe640 1 -- 172.21.15.87:0/832855283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 msgr2=0x5560711b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.741+0000 7ff6477fe640 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.741+0000 7ff6477fe640 1 -- 172.21.15.87:0/832855283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556070ef07d0 con 0x5560711b8a80 2024-04-22T13:34:06.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.741+0000 7ff6477fe640 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff63800a860 tx=0x7ff63800ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0a8568bd532005 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.741+0000 7ff62e7fc640 1 -- 172.21.15.87:0/832855283 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638015070 con 0x5560711b8a80 2024-04-22T13:34:06.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.742+0000 7ff62e7fc640 1 -- 172.21.15.87:0/832855283 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6380113f0 con 0x5560711b8a80 2024-04-22T13:34:06.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.742+0000 7ff62e7fc640 1 -- 172.21.15.87:0/832855283 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638015070 con 0x5560711b8a80 2024-04-22T13:34:06.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.743+0000 7ff651417740 1 -- 172.21.15.87:0/832855283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 msgr2=0x5560711bae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.743+0000 7ff651417740 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff63800a860 tx=0x7ff63800ad20 comp rx=0 tx=0).stop 2024-04-22T13:34:06.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff64ccf1640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 0x7ff6300a6460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff63c036040 tx=0x7ff63c005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 -- 172.21.15.87:0/832855283 shutdown_connections 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711bae70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 --2- 172.21.15.87:0/832855283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 -- 172.21.15.87:0/832855283 >> 172.21.15.87:0/832855283 conn(0x5560711a9cb0 msgr2=0x5560711aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.744+0000 7ff651417740 1 -- 172.21.15.87:0/832855283 shutdown_connections 2024-04-22T13:34:06.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.745+0000 7ff651417740 1 -- 172.21.15.87:0/832855283 wait complete. 2024-04-22T13:34:06.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.745+0000 7ff651417740 1 -- start start 2024-04-22T13:34:06.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.746+0000 7ff651417740 1 Processor -- start 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5560711cc880 con 0x5560711b8a80 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5560710cd2f0 con 0x5560711afea0 2024-04-22T13:34:06.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5560711cca00 con 0x5560711af580 2024-04-22T13:34:06.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.747+0000 7ff651417740 10 client.0 _ll_register_callbacks cb 0x5560710da580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:06.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.752+0000 7ff647fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.752+0000 7ff647fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39402/0 (socket says 172.21.15.87:39402) 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.752+0000 7ff647fff640 1 -- 172.21.15.87:0/1449602948 learned_addr learned my addr 172.21.15.87:0/1449602948 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.753+0000 7ff647fff640 1 -- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 msgr2=0x5560711b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.753+0000 7ff647fff640 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.753+0000 7ff647fff640 1 -- 172.21.15.87:0/1449602948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 msgr2=0x5560711b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.753+0000 7ff647fff640 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.753+0000 7ff647fff640 1 -- 172.21.15.87:0/1449602948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556070e263f0 con 0x5560711af580 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.755+0000 7ff647fff640 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff64000c7d0 tx=0x7ff64009bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de2bc0549c1dc75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.755+0000 7ff6277fe640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.756+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640014980 con 0x5560711af580 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.756+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff6400cd3a0 con 0x5560711af580 2024-04-22T13:34:06.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.756+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6380113f0 con 0x5560711af580 2024-04-22T13:34:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.757+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff6400980f0 con 0x5560711af580 2024-04-22T13:34:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.758+0000 7ff651417740 10 client.25651 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:06.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.758+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556070de60f0 con 0x5560711af580 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.759+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6380113f0 con 0x5560711af580 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.759+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff640014440 con 0x5560711af580 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 10 client.25651 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 10 client.25651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 20 client.25651 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5560711d9f90 0x5560711dc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 4 client.25651 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:06.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.760+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556071121c00 con 0x5560711d9f90 2024-04-22T13:34:06.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.762+0000 7ff6477fe640 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5560711d9f90 0x5560711dc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.768+0000 7ff6477fe640 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5560711d9f90 0x5560711dc430 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed27bbd7629f5831 server_cookie=66ea5592a98865a8 in_seq=0 out_seq=0 2024-04-22T13:34:06.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.769+0000 7ff6277fe640 10 client.25651 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:06.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.770+0000 7ff6277fe640 1 -- 172.21.15.87:0/1449602948 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556071121c00 con 0x5560711d9f90 2024-04-22T13:34:06.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.771+0000 7ff6277fe640 10 client.25651 handle_command_reply: tid=0 2024-04-22T13:34:06.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.771+0000 7ff651417740 1 client.25651 shutdown 2024-04-22T13:34:06.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.771+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 msgr2=0x5560711b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.771+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff64000c7d0 tx=0x7ff64009bb40 comp rx=0 tx=0).stop 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 shutdown_connections 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5560711d9f90 0x5560711dc430 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 --2- 172.21.15.87:0/1449602948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711b5a80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.772+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 >> 172.21.15.87:0/1449602948 conn(0x5560711a9cb0 msgr2=0x5560711aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.773+0000 7ff6277fe640 0 client.25651 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:06.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.773+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 shutdown_connections 2024-04-22T13:34:06.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.773+0000 7ff651417740 1 -- 172.21.15.87:0/1449602948 wait complete. 2024-04-22T13:34:06.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.774+0000 7ff651417740 20 client.25651 trim_cache size 0 max 16384 2024-04-22T13:34:06.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.836+0000 7ff651417740 1 Processor -- start 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.840+0000 7ff651417740 1 -- start start 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711d5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481ae210 con 0x5560711b8a80 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481abb80 con 0x5560711afea0 2024-04-22T13:34:06.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.841+0000 7ff651417740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481014b0 con 0x5560711af580 2024-04-22T13:34:06.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.842+0000 7ff647fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711d5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.843+0000 7ff64ccf1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.843+0000 7ff64ccf1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711e1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57342/0 (socket says 172.21.15.87:57342) 2024-04-22T13:34:06.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.843+0000 7ff64ccf1640 1 -- 172.21.15.87:0/4294828768 learned_addr learned my addr 172.21.15.87:0/4294828768 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:06.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.843+0000 7ff6477fe640 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.845+0000 7ff6477fe640 1 -- 172.21.15.87:0/4294828768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 msgr2=0x5560711d5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.845+0000 7ff6477fe640 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711d5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.845+0000 7ff6477fe640 1 -- 172.21.15.87:0/4294828768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 msgr2=0x5560711e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.845+0000 7ff6477fe640 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.845+0000 7ff6477fe640 1 -- 172.21.15.87:0/4294828768 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560711d8690 con 0x5560711afea0 2024-04-22T13:34:06.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.851+0000 7ff6477fe640 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b6320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x5560711aae40 tx=0x7ff63801c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eb7381822ab21be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.853+0000 7ff62dffb640 1 -- 172.21.15.87:0/4294828768 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638015070 con 0x5560711afea0 2024-04-22T13:34:06.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.853+0000 7ff62dffb640 1 -- 172.21.15.87:0/4294828768 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63800a430 con 0x5560711afea0 2024-04-22T13:34:06.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.854+0000 7ff651417740 1 -- 172.21.15.87:0/4294828768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 msgr2=0x5560711b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.855+0000 7ff651417740 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b6320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x5560711aae40 tx=0x7ff63801c040 comp rx=0 tx=0).stop 2024-04-22T13:34:06.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.857+0000 7ff651417740 1 -- 172.21.15.87:0/4294828768 shutdown_connections 2024-04-22T13:34:06.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.857+0000 7ff651417740 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711b8a80 0x5560711e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.857+0000 7ff651417740 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711afea0 0x5560711b6320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.857+0000 7ff651417740 1 --2- 172.21.15.87:0/4294828768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711d5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.857+0000 7ff651417740 1 -- 172.21.15.87:0/4294828768 >> 172.21.15.87:0/4294828768 conn(0x5560711a9cb0 msgr2=0x5560711a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.863+0000 7ff651417740 1 -- 172.21.15.87:0/4294828768 shutdown_connections 2024-04-22T13:34:06.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.866+0000 7ff651417740 1 -- 172.21.15.87:0/4294828768 wait complete. 2024-04-22T13:34:06.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.867+0000 7ff651417740 1 -- start start 2024-04-22T13:34:06.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.868+0000 7ff651417740 1 Processor -- start 2024-04-22T13:34:06.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711b8a80 0x55607123e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff64810f490 con 0x5560711afea0 2024-04-22T13:34:06.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481108e0 con 0x5560711b8a80 2024-04-22T13:34:06.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff64810fb40 con 0x5560711af580 2024-04-22T13:34:06.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.869+0000 7ff651417740 10 client.0 _ll_register_callbacks cb 0x7ff64009c180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:06.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.876+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.876+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57354/0 (socket says 172.21.15.87:57354) 2024-04-22T13:34:06.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.876+0000 7ff6477fe640 1 -- 172.21.15.87:0/1991688997 learned_addr learned my addr 172.21.15.87:0/1991688997 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:06.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.876+0000 7ff647fff640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.878+0000 7ff64ccf1640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711b8a80 0x55607123e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.886+0000 7ff6477fe640 1 -- 172.21.15.87:0/1991688997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 msgr2=0x5560711ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.886+0000 7ff6477fe640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.886+0000 7ff6477fe640 1 -- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711b8a80 msgr2=0x55607123e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:06.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.886+0000 7ff6477fe640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711b8a80 0x55607123e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.886+0000 7ff6477fe640 1 -- 172.21.15.87:0/1991688997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55607124faf0 con 0x5560711afea0 2024-04-22T13:34:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.894+0000 7ff6477fe640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff638013010 tx=0x7ff638004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1838d6e39664e2c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.894+0000 7ff6277fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:06.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.899+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638034060 con 0x5560711afea0 2024-04-22T13:34:06.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.899+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff638004250 con 0x5560711afea0 2024-04-22T13:34:06.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.900+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63800a430 con 0x5560711afea0 2024-04-22T13:34:06.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.906+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff638015040 con 0x5560711afea0 2024-04-22T13:34:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.906+0000 7ff651417740 10 client.16386 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:06.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.906+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556071250c60 con 0x5560711afea0 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63800a430 con 0x5560711afea0 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff6380037c0 con 0x5560711afea0 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 10 client.16386 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 10 client.16386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 20 client.16386 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556071251220 0x556071253640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 4 client.16386 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:34:06.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.909+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55607111ba70 con 0x556071251220 2024-04-22T13:34:06.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.911+0000 7ff647fff640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556071251220 0x556071253640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:06.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.924+0000 7ff647fff640 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556071251220 0x556071253640 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec285a6be9f32b98 server_cookie=896915787e5f36d3 in_seq=0 out_seq=0 2024-04-22T13:34:06.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.926+0000 7ff6277fe640 10 client.16386 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:06.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.929+0000 7ff6277fe640 1 -- 172.21.15.87:0/1991688997 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55607111ba70 con 0x556071251220 2024-04-22T13:34:06.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.929+0000 7ff6277fe640 10 client.16386 handle_command_reply: tid=0 2024-04-22T13:34:06.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.930+0000 7ff651417740 1 client.16386 shutdown 2024-04-22T13:34:06.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.931+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 msgr2=0x5560711cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.931+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff638013010 tx=0x7ff638004690 comp rx=0 tx=0).stop 2024-04-22T13:34:06.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 shutdown_connections 2024-04-22T13:34:06.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556071251220 0x556071253640 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560711b8a80 0x55607123e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5560711afea0 0x5560711cd150 secure :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff638013010 tx=0x7ff638004690 comp rx=0 tx=0).stop 2024-04-22T13:34:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 --2- 172.21.15.87:0/1991688997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560711af580 0x5560711ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.937+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 >> 172.21.15.87:0/1991688997 conn(0x5560711a9cb0 msgr2=0x5560711a8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.938+0000 7ff6277fe640 0 client.16386 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:06.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.944+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 shutdown_connections 2024-04-22T13:34:06.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.944+0000 7ff651417740 1 -- 172.21.15.87:0/1991688997 wait complete. 2024-04-22T13:34:06.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.947+0000 7ff651417740 20 client.16386 trim_cache size 0 max 16384 2024-04-22T13:34:06.945 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:34:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:34:06.945 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:06.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.956+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 msgr2=0x7ff6300a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.956+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 0x7ff6300a6460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff63c036040 tx=0x7ff63c005fd0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.956+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 msgr2=0x7ff6481a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:06.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.956+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff64000ba50 tx=0x7ff64000bf10 comp rx=0 tx=0).stop 2024-04-22T13:34:06.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 shutdown_connections 2024-04-22T13:34:06.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff6300a3fa0 0x7ff6300a6460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff648106f40 0x7ff64819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff648104930 0x7ff6481a4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 --2- 172.21.15.87:0/3545986012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff64806aaf0 0x7ff6481a4950 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:06.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.959+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 >> 172.21.15.87:0/3545986012 conn(0x7ff648077c40 msgr2=0x7ff6480fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:06.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.961+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 shutdown_connections 2024-04-22T13:34:06.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:06.963+0000 7ff64e754640 1 -- 172.21.15.87:0/3545986012 wait complete. 2024-04-22T13:34:06.997 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:34:06.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/187525192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa4108f50 msgr2=0x7f1aa41093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/187525192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa4108f50 0x7f1aa41093d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a980074e0 tx=0x7f1a9802c8f0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/187525192 shutdown_connections 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/187525192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/187525192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa4108f50 0x7f1aa41093d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/187525192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa410e600 0x7f1aa410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.338+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/187525192 >> 172.21.15.87:0/187525192 conn(0x7f1aa406e410 msgr2=0x7f1aa40719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.340+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/187525192 shutdown_connections 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.342+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/187525192 wait complete. 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.345+0000 7f1aa8ea0640 1 Processor -- start 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.345+0000 7f1aa8ea0640 1 -- start start 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.345+0000 7f1aa8ea0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa4108f50 0x7f1aa4084be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.345+0000 7f1aa8ea0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.345+0000 7f1aa8ea0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 0x7f1aa407f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.346+0000 7f1aa8ea0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa407d500 con 0x7f1aa4108f50 2024-04-22T13:34:07.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.346+0000 7f1aa8ea0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa407d380 con 0x7f1aa410e600 2024-04-22T13:34:07.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.346+0000 7f1aa8ea0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa407d680 con 0x7f1aa4109910 2024-04-22T13:34:07.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.347+0000 7f1aa2d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 0x7f1aa407f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:07.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.347+0000 7f1aa2d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 0x7f1aa407f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:41906/0 (socket says 172.21.15.87:41906) 2024-04-22T13:34:07.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.347+0000 7f1aa2d9d640 1 -- 172.21.15.87:0/3456664474 learned_addr learned my addr 172.21.15.87:0/3456664474 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:07.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.348+0000 7f1aa1d9b640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.350+0000 7f1aa259c640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa4108f50 0x7f1aa4084be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.353+0000 7f1aa1d9b640 1 -- 172.21.15.87:0/3456664474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 msgr2=0x7f1aa407f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.353+0000 7f1aa1d9b640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 0x7f1aa407f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.353+0000 7f1aa1d9b640 1 -- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa4108f50 msgr2=0x7f1aa4084be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.353+0000 7f1aa1d9b640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa4108f50 0x7f1aa4084be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:07.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.353+0000 7f1aa1d9b640 1 -- 172.21.15.87:0/3456664474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa4131300 con 0x7f1aa4109910 2024-04-22T13:34:07.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.359+0000 7f1aa1d9b640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407eca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a980074b0 tx=0x7f1a98007dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:07.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.364+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9803f030 con 0x7f1aa4109910 2024-04-22T13:34:07.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.364+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa4131590 con 0x7f1aa4109910 2024-04-22T13:34:07.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.364+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa4131ae0 con 0x7f1aa4109910 2024-04-22T13:34:07.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.364+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a98004250 con 0x7f1aa4109910 2024-04-22T13:34:07.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.367+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800a170 con 0x7f1aa4109910 2024-04-22T13:34:07.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.373+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a84005290 con 0x7f1aa4109910 2024-04-22T13:34:07.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.378+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a98042070 con 0x7f1aa4109910 2024-04-22T13:34:07.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.380+0000 7f1a937fe640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 0x7f1a8c0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:07.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.381+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1a980f1490 con 0x7f1aa4109910 2024-04-22T13:34:07.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.385+0000 7f1aa259c640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 0x7f1a8c0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:07.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.389+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a980bdb40 con 0x7f1aa4109910 2024-04-22T13:34:07.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:07.409+0000 7f1aa259c640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 0x7f1a8c0a63f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009990 tx=0x7f1a94009110 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:07.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:07 smithi087 ceph-mon[27999]: pgmap v70: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 502 op/s 2024-04-22T13:34:07.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:07 smithi120 ceph-mon[31101]: pgmap v70: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 502 op/s 2024-04-22T13:34:07.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:07 smithi132 ceph-mon[31531]: pgmap v70: 97 pgs: 97 active+clean; 7.7 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 655 MiB/s wr, 502 op/s 2024-04-22T13:34:08.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.036+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a84005850 con 0x7f1aa4109910 2024-04-22T13:34:08.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.040+0000 7f1a937fe640 1 -- 172.21.15.87:0/3456664474 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f1a980bd440 con 0x7f1aa4109910 2024-04-22T13:34:08.036 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:34:08.037 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:34:08.037 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:34:08.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.050+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 msgr2=0x7f1a8c0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.050+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 0x7f1a8c0a63f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009990 tx=0x7f1a94009110 comp rx=0 tx=0).stop 2024-04-22T13:34:08.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.050+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 msgr2=0x7f1aa407eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.050+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407eca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a980074b0 tx=0x7f1a98007dc0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 shutdown_connections 2024-04-22T13:34:08.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a8c0a3f30 0x7f1a8c0a63f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1aa410e600 0x7f1aa407f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1aa4109910 0x7f1aa407eca0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 --2- 172.21.15.87:0/3456664474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1aa4108f50 0x7f1aa4084be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 >> 172.21.15.87:0/3456664474 conn(0x7f1aa406e410 msgr2=0x7f1aa4071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 shutdown_connections 2024-04-22T13:34:08.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.052+0000 7f1aa8ea0640 1 -- 172.21.15.87:0/3456664474 wait complete. 2024-04-22T13:34:08.094 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:34:08.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.351+0000 7f8990f4f640 1 -- 172.21.15.87:0/475003517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c108f50 msgr2=0x7f898c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.351+0000 7f8990f4f640 1 --2- 172.21.15.87:0/475003517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c108f50 0x7f898c1093b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8984009ca0 tx=0x7f898402c1a0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 -- 172.21.15.87:0/475003517 shutdown_connections 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 --2- 172.21.15.87:0/475003517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 --2- 172.21.15.87:0/475003517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c108f50 0x7f898c1093b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 --2- 172.21.15.87:0/475003517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c1105f0 0x7f898c1109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 -- 172.21.15.87:0/475003517 >> 172.21.15.87:0/475003517 conn(0x7f898c06ced0 msgr2=0x7f898c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 -- 172.21.15.87:0/475003517 shutdown_connections 2024-04-22T13:34:08.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.352+0000 7f8990f4f640 1 -- 172.21.15.87:0/475003517 wait complete. 2024-04-22T13:34:08.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.353+0000 7f8990f4f640 1 Processor -- start 2024-04-22T13:34:08.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 -- start start 2024-04-22T13:34:08.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c1105f0 0x7f898c11d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c116130 0x7f898c1165e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f898c120f00 con 0x7f898c1099d0 2024-04-22T13:34:08.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f898c120d80 con 0x7f898c116130 2024-04-22T13:34:08.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.355+0000 7f8990f4f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f898c121080 con 0x7f898c1105f0 2024-04-22T13:34:08.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.357+0000 7f898b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.357+0000 7f898b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57390/0 (socket says 172.21.15.87:57390) 2024-04-22T13:34:08.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.357+0000 7f898b7fe640 1 -- 172.21.15.87:0/398319529 learned_addr learned my addr 172.21.15.87:0/398319529 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.357+0000 7f898bfff640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c116130 0x7f898c1165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.358+0000 7f898affd640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c1105f0 0x7f898c11d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.360+0000 7f898b7fe640 1 -- 172.21.15.87:0/398319529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c1105f0 msgr2=0x7f898c11d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.360+0000 7f898b7fe640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c1105f0 0x7f898c11d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.360+0000 7f898b7fe640 1 -- 172.21.15.87:0/398319529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c116130 msgr2=0x7f898c1165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.360+0000 7f898b7fe640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c116130 0x7f898c1165e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.360+0000 7f898b7fe640 1 -- 172.21.15.87:0/398319529 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c116db0 con 0x7f898c1099d0 2024-04-22T13:34:08.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.362+0000 7f898b7fe640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f897c00e9f0 tx=0x7f897c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.364+0000 7f8988ff9640 1 -- 172.21.15.87:0/398319529 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c00cd80 con 0x7f898c1099d0 2024-04-22T13:34:08.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.364+0000 7f8988ff9640 1 -- 172.21.15.87:0/398319529 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897c0108e0 con 0x7f898c1099d0 2024-04-22T13:34:08.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.364+0000 7f8988ff9640 1 -- 172.21.15.87:0/398319529 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c018980 con 0x7f898c1099d0 2024-04-22T13:34:08.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.364+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c11abd0 con 0x7f898c1099d0 2024-04-22T13:34:08.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.364+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c11b0a0 con 0x7f898c1099d0 2024-04-22T13:34:08.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.370+0000 7f8994d9c740 1 Processor -- start 2024-04-22T13:34:08.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.370+0000 7f8994d9c740 1 -- start start 2024-04-22T13:34:08.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.371+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6f36d0 0x55ddab6f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.371+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.371+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6fcc00 0x55ddab6ff050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.371+0000 7f8994d9c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ddab6581f0 con 0x55ddab6fcc00 2024-04-22T13:34:08.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.372+0000 7f8994d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddab5f54b0 con 0x55ddab6f4080 2024-04-22T13:34:08.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.372+0000 7f8994d9c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ddab6541b0 con 0x55ddab6f36d0 2024-04-22T13:34:08.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.372+0000 7f898b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.372+0000 7f898b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:41912/0 (socket says 172.21.15.87:41912) 2024-04-22T13:34:08.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.372+0000 7f898b7fe640 1 -- 172.21.15.87:0/3921504400 learned_addr learned my addr 172.21.15.87:0/3921504400 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.378+0000 7f898b7fe640 1 -- 172.21.15.87:0/3921504400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6f36d0 msgr2=0x55ddab6f3ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.378+0000 7f898b7fe640 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6f36d0 0x55ddab6f3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.378+0000 7f898b7fe640 1 -- 172.21.15.87:0/3921504400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6fcc00 msgr2=0x55ddab6ff050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.378+0000 7f898b7fe640 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6fcc00 0x55ddab6ff050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.378+0000 7f898b7fe640 1 -- 172.21.15.87:0/3921504400 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddab5e71b0 con 0x55ddab6f4080 2024-04-22T13:34:08.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.379+0000 7f898b7fe640 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f897c0d4ee0 tx=0x7f897c0d5f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9129a2c044d00c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.379+0000 7f8988ff9640 1 -- 172.21.15.87:0/398319529 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f897c010430 con 0x7f898c1099d0 2024-04-22T13:34:08.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.381+0000 7f8988ff9640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 0x7f89740a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.381+0000 7f898bfff640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 0x7f89740a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.382+0000 7f898bfff640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 0x7f89740a6370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8980009900 tx=0x7f89800090e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.383+0000 7f8988ff9640 1 -- 172.21.15.87:0/398319529 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f897c09fd10 con 0x7f898c1099d0 2024-04-22T13:34:08.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.383+0000 7f8971ffb640 1 -- 172.21.15.87:0/3921504400 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c0dc070 con 0x55ddab6f4080 2024-04-22T13:34:08.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.383+0000 7f8971ffb640 1 -- 172.21.15.87:0/3921504400 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897c0a3e30 con 0x55ddab6f4080 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.384+0000 7f8994d9c740 1 -- 172.21.15.87:0/3921504400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 msgr2=0x55ddab6fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.384+0000 7f8994d9c740 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f897c0d4ee0 tx=0x7f897c0d5f10 comp rx=0 tx=0).stop 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 -- 172.21.15.87:0/3921504400 shutdown_connections 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6fcc00 0x55ddab6ff050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f4080 0x55ddab6fc630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 --2- 172.21.15.87:0/3921504400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6f36d0 0x55ddab6f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 -- 172.21.15.87:0/3921504400 >> 172.21.15.87:0/3921504400 conn(0x55ddab6edda0 msgr2=0x55ddab6ee180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 -- 172.21.15.87:0/3921504400 shutdown_connections 2024-04-22T13:34:08.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 -- 172.21.15.87:0/3921504400 wait complete. 2024-04-22T13:34:08.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.385+0000 7f8994d9c740 1 -- start start 2024-04-22T13:34:08.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.386+0000 7f8994d9c740 1 Processor -- start 2024-04-22T13:34:08.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab6fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ddab6ffb70 con 0x55ddab6f4080 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddab611500 con 0x55ddab6f36d0 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ddab6ffcf0 con 0x55ddab6fcc00 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.387+0000 7f8994d9c740 10 client.0 _ll_register_callbacks cb 0x55ddab5dfb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.388+0000 7f898affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.388+0000 7f898affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57414/0 (socket says 172.21.15.87:57414) 2024-04-22T13:34:08.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.388+0000 7f898affd640 1 -- 172.21.15.87:0/2828580916 learned_addr learned my addr 172.21.15.87:0/2828580916 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.388+0000 7f898bfff640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.389+0000 7f898affd640 1 -- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 msgr2=0x55ddab6f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.389+0000 7f898affd640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6f9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.389+0000 7f898affd640 1 -- 172.21.15.87:0/2828580916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 msgr2=0x55ddab6fc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.389+0000 7f898affd640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab6fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.389+0000 7f898affd640 1 -- 172.21.15.87:0/2828580916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddab36a3f0 con 0x55ddab6f4080 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.390+0000 7f898affd640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8984009ca0 tx=0x7f8984003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da2255189fa2cf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.390+0000 7f896affd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:08.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.390+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984043070 con 0x55ddab6f4080 2024-04-22T13:34:08.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.391+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8984004040 con 0x55ddab6f4080 2024-04-22T13:34:08.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.391+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f897c0a3e30 con 0x55ddab6f4080 2024-04-22T13:34:08.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.391+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f898404d050 con 0x55ddab6f4080 2024-04-22T13:34:08.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.392+0000 7f8994d9c740 10 client.16395 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:08.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.392+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ddab4347d0 con 0x55ddab6f4080 2024-04-22T13:34:08.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897c0a3e30 con 0x55ddab6f4080 2024-04-22T13:34:08.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8984032bd0 con 0x55ddab6f4080 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 10 client.16395 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 10 client.16395 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 20 client.16395 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab70cd50 0x55ddab70f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 4 client.16395 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:08.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.393+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddab5f7930 con 0x55ddab70cd50 2024-04-22T13:34:08.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.395+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab70cd50 0x55ddab70f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.424+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab70cd50 0x55ddab70f1b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ebb1390fd1938dd server_cookie=620ab847d3833b in_seq=0 out_seq=0 2024-04-22T13:34:08.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.425+0000 7f896affd640 10 client.16395 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:08.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.428+0000 7f896affd640 1 -- 172.21.15.87:0/2828580916 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddab5f7930 con 0x55ddab70cd50 2024-04-22T13:34:08.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.428+0000 7f896affd640 10 client.16395 handle_command_reply: tid=0 2024-04-22T13:34:08.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.428+0000 7f8994d9c740 1 client.16395 shutdown 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 msgr2=0x55ddab6f9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8984009ca0 tx=0x7f8984003e90 comp rx=0 tx=0).stop 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 shutdown_connections 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab70cd50 0x55ddab70f1b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6f9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f9730 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2828580916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab6fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 >> 172.21.15.87:0/2828580916 conn(0x55ddab6edda0 msgr2=0x55ddab6efaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f896affd640 0 client.16395 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:08.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.430+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 shutdown_connections 2024-04-22T13:34:08.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.433+0000 7f8994d9c740 1 -- 172.21.15.87:0/2828580916 wait complete. 2024-04-22T13:34:08.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.436+0000 7f8994d9c740 20 client.16395 trim_cache size 0 max 16384 2024-04-22T13:34:08.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.468+0000 7f8994d9c740 1 Processor -- start 2024-04-22T13:34:08.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.470+0000 7f8994d9c740 1 -- start start 2024-04-22T13:34:08.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.471+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab708ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.471+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f898affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f898affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57416/0 (socket says 172.21.15.87:57416) 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f898affd640 1 -- 172.21.15.87:0/2838196760 learned_addr learned my addr 172.21.15.87:0/2838196760 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c79a0 con 0x55ddab6f4080 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c5270 con 0x55ddab6f36d0 2024-04-22T13:34:08.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.472+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c30c0 con 0x55ddab6fcc00 2024-04-22T13:34:08.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.473+0000 7f898affd640 1 -- 172.21.15.87:0/2838196760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 msgr2=0x55ddab6fbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.473+0000 7f898affd640 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.473+0000 7f898affd640 1 -- 172.21.15.87:0/2838196760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 msgr2=0x55ddab708ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:08.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.473+0000 7f898affd640 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab708ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.473+0000 7f898affd640 1 -- 172.21.15.87:0/2838196760 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddab6f5310 con 0x55ddab6f4080 2024-04-22T13:34:08.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.474+0000 7f898affd640 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8984000f80 tx=0x7f8984003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fce2fa8d423254 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.477+0000 7f89717fa640 1 -- 172.21.15.87:0/2838196760 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984057070 con 0x55ddab6f4080 2024-04-22T13:34:08.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.477+0000 7f89717fa640 1 -- 172.21.15.87:0/2838196760 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8984009780 con 0x55ddab6f4080 2024-04-22T13:34:08.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.479+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 msgr2=0x55ddab6f4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.479+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8984000f80 tx=0x7f8984003de0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.483+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 shutdown_connections 2024-04-22T13:34:08.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.483+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab6fcc00 0x55ddab6fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.483+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f4080 0x55ddab6f4460 secure :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8984000f80 tx=0x7f8984003de0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.483+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2838196760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6f36d0 0x55ddab708ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.483+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 >> 172.21.15.87:0/2838196760 conn(0x55ddab6edda0 msgr2=0x55ddab6ece20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.487+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 shutdown_connections 2024-04-22T13:34:08.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.489+0000 7f8994d9c740 1 -- 172.21.15.87:0/2838196760 wait complete. 2024-04-22T13:34:08.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.489+0000 7f8994d9c740 1 -- start start 2024-04-22T13:34:08.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.490+0000 7f8994d9c740 1 Processor -- start 2024-04-22T13:34:08.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 0x55ddab700a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6fcc00 0x55ddab700f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c3730 con 0x55ddab6f36d0 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c4ba0 con 0x55ddab6fcc00 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1c3e00 con 0x55ddab701480 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.491+0000 7f8994d9c740 10 client.0 _ll_register_callbacks cb 0x7f8984003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39500/0 (socket says 172.21.15.87:39500) 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 -- 172.21.15.87:0/2583453406 learned_addr learned my addr 172.21.15.87:0/2583453406 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898affd640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6fcc00 0x55ddab700f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 -- 172.21.15.87:0/2583453406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6fcc00 msgr2=0x55ddab700f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 0x55ddab700a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6fcc00 0x55ddab700f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.492+0000 7f898bfff640 1 -- 172.21.15.87:0/2583453406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 msgr2=0x55ddab700a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.493+0000 7f898bfff640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 0x55ddab700a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.493+0000 7f898bfff640 1 -- 172.21.15.87:0/2583453406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddab772830 con 0x55ddab701480 2024-04-22T13:34:08.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.493+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 0x55ddab700a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:08.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.494+0000 7f898bfff640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f898000eee0 tx=0x7f8980015850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df9ddc3ceb58567f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.494+0000 7f896affd640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:08.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.494+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980025070 con 0x55ddab701480 2024-04-22T13:34:08.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.495+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8984009780 con 0x55ddab701480 2024-04-22T13:34:08.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.495+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8980013660 con 0x55ddab701480 2024-04-22T13:34:08.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.501+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8980031050 con 0x55ddab701480 2024-04-22T13:34:08.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.504+0000 7f8994d9c740 10 client.25672 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:08.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.504+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ddab773ad0 con 0x55ddab701480 2024-04-22T13:34:08.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.507+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8984009780 con 0x55ddab701480 2024-04-22T13:34:08.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8980021a60 con 0x55ddab701480 2024-04-22T13:34:08.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 10 client.25672 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 10 client.25672 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 20 client.25672 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab774200 0x55ddab776620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 4 client.25672 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.509+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ddab665c10 con 0x55ddab774200 2024-04-22T13:34:08.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.510+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab774200 0x55ddab776620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.511+0000 7f896affd640 10 client.25672 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:08.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.511+0000 7f898b7fe640 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab774200 0x55ddab776620 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3766b86f944f5e4 server_cookie=4c7ca11be2793b26 in_seq=0 out_seq=0 2024-04-22T13:34:08.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.511+0000 7f896affd640 1 -- 172.21.15.87:0/2583453406 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ddab665c10 con 0x55ddab774200 2024-04-22T13:34:08.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.512+0000 7f896affd640 10 client.25672 handle_command_reply: tid=0 2024-04-22T13:34:08.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.512+0000 7f8994d9c740 1 client.25672 shutdown 2024-04-22T13:34:08.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.513+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 msgr2=0x55ddab7722f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.513+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f898000eee0 tx=0x7f8980015850 comp rx=0 tx=0).stop 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 shutdown_connections 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ddab774200 0x55ddab776620 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ddab701480 0x55ddab7722f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddab6fcc00 0x55ddab700f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 --2- 172.21.15.87:0/2583453406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ddab6f36d0 0x55ddab700a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f896affd640 0 client.25672 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:08.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.515+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 >> 172.21.15.87:0/2583453406 conn(0x55ddab6edda0 msgr2=0x55ddab6f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.517+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 shutdown_connections 2024-04-22T13:34:08.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.520+0000 7f8994d9c740 1 -- 172.21.15.87:0/2583453406 wait complete. 2024-04-22T13:34:08.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.520+0000 7f8994d9c740 20 client.25672 trim_cache size 0 max 16384 2024-04-22T13:34:08.516 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.521+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 msgr2=0x7f89740a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.521+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 0x7f89740a6370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8980009900 tx=0x7f89800090e0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.521+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 msgr2=0x7f898c11d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.521+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f897c00e9f0 tx=0x7f897c00eeb0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 shutdown_connections 2024-04-22T13:34:08.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89740a3eb0 0x7f89740a6370 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f898c116130 0x7f898c1165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f898c1105f0 0x7f898c11d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 --2- 172.21.15.87:0/398319529 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f898c1099d0 0x7f898c11d0b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 >> 172.21.15.87:0/398319529 conn(0x7f898c06ced0 msgr2=0x7f898c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.524+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 shutdown_connections 2024-04-22T13:34:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.525+0000 7f8990f4f640 1 -- 172.21.15.87:0/398319529 wait complete. 2024-04-22T13:34:08.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.916+0000 7f44293a0640 1 -- 172.21.15.87:0/399084289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418032920 con 0x7f442410e5f0 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 -- 172.21.15.87:0/399084289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 msgr2=0x7f442410e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 --2- 172.21.15.87:0/399084289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f442410e9d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4418009ca0 tx=0x7f441802c120 comp rx=0 tx=0).stop 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 -- 172.21.15.87:0/399084289 shutdown_connections 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 --2- 172.21.15.87:0/399084289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 0x7f44241153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 --2- 172.21.15.87:0/399084289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4424108580 0x7f4424108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 --2- 172.21.15.87:0/399084289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f442410e9d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 -- 172.21.15.87:0/399084289 >> 172.21.15.87:0/399084289 conn(0x7f4424076490 msgr2=0x7f44240788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 -- 172.21.15.87:0/399084289 shutdown_connections 2024-04-22T13:34:08.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.917+0000 7f442c606640 1 -- 172.21.15.87:0/399084289 wait complete. 2024-04-22T13:34:08.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.918+0000 7f442c606640 1 Processor -- start 2024-04-22T13:34:08.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.918+0000 7f442c606640 1 -- start start 2024-04-22T13:34:08.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4424108580 0x7f442419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 0x7f442419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4424118880 con 0x7f442410e5f0 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4424118700 con 0x7f4424108f40 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442aba3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442aba3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57434/0 (socket says 172.21.15.87:57434) 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442aba3640 1 -- 172.21.15.87:0/4187489104 learned_addr learned my addr 172.21.15.87:0/4187489104 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4424118a00 con 0x7f4424108580 2024-04-22T13:34:08.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.919+0000 7f4429ba1640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 0x7f442419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 -- 172.21.15.87:0/4187489104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4424108580 msgr2=0x7f442419ee80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4424108580 0x7f442419ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 -- 172.21.15.87:0/4187489104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 msgr2=0x7f442419f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 0x7f442419f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 -- 172.21.15.87:0/4187489104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44241a3e70 con 0x7f442410e5f0 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f442aba3640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f441c009fd0 tx=0x7f441c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.920+0000 7f44237fe640 1 -- 172.21.15.87:0/4187489104 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c011830 con 0x7f442410e5f0 2024-04-22T13:34:08.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.921+0000 7f44237fe640 1 -- 172.21.15.87:0/4187489104 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f441c016070 con 0x7f442410e5f0 2024-04-22T13:34:08.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.921+0000 7f44237fe640 1 -- 172.21.15.87:0/4187489104 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c010de0 con 0x7f442410e5f0 2024-04-22T13:34:08.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.921+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44241a4160 con 0x7f442410e5f0 2024-04-22T13:34:08.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.921+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44241ab9f0 con 0x7f442410e5f0 2024-04-22T13:34:08.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:08 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3456664474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:08.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.925+0000 7f442f1f4740 1 Processor -- start 2024-04-22T13:34:08.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.926+0000 7f44237fe640 1 -- 172.21.15.87:0/4187489104 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f441c0119d0 con 0x7f442410e5f0 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 -- start start 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d08358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d0840e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f6d079a1f0 con 0x55f6d0835e80 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6d07374b0 con 0x55f6d08354d0 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442f1f4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f6d07961b0 con 0x55f6d083ea00 2024-04-22T13:34:08.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.927+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f44237fe640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 0x7f44140a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57440/0 (socket says 172.21.15.87:57440) 2024-04-22T13:34:08.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 -- 172.21.15.87:0/3781272662 learned_addr learned my addr 172.21.15.87:0/3781272662 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f44237fe640 1 -- 172.21.15.87:0/4187489104 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f441c0cfdc0 con 0x7f442410e5f0 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442aba3640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 0x7f44140a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f4429ba1640 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d08358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 -- 172.21.15.87:0/3781272662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 msgr2=0x55f6d0840e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d0840e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 -- 172.21.15.87:0/3781272662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 msgr2=0x55f6d08358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d08358b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.928+0000 7f442a3a2640 1 -- 172.21.15.87:0/3781272662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d07291b0 con 0x55f6d0835e80 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.929+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4418037040 tx=0x7f44180097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a56d8c20e091cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.929+0000 7f442aba3640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 0x7f44140a62e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f441c0cd690 tx=0x7f441c00bf90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.929+0000 7f4420bfe640 1 -- 172.21.15.87:0/3781272662 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418041020 con 0x55f6d0835e80 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.929+0000 7f4420bfe640 1 -- 172.21.15.87:0/3781272662 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44180329d0 con 0x55f6d0835e80 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.930+0000 7f4420bfe640 1 -- 172.21.15.87:0/3781272662 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418039040 con 0x55f6d0835e80 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.930+0000 7f442f1f4740 1 -- 172.21.15.87:0/3781272662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 msgr2=0x55f6d083e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.930+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4418037040 tx=0x7f44180097e0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 -- 172.21.15.87:0/3781272662 shutdown_connections 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d0840e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083e430 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3781272662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d08358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 -- 172.21.15.87:0/3781272662 >> 172.21.15.87:0/3781272662 conn(0x55f6d082fba0 msgr2=0x55f6d082ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 -- 172.21.15.87:0/3781272662 shutdown_connections 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 -- 172.21.15.87:0/3781272662 wait complete. 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 -- start start 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.931+0000 7f442f1f4740 1 Processor -- start 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d083bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f6d0841960 con 0x55f6d0835e80 2024-04-22T13:34:08.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6d0753500 con 0x55f6d08354d0 2024-04-22T13:34:08.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f6d0841ae0 con 0x55f6d083ea00 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:41980/0 (socket says 172.21.15.87:41980) 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442f1f4740 10 client.0 _ll_register_callbacks cb 0x55f6d0721b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f4429ba1640 1 -- 172.21.15.87:0/1693384756 learned_addr learned my addr 172.21.15.87:0/1693384756 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.933+0000 7f442a3a2640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d083bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 -- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 msgr2=0x55f6d083bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d083bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 -- 172.21.15.87:0/1693384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 msgr2=0x55f6d083b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.934+0000 7f4429ba1640 1 -- 172.21.15.87:0/1693384756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d04ac3f0 con 0x55f6d08354d0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.935+0000 7f4429ba1640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f441000bef0 tx=0x7f441000b230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c75d8d69806b39fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.935+0000 7f440dffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.935+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441001c060 con 0x55f6d08354d0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.935+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4410004230 con 0x55f6d08354d0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.935+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44180329d0 con 0x55f6d08354d0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.936+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4410004430 con 0x55f6d08354d0 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.936+0000 7f442f1f4740 10 client.25667 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:08.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.936+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f6d05767d0 con 0x55f6d08354d0 2024-04-22T13:34:08.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.936+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44180329d0 con 0x55f6d08354d0 2024-04-22T13:34:08.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.936+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f441001f020 con 0x55f6d08354d0 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 10 client.25667 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 10 client.25667 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 20 client.25667 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d0851060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 4 client.25667 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6d0739930 con 0x55f6d084ec70 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.937+0000 7f442a3a2640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d0851060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.939+0000 7f440dffb640 10 client.25667 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:08.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.939+0000 7f442a3a2640 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d0851060 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=982e8147499b5cd0 server_cookie=e6c473623bd83f2c in_seq=0 out_seq=0 2024-04-22T13:34:08.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f440dffb640 1 -- 172.21.15.87:0/1693384756 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6d0739930 con 0x55f6d084ec70 2024-04-22T13:34:08.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f440dffb640 10 client.25667 handle_command_reply: tid=0 2024-04-22T13:34:08.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f442f1f4740 1 client.25667 shutdown 2024-04-22T13:34:08.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 msgr2=0x55f6d083e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f441000bef0 tx=0x7f441000b230 comp rx=0 tx=0).stop 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 shutdown_connections 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.940+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d0851060 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f440dffb640 0 client.25667 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d083ea00 0x55f6d083bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d083b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 --2- 172.21.15.87:0/1693384756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d08354d0 0x55f6d083e2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 >> 172.21.15.87:0/1693384756 conn(0x55f6d082fba0 msgr2=0x55f6d08318f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 shutdown_connections 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 1 -- 172.21.15.87:0/1693384756 wait complete. 2024-04-22T13:34:08.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.941+0000 7f442f1f4740 20 client.25667 trim_cache size 0 max 16384 2024-04-22T13:34:08.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.966+0000 7f442f1f4740 1 Processor -- start 2024-04-22T13:34:08.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.966+0000 7f442f1f4740 1 -- start start 2024-04-22T13:34:08.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:08 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3456664474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.967+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 0x55f6d084a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.967+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.967+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d083ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.967+0000 7f442f1f4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b6b00 con 0x55f6d0835e80 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 0x55f6d084a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57464/0 (socket says 172.21.15.87:57464) 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 0x55f6d084a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39536/0 (socket says 172.21.15.87:39536) 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f442a3a2640 1 -- 172.21.15.87:0/2874389014 learned_addr learned my addr 172.21.15.87:0/2874389014 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b43d0 con 0x55f6d083ea00 2024-04-22T13:34:08.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b2240 con 0x55f6d08354d0 2024-04-22T13:34:08.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.968+0000 7f4429ba1640 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d083ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 -- 172.21.15.87:0/2874389014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 msgr2=0x55f6d084a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 0x55f6d084a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 -- 172.21.15.87:0/2874389014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 msgr2=0x55f6d083ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d083ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 -- 172.21.15.87:0/2874389014 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d08411c0 con 0x55f6d0835e80 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.969+0000 7f442a3a2640 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4418000f80 tx=0x7f441802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b83332a971c294e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.970+0000 7f440ffff640 1 -- 172.21.15.87:0/2874389014 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441804e070 con 0x55f6d0835e80 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.970+0000 7f440ffff640 1 -- 172.21.15.87:0/2874389014 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44180329d0 con 0x55f6d0835e80 2024-04-22T13:34:08.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.970+0000 7f440ffff640 1 -- 172.21.15.87:0/2874389014 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418039030 con 0x55f6d0835e80 2024-04-22T13:34:08.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.971+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 msgr2=0x55f6d082ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:08.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.971+0000 7f442f1f4740 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4418000f80 tx=0x7f441802e040 comp rx=0 tx=0).stop 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 shutdown_connections 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d083ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d0835e80 0x55f6d082ee40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 --2- 172.21.15.87:0/2874389014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d08354d0 0x55f6d084a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 >> 172.21.15.87:0/2874389014 conn(0x55f6d082fba0 msgr2=0x55f6d0836b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 shutdown_connections 2024-04-22T13:34:08.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.972+0000 7f442f1f4740 1 -- 172.21.15.87:0/2874389014 wait complete. 2024-04-22T13:34:08.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.973+0000 7f442f1f4740 1 -- start start 2024-04-22T13:34:08.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.973+0000 7f442f1f4740 1 Processor -- start 2024-04-22T13:34:08.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.974+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 0x55f6d0846d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.974+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.975+0000 7f442f1f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d0837e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.975+0000 7f442f1f4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b2890 con 0x55f6d08354d0 2024-04-22T13:34:08.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.975+0000 7f442f1f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b3d00 con 0x55f6d083ea00 2024-04-22T13:34:08.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.975+0000 7f442f1f4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f44241b2f60 con 0x55f6d0835e80 2024-04-22T13:34:08.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d0837e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f4429ba1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d0837e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:42004/0 (socket says 172.21.15.87:42004) 2024-04-22T13:34:08.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f442a3a2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39540/0 (socket says 172.21.15.87:39540) 2024-04-22T13:34:08.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f4429ba1640 1 -- 172.21.15.87:0/3754008917 learned_addr learned my addr 172.21.15.87:0/3754008917 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.976+0000 7f4429ba1640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 0x55f6d0846d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 -- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 msgr2=0x55f6d08472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 -- 172.21.15.87:0/3754008917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 msgr2=0x55f6d0837e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d0837e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 -- 172.21.15.87:0/3754008917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d0838670 con 0x55f6d08354d0 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f4429ba1640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 0x55f6d0846d60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f441000bef0 tx=0x7f441000bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aec1119d4fdff41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:08.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f442f1f4740 10 client.0 _ll_register_callbacks cb 0x7f44100144c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44180329d0 con 0x55f6d08354d0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f440dffb640 10 client.16416 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.977+0000 7f440dffb640 1 -- 172.21.15.87:0/3754008917 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410036060 con 0x55f6d08354d0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f440dffb640 1 -- 172.21.15.87:0/3754008917 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f441000f750 con 0x55f6d08354d0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f440dffb640 1 -- 172.21.15.87:0/3754008917 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f441000f950 con 0x55f6d08354d0 2024-04-22T13:34:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 10 client.16416 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55f6d08b4bb0 con 0x55f6d08354d0 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f440dffb640 1 -- 172.21.15.87:0/3754008917 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f441000faf0 con 0x55f6d08354d0 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 10 client.16416 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 10 client.16416 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 20 client.16416 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d08b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 4 client.16416 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.978+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f44180329d0 con 0x55f6d084ec70 2024-04-22T13:34:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:08.981+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d08b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:08.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:08 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3456664474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:09.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.192+0000 7f442a3a2640 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d08b71f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b179faf538b0ad0e server_cookie=576a57c3afa80aec in_seq=0 out_seq=0 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.192+0000 7f440dffb640 10 client.16416 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.193+0000 7f440dffb640 1 -- 172.21.15.87:0/3754008917 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f44180329d0 con 0x55f6d084ec70 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.193+0000 7f440dffb640 10 client.16416 handle_command_reply: tid=0 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.193+0000 7f442f1f4740 1 client.16416 shutdown 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.193+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 msgr2=0x55f6d0846d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.193+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 0x55f6d0846d60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f441000bef0 tx=0x7f441000bd00 comp rx=0 tx=0).stop 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 shutdown_connections 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55f6d084ec70 0x55f6d08b71f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6d083ea00 0x55f6d0837e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f6d0835e80 0x55f6d08472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 --2- 172.21.15.87:0/3754008917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f6d08354d0 0x55f6d0846d60 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 >> 172.21.15.87:0/3754008917 conn(0x55f6d082fba0 msgr2=0x55f6d083aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f440dffb640 0 client.16416 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 shutdown_connections 2024-04-22T13:34:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.194+0000 7f442f1f4740 1 -- 172.21.15.87:0/3754008917 wait complete. 2024-04-22T13:34:09.192 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:09.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.195+0000 7f442f1f4740 20 client.16416 trim_cache size 0 max 16384 2024-04-22T13:34:09.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.199+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 msgr2=0x7f44140a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.199+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 0x7f44140a62e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f441c0cd690 tx=0x7f441c00bf90 comp rx=0 tx=0).stop 2024-04-22T13:34:09.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.199+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 msgr2=0x7f44241a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.199+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f441c009fd0 tx=0x7f441c00b8e0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 shutdown_connections 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f44140a3e20 0x7f44140a62e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f442410e5f0 0x7f44241a3740 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4424108f40 0x7f442419f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 --2- 172.21.15.87:0/4187489104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4424108580 0x7f442419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.200+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 >> 172.21.15.87:0/4187489104 conn(0x7f4424076490 msgr2=0x7f4424077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.202+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 shutdown_connections 2024-04-22T13:34:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.203+0000 7f442c606640 1 -- 172.21.15.87:0/4187489104 wait complete. 2024-04-22T13:34:09.252 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:34:09.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:09 smithi087 ceph-mon[27999]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 868 MiB/s wr, 1.14k op/s 2024-04-22T13:34:09.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.759+0000 7f3463458640 1 -- 172.21.15.87:0/754538471 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c1105f0 msgr2=0x7f345c1109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.759+0000 7f3463458640 1 --2- 172.21.15.87:0/754538471 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c1105f0 0x7f345c1109d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3458009900 tx=0x7f345802bd10 comp rx=0 tx=0).stop 2024-04-22T13:34:09.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.761+0000 7f3463458640 1 -- 172.21.15.87:0/754538471 shutdown_connections 2024-04-22T13:34:09.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.761+0000 7f3463458640 1 --2- 172.21.15.87:0/754538471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 0x7f345c11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.761+0000 7f3463458640 1 --2- 172.21.15.87:0/754538471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c108f50 0x7f345c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.761+0000 7f3463458640 1 --2- 172.21.15.87:0/754538471 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c1105f0 0x7f345c1109d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.761+0000 7f3463458640 1 -- 172.21.15.87:0/754538471 >> 172.21.15.87:0/754538471 conn(0x7f345c06ced0 msgr2=0x7f345c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:09.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.786+0000 7f3463458640 1 -- 172.21.15.87:0/754538471 shutdown_connections 2024-04-22T13:34:09.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.795+0000 7f3463458640 1 -- 172.21.15.87:0/754538471 wait complete. 2024-04-22T13:34:09.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.799+0000 7f3463458640 1 Processor -- start 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 -- start start 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c108f50 0x7f345c1a7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 0x7f345c1a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f345c120f20 con 0x7f345c108f50 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f345c120da0 con 0x7f345c1099d0 2024-04-22T13:34:09.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.800+0000 7f3463458640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f345c1210a0 con 0x7f345c1105f0 2024-04-22T13:34:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.803+0000 7f3462c57640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.803+0000 7f3462c57640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:55966/0 (socket says 172.21.15.87:55966) 2024-04-22T13:34:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.803+0000 7f3462c57640 1 -- 172.21.15.87:0/2050569438 learned_addr learned my addr 172.21.15.87:0/2050569438 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.803+0000 7f3461c55640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 0x7f345c1a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.807+0000 7f3462c57640 1 -- 172.21.15.87:0/2050569438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 msgr2=0x7f345c1a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.807+0000 7f3462c57640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 0x7f345c1a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.807+0000 7f3462c57640 1 -- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c108f50 msgr2=0x7f345c1a7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.807+0000 7f3462c57640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c108f50 0x7f345c1a7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.807+0000 7f3462c57640 1 -- 172.21.15.87:0/2050569438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345c1ac3c0 con 0x7f345c1105f0 2024-04-22T13:34:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.814+0000 7f3462c57640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f345400b9c0 tx=0x7f345400be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.815+0000 7f34537fe640 1 -- 172.21.15.87:0/2050569438 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3454019070 con 0x7f345c1105f0 2024-04-22T13:34:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.815+0000 7f34537fe640 1 -- 172.21.15.87:0/2050569438 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345400cba0 con 0x7f345c1105f0 2024-04-22T13:34:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.815+0000 7f34537fe640 1 -- 172.21.15.87:0/2050569438 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540075d0 con 0x7f345c1105f0 2024-04-22T13:34:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.815+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345c1ac6b0 con 0x7f345c1105f0 2024-04-22T13:34:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.815+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f345c110f10 con 0x7f345c1105f0 2024-04-22T13:34:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.820+0000 7f3467385740 1 Processor -- start 2024-04-22T13:34:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.833+0000 7f3467385740 1 -- start start 2024-04-22T13:34:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc1a080 0x55804fc22630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f34537fe640 1 -- 172.21.15.87:0/2050569438 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f345401c070 con 0x7f345c1105f0 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc25050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55804fb7e1f0 con 0x55804fc196d0 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55804fb1b4b0 con 0x55804fc1a080 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.837+0000 7f3467385740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55804fb7a1b0 con 0x55804fc22c00 2024-04-22T13:34:09.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.838+0000 7f3461c55640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.838+0000 7f3461c55640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43790/0 (socket says 172.21.15.87:43790) 2024-04-22T13:34:09.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.838+0000 7f3461c55640 1 -- 172.21.15.87:0/4252591219 learned_addr learned my addr 172.21.15.87:0/4252591219 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:09.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.840+0000 7f34537fe640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 0x7f34480a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.842+0000 7f34537fe640 1 -- 172.21.15.87:0/2050569438 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34540d0b70 con 0x7f345c1105f0 2024-04-22T13:34:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.842+0000 7f3462c57640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 0x7f34480a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.843+0000 7f3461c55640 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc25050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.846+0000 7f3462c57640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 0x7f34480a63f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34540996b0 tx=0x7f3454011000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3462456640 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc1a080 0x55804fc22630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3461c55640 1 -- 172.21.15.87:0/4252591219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 msgr2=0x55804fc25050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3461c55640 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc25050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3461c55640 1 -- 172.21.15.87:0/4252591219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc1a080 msgr2=0x55804fc22630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3461c55640 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc1a080 0x55804fc22630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.847+0000 7f3461c55640 1 -- 172.21.15.87:0/4252591219 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55804fb0d1b0 con 0x55804fc196d0 2024-04-22T13:34:09.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.863+0000 7f3461c55640 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f344c013500 tx=0x7f344c00a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0081995a77613f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:09.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.871+0000 7f3450ff9640 1 -- 172.21.15.87:0/4252591219 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c0131b0 con 0x55804fc196d0 2024-04-22T13:34:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.871+0000 7f3450ff9640 1 -- 172.21.15.87:0/4252591219 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f344c0054d0 con 0x55804fc196d0 2024-04-22T13:34:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.872+0000 7f3467385740 1 -- 172.21.15.87:0/4252591219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 msgr2=0x55804fc19ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.872+0000 7f3467385740 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f344c013500 tx=0x7f344c00a880 comp rx=0 tx=0).stop 2024-04-22T13:34:09.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.875+0000 7f3467385740 1 -- 172.21.15.87:0/4252591219 shutdown_connections 2024-04-22T13:34:09.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.875+0000 7f3467385740 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc25050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.875+0000 7f3467385740 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc1a080 0x55804fc22630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.875+0000 7f3467385740 1 --2- 172.21.15.87:0/4252591219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc19ab0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.875+0000 7f3467385740 1 -- 172.21.15.87:0/4252591219 >> 172.21.15.87:0/4252591219 conn(0x55804fc13da0 msgr2=0x55804fc14180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:09.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.878+0000 7f3467385740 1 -- 172.21.15.87:0/4252591219 shutdown_connections 2024-04-22T13:34:09.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.878+0000 7f3467385740 1 -- 172.21.15.87:0/4252591219 wait complete. 2024-04-22T13:34:09.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.880+0000 7f3467385740 1 -- start start 2024-04-22T13:34:09.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.880+0000 7f3467385740 1 Processor -- start 2024-04-22T13:34:09.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc196d0 0x55804fc22390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc1a080 0x55804fc1f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55804fc25bc0 con 0x55804fc1a080 2024-04-22T13:34:09.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55804fb37500 con 0x55804fc196d0 2024-04-22T13:34:09.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55804fc25d40 con 0x55804fc22c00 2024-04-22T13:34:09.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.881+0000 7f3467385740 10 client.0 _ll_register_callbacks cb 0x55804fb05b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:09.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.884+0000 7f3461c55640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.884+0000 7f3461c55640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:55982/0 (socket says 172.21.15.87:55982) 2024-04-22T13:34:09.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.884+0000 7f3461c55640 1 -- 172.21.15.87:0/277025917 learned_addr learned my addr 172.21.15.87:0/277025917 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.884+0000 7f3461c55640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc196d0 0x55804fc22390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.885+0000 7f3461c55640 1 -- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc196d0 msgr2=0x55804fc22390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.885+0000 7f3461c55640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc196d0 0x55804fc22390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.885+0000 7f3461c55640 1 -- 172.21.15.87:0/277025917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc1a080 msgr2=0x55804fc1f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.885+0000 7f3461c55640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc1a080 0x55804fc1f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.885+0000 7f3461c55640 1 -- 172.21.15.87:0/277025917 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55804f8903f0 con 0x55804fc22c00 2024-04-22T13:34:09.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.886+0000 7f3461c55640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f344c004c00 tx=0x7f344c004b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9539868092770741 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.887+0000 7f3445ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.887+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c005000 con 0x55804fc22c00 2024-04-22T13:34:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.888+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f344c0051a0 con 0x55804fc22c00 2024-04-22T13:34:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.892+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344c0054d0 con 0x55804fc22c00 2024-04-22T13:34:09.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.895+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f344c0108a0 con 0x55804fc22c00 2024-04-22T13:34:09.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.896+0000 7f3467385740 10 client.25687 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:09.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.896+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55804f95a7d0 con 0x55804fc22c00 2024-04-22T13:34:09.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.897+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f344c0054d0 con 0x55804fc22c00 2024-04-22T13:34:09.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.897+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f344c01e8b0 con 0x55804fc22c00 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 10 client.25687 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 20 client.25687 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc330a0 0x55804fc35500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 4 client.25687 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:09.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.898+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55804fb1d930 con 0x55804fc330a0 2024-04-22T13:34:09.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.913+0000 7f3462456640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc330a0 0x55804fc35500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:09.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:09 smithi120 ceph-mon[31101]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 868 MiB/s wr, 1.14k op/s 2024-04-22T13:34:09.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.938+0000 7f3445ffb640 10 client.25687 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:09.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.938+0000 7f3462456640 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc330a0 0x55804fc35500 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8e25ee94825d22 server_cookie=d7fa83fe1ff88f73 in_seq=0 out_seq=0 2024-04-22T13:34:09.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.946+0000 7f3445ffb640 1 -- 172.21.15.87:0/277025917 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55804fb1d930 con 0x55804fc330a0 2024-04-22T13:34:09.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.946+0000 7f3445ffb640 10 client.25687 handle_command_reply: tid=0 2024-04-22T13:34:09.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.946+0000 7f3467385740 1 client.25687 shutdown 2024-04-22T13:34:09.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.950+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 msgr2=0x55804fc1fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:09.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.950+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f344c004c00 tx=0x7f344c004b00 comp rx=0 tx=0).stop 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 shutdown_connections 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc330a0 0x55804fc35500 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc22c00 0x55804fc1fc60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc1a080 0x55804fc1f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 --2- 172.21.15.87:0/277025917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc196d0 0x55804fc22390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3445ffb640 0 client.25687 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:09.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.951+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 >> 172.21.15.87:0/277025917 conn(0x55804fc13da0 msgr2=0x55804fc15af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:09.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.952+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 shutdown_connections 2024-04-22T13:34:09.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.953+0000 7f3467385740 1 -- 172.21.15.87:0/277025917 wait complete. 2024-04-22T13:34:09.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:09.954+0000 7f3467385740 20 client.25687 trim_cache size 0 max 16384 2024-04-22T13:34:09.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:09 smithi132 ceph-mon[31531]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 868 MiB/s wr, 1.14k op/s 2024-04-22T13:34:10.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 Processor -- start 2024-04-22T13:34:10.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 -- start start 2024-04-22T13:34:10.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc21e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3440010fb0 con 0x55804fc196d0 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f344000e880 con 0x55804fc22c00 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.019+0000 7f3467385740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f344000c6d0 con 0x55804fc1a080 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.021+0000 7f3461c55640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.021+0000 7f3461c55640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43816/0 (socket says 172.21.15.87:43816) 2024-04-22T13:34:10.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.021+0000 7f3461c55640 1 -- 172.21.15.87:0/1496275878 learned_addr learned my addr 172.21.15.87:0/1496275878 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc21e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 -- 172.21.15.87:0/1496275878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 msgr2=0x55804fc1e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 -- 172.21.15.87:0/1496275878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 msgr2=0x55804fc21e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc21e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.022+0000 7f3461c55640 1 -- 172.21.15.87:0/1496275878 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55804fc19e20 con 0x55804fc196d0 2024-04-22T13:34:10.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.029+0000 7f3461c55640 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f344c0049f0 tx=0x7f344c004b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942124dfc777fb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:10.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.039+0000 7f3447fff640 1 -- 172.21.15.87:0/1496275878 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c00abc0 con 0x55804fc196d0 2024-04-22T13:34:10.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.039+0000 7f3447fff640 1 -- 172.21.15.87:0/1496275878 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f344c035020 con 0x55804fc196d0 2024-04-22T13:34:10.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.039+0000 7f3447fff640 1 -- 172.21.15.87:0/1496275878 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c005150 con 0x55804fc196d0 2024-04-22T13:34:10.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.040+0000 7f3467385740 1 -- 172.21.15.87:0/1496275878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 msgr2=0x55804fc2eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:10.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.040+0000 7f3467385740 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f344c0049f0 tx=0x7f344c004b90 comp rx=0 tx=0).stop 2024-04-22T13:34:10.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.044+0000 7f3467385740 1 -- 172.21.15.87:0/1496275878 shutdown_connections 2024-04-22T13:34:10.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.044+0000 7f3467385740 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc21e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.045+0000 7f3467385740 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.045+0000 7f3467385740 1 --2- 172.21.15.87:0/1496275878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2eb00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.045+0000 7f3467385740 1 -- 172.21.15.87:0/1496275878 >> 172.21.15.87:0/1496275878 conn(0x55804fc13da0 msgr2=0x55804fc1e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:10.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.050+0000 7f3467385740 1 -- 172.21.15.87:0/1496275878 shutdown_connections 2024-04-22T13:34:10.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.055+0000 7f3467385740 1 -- 172.21.15.87:0/1496275878 wait complete. 2024-04-22T13:34:10.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.055+0000 7f3467385740 1 -- start start 2024-04-22T13:34:10.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.056+0000 7f3467385740 1 Processor -- start 2024-04-22T13:34:10.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.057+0000 7f3467385740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.057+0000 7f3467385740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.057+0000 7f3467385740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc1d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.058+0000 7f3467385740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f344000cd40 con 0x55804fc196d0 2024-04-22T13:34:10.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.058+0000 7f3467385740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f344000e1b0 con 0x55804fc22c00 2024-04-22T13:34:10.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.058+0000 7f3467385740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f344000d410 con 0x55804fc1a080 2024-04-22T13:34:10.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.058+0000 7f3467385740 10 client.0 _ll_register_callbacks cb 0x7f344c00ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:10.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.058+0000 7f3462456640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:10.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.059+0000 7f3462456640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56002/0 (socket says 172.21.15.87:56002) 2024-04-22T13:34:10.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.059+0000 7f3462456640 1 -- 172.21.15.87:0/1287650476 learned_addr learned my addr 172.21.15.87:0/1287650476 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:10.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.061+0000 7f3462456640 1 -- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 msgr2=0x55804fc1d0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:34:10.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.061+0000 7f3462456640 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc1d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.061+0000 7f3462456640 1 -- 172.21.15.87:0/1287650476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 msgr2=0x55804fc2b4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:34:10.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.061+0000 7f3462456640 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.061+0000 7f3462456640 1 -- 172.21.15.87:0/1287650476 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55804fc1b9b0 con 0x55804fc1a080 2024-04-22T13:34:10.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.064+0000 7f3462456640 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3458036040 tx=0x7f3458003a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d984a8c5c2cc84e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:10.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.065+0000 7f3445ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:10.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.066+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458003c80 con 0x55804fc1a080 2024-04-22T13:34:10.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.066+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344c035020 con 0x55804fc1a080 2024-04-22T13:34:10.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.066+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f345803a040 con 0x55804fc1a080 2024-04-22T13:34:10.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.070+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3458052050 con 0x55804fc1a080 2024-04-22T13:34:10.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.070+0000 7f3467385740 10 client.25690 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:10.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.070+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55804fc98900 con 0x55804fc1a080 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f344c035020 con 0x55804fc1a080 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f34580046c0 con 0x55804fc1a080 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 10 client.25690 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 10 client.25690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 20 client.25690 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc990b0 0x55804fc9b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 4 client.25690 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:10.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.073+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55804fb8bc10 con 0x55804fc990b0 2024-04-22T13:34:10.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.076+0000 7f3461c55640 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc990b0 0x55804fc9b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:10.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.103+0000 7f3461c55640 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc990b0 0x55804fc9b4d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6a99378f3a0a11 server_cookie=4539283042177b7a in_seq=0 out_seq=0 2024-04-22T13:34:10.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.104+0000 7f3445ffb640 10 client.25690 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:10.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.110+0000 7f3445ffb640 1 -- 172.21.15.87:0/1287650476 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55804fb8bc10 con 0x55804fc990b0 2024-04-22T13:34:10.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.110+0000 7f3445ffb640 10 client.25690 handle_command_reply: tid=0 2024-04-22T13:34:10.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.110+0000 7f3467385740 1 client.25690 shutdown 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.111+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 msgr2=0x55804fc1cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.111+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3458036040 tx=0x7f3458003a50 comp rx=0 tx=0).stop 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.111+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 shutdown_connections 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.111+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55804fc990b0 0x55804fc9b4d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.112+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55804fc22c00 0x55804fc1d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.112+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55804fc1a080 0x55804fc1cb80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.112+0000 7f3467385740 1 --2- 172.21.15.87:0/1287650476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55804fc196d0 0x55804fc2b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.112+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 >> 172.21.15.87:0/1287650476 conn(0x55804fc13da0 msgr2=0x55804fc1e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:10.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.112+0000 7f3445ffb640 0 client.25690 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:10.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.116+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 shutdown_connections 2024-04-22T13:34:10.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.116+0000 7f3467385740 1 -- 172.21.15.87:0/1287650476 wait complete. 2024-04-22T13:34:10.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.117+0000 7f3467385740 20 client.25690 trim_cache size 0 max 16384 2024-04-22T13:34:10.113 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:10.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.127+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 msgr2=0x7f34480a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:10.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.127+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 0x7f34480a63f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34540996b0 tx=0x7f3454011000 comp rx=0 tx=0).stop 2024-04-22T13:34:10.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.127+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 msgr2=0x7f345c1abcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:10.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.127+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f345400b9c0 tx=0x7f345400be80 comp rx=0 tx=0).stop 2024-04-22T13:34:10.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 shutdown_connections 2024-04-22T13:34:10.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34480a3f30 0x7f34480a63f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f345c1105f0 0x7f345c1abcc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f345c1099d0 0x7f345c1a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 --2- 172.21.15.87:0/2050569438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f345c108f50 0x7f345c1a7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:10.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.129+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 >> 172.21.15.87:0/2050569438 conn(0x7f345c06ced0 msgr2=0x7f345c070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:10.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.133+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 shutdown_connections 2024-04-22T13:34:10.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:10.134+0000 7f3463458640 1 -- 172.21.15.87:0/2050569438 wait complete. 2024-04-22T13:34:11.198 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:34:11.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:34:11.199 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:34:11.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.508+0000 7f81041f4640 1 -- 172.21.15.87:0/1470403109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 msgr2=0x7f80fc10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:11.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.508+0000 7f81041f4640 1 --2- 172.21.15.87:0/1470403109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 0x7f80fc10e290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009ca0 tx=0x7f80f402c140 comp rx=0 tx=0).stop 2024-04-22T13:34:11.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.510+0000 7f81041f4640 1 -- 172.21.15.87:0/1470403109 shutdown_connections 2024-04-22T13:34:11.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.510+0000 7f81041f4640 1 --2- 172.21.15.87:0/1470403109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80fc11b730 0x7f80fc11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.510+0000 7f81041f4640 1 --2- 172.21.15.87:0/1470403109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc10e7d0 0x7f80fc10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.510+0000 7f81041f4640 1 --2- 172.21.15.87:0/1470403109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 0x7f80fc10e290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.510+0000 7f81041f4640 1 -- 172.21.15.87:0/1470403109 >> 172.21.15.87:0/1470403109 conn(0x7f80fc06d060 msgr2=0x7f80fc06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:11.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.511+0000 7f81041f4640 1 -- 172.21.15.87:0/1470403109 shutdown_connections 2024-04-22T13:34:11.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.514+0000 7f81041f4640 1 -- 172.21.15.87:0/1470403109 wait complete. 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.515+0000 7f81041f4640 1 Processor -- start 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.515+0000 7f81041f4640 1 -- start start 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.515+0000 7f81041f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80fc10e7d0 0x7f80fc1a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f81041f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 0x7f80fc1a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f81041f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f81041f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc121020 con 0x7f80fc11b730 2024-04-22T13:34:11.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f81041f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc120ea0 con 0x7f80fc10e7d0 2024-04-22T13:34:11.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f81041f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1211a0 con 0x7f80fc1107a0 2024-04-22T13:34:11.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.516+0000 7f8102791640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43834/0 (socket says 172.21.15.87:43834) 2024-04-22T13:34:11.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 -- 172.21.15.87:0/1381446826 learned_addr learned my addr 172.21.15.87:0/1381446826 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 -- 172.21.15.87:0/1381446826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 msgr2=0x7f80fc1a7b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 0x7f80fc1a7b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 -- 172.21.15.87:0/1381446826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80fc10e7d0 msgr2=0x7f80fc1a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80fc10e7d0 0x7f80fc1a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.517+0000 7f8102791640 1 -- 172.21.15.87:0/1381446826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc1ac640 con 0x7f80fc11b730 2024-04-22T13:34:11.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.518+0000 7f8102791640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e970 tx=0x7f80ec00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.523+0000 7f80f2ffd640 1 -- 172.21.15.87:0/1381446826 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00ccc0 con 0x7f80fc11b730 2024-04-22T13:34:11.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.523+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc1ac930 con 0x7f80fc11b730 2024-04-22T13:34:11.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.523+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc1b41c0 con 0x7f80fc11b730 2024-04-22T13:34:11.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.524+0000 7f80f2ffd640 1 -- 172.21.15.87:0/1381446826 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ec00ce60 con 0x7f80fc11b730 2024-04-22T13:34:11.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.524+0000 7f80f2ffd640 1 -- 172.21.15.87:0/1381446826 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec010430 con 0x7f80fc11b730 2024-04-22T13:34:11.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.529+0000 7f8106de9740 1 Processor -- start 2024-04-22T13:34:11.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.530+0000 7f8106de9740 1 -- start start 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.531+0000 7f8106de9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc871268e0 0x55dc87126cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.531+0000 7f8106de9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.531+0000 7f8106de9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc8712fe10 0x55dc87132260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.531+0000 7f8106de9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55dc8708af00 con 0x55dc871268e0 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.532+0000 7f80f2ffd640 1 -- 172.21.15.87:0/1381446826 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f80ec0105d0 con 0x7f80fc11b730 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.532+0000 7f8106de9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dc87028590 con 0x55dc8712fe10 2024-04-22T13:34:11.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.532+0000 7f8106de9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dc87086ec0 con 0x55dc87127290 2024-04-22T13:34:11.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.532+0000 7f8101f90640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.533+0000 7f8101f90640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56040/0 (socket says 172.21.15.87:56040) 2024-04-22T13:34:11.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.533+0000 7f8101f90640 1 -- 172.21.15.87:0/1117917234 learned_addr learned my addr 172.21.15.87:0/1117917234 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:11.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.533+0000 7f810178f640 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc871268e0 0x55dc87126cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.533+0000 7f810178f640 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc8712fe10 0x55dc87132260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.534+0000 7f80f2ffd640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 0x7f80e80a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.534+0000 7f8102791640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 0x7f80e80a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8101f90640 1 -- 172.21.15.87:0/1117917234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc8712fe10 msgr2=0x55dc87132260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8101f90640 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc8712fe10 0x55dc87132260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8101f90640 1 -- 172.21.15.87:0/1117917234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc871268e0 msgr2=0x55dc87126cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8101f90640 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc871268e0 0x55dc87126cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8101f90640 1 -- 172.21.15.87:0/1117917234 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc86e657e0 con 0x55dc87127290 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f80f2ffd640 1 -- 172.21.15.87:0/1381446826 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f80ec0d0b10 con 0x7f80fc11b730 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.535+0000 7f8102791640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 0x7f80e80a63e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0a5200 tx=0x7f80ec0c7bd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.537+0000 7f8101f90640 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80f402c750 tx=0x7f80f4004aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5175ed08e4984d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.537+0000 7f80e7fff640 1 -- 172.21.15.87:0/1117917234 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4041020 con 0x55dc87127290 2024-04-22T13:34:11.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.537+0000 7f80e7fff640 1 -- 172.21.15.87:0/1117917234 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f4032450 con 0x55dc87127290 2024-04-22T13:34:11.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 -- 172.21.15.87:0/1117917234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 msgr2=0x55dc8712f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:11.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f80f402c750 tx=0x7f80f4004aa0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 -- 172.21.15.87:0/1117917234 shutdown_connections 2024-04-22T13:34:11.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc8712fe10 0x55dc87132260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87127290 0x55dc8712f840 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 --2- 172.21.15.87:0/1117917234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc871268e0 0x55dc87126cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.540+0000 7f8106de9740 1 -- 172.21.15.87:0/1117917234 >> 172.21.15.87:0/1117917234 conn(0x55dc87120fb0 msgr2=0x55dc87121390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:11.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.545+0000 7f8106de9740 1 -- 172.21.15.87:0/1117917234 shutdown_connections 2024-04-22T13:34:11.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.545+0000 7f8106de9740 1 -- 172.21.15.87:0/1117917234 wait complete. 2024-04-22T13:34:11.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.546+0000 7f8106de9740 1 -- start start 2024-04-22T13:34:11.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.546+0000 7f8106de9740 1 Processor -- start 2024-04-22T13:34:11.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.548+0000 7f8106de9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8712ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.548+0000 7f8106de9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.548+0000 7f8106de9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc8712d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.548+0000 7f8106de9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55dc87132d70 con 0x55dc8712fe10 2024-04-22T13:34:11.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.549+0000 7f8106de9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dc87044580 con 0x55dc87127290 2024-04-22T13:34:11.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.549+0000 7f8106de9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dc87132ef0 con 0x55dc871268e0 2024-04-22T13:34:11.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.549+0000 7f8106de9740 10 client.0 _ll_register_callbacks cb 0x55dc8704a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:11.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.551+0000 7f8101f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.551+0000 7f8101f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54886/0 (socket says 172.21.15.87:54886) 2024-04-22T13:34:11.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.551+0000 7f8101f90640 1 -- 172.21.15.87:0/2160204802 learned_addr learned my addr 172.21.15.87:0/2160204802 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:11.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.552+0000 7f810178f640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc8712d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.553+0000 7f810178f640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8712ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.554+0000 7f810178f640 1 -- 172.21.15.87:0/2160204802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 msgr2=0x55dc8712ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.554+0000 7f810178f640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8712ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.554+0000 7f810178f640 1 -- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 msgr2=0x55dc8712d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.554+0000 7f810178f640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.554+0000 7f810178f640 1 -- 172.21.15.87:0/2160204802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc86ec5f70 con 0x55dc8712fe10 2024-04-22T13:34:11.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.555+0000 7f810178f640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc8712d730 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f80fc11d5e0 tx=0x7f80f8006880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9648bb6e28f21f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.557+0000 7f80e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:11.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.557+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8011040 con 0x55dc8712fe10 2024-04-22T13:34:11.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.558+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f80f80167f0 con 0x55dc8712fe10 2024-04-22T13:34:11.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.558+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f4032450 con 0x55dc8712fe10 2024-04-22T13:34:11.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.565+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f80f80169f0 con 0x55dc8712fe10 2024-04-22T13:34:11.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.569+0000 7f8106de9740 10 client.16434 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:11.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.569+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dc86d5d170 con 0x55dc8712fe10 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.570+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f4032450 con 0x55dc8712fe10 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.570+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f80f8021030 con 0x55dc8712fe10 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.572+0000 7f8106de9740 10 client.16434 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.573+0000 7f8106de9740 10 client.16434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.573+0000 7f8106de9740 20 client.16434 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.573+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc87140370 0x55dc87142790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.573+0000 7f8106de9740 4 client.16434 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:11.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.573+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc87098920 con 0x55dc87140370 2024-04-22T13:34:11.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.576+0000 7f8101f90640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc87140370 0x55dc87142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.583+0000 7f80e57fa640 10 client.16434 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:11.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.583+0000 7f8101f90640 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc87140370 0x55dc87142790 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de2a1978c337bc5a server_cookie=8480ddb53eb608c7 in_seq=0 out_seq=0 2024-04-22T13:34:11.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.590+0000 7f80e57fa640 1 -- 172.21.15.87:0/2160204802 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc87098920 con 0x55dc87140370 2024-04-22T13:34:11.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.590+0000 7f80e57fa640 10 client.16434 handle_command_reply: tid=0 2024-04-22T13:34:11.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.592+0000 7f8106de9740 1 client.16434 shutdown 2024-04-22T13:34:11.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 msgr2=0x55dc8712d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc8712d730 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f80fc11d5e0 tx=0x7f80f8006880 comp rx=0 tx=0).stop 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 shutdown_connections 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc87140370 0x55dc87142790 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc8712d730 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 --2- 172.21.15.87:0/2160204802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8712ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f80e57fa640 0 client.16434 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:11.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 >> 172.21.15.87:0/2160204802 conn(0x55dc87120fb0 msgr2=0x55dc87122d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:11.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.593+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 shutdown_connections 2024-04-22T13:34:11.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.594+0000 7f8106de9740 1 -- 172.21.15.87:0/2160204802 wait complete. 2024-04-22T13:34:11.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.597+0000 7f8106de9740 20 client.16434 trim_cache size 0 max 16384 2024-04-22T13:34:11.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.645+0000 7f8106de9740 1 Processor -- start 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 -- start start 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87148040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f80e0011000 con 0x55dc8712fe10 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80e000e8d0 con 0x55dc87127290 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8106de9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f80e000c720 con 0x55dc871268e0 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f8101f90640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56060/0 (socket says 172.21.15.87:56060) 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 -- 172.21.15.87:0/1812462129 learned_addr learned my addr 172.21.15.87:0/1812462129 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 -- 172.21.15.87:0/1812462129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 msgr2=0x55dc8712d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 -- 172.21.15.87:0/1812462129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 msgr2=0x55dc87148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.646+0000 7f810178f640 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.647+0000 7f810178f640 1 -- 172.21.15.87:0/1812462129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc87120de0 con 0x55dc871268e0 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.647+0000 7f8101f90640 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.647+0000 7f810178f640 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55dc8713c290 tx=0x7f80f8006880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16057a57ab30bdae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.649+0000 7f80e77fe640 1 -- 172.21.15.87:0/1812462129 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f80165f0 con 0x55dc871268e0 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.649+0000 7f80e77fe640 1 -- 172.21.15.87:0/1812462129 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f8016790 con 0x55dc871268e0 2024-04-22T13:34:11.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.649+0000 7f80e77fe640 1 -- 172.21.15.87:0/1812462129 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f80196c0 con 0x55dc871268e0 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 -- 172.21.15.87:0/1812462129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 msgr2=0x55dc8713bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55dc8713c290 tx=0x7f80f8006880 comp rx=0 tx=0).stop 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 -- 172.21.15.87:0/1812462129 shutdown_connections 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc8712d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 --2- 172.21.15.87:0/1812462129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc871268e0 0x55dc8713bd10 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55dc8713c290 tx=0x7f80f8006880 comp rx=0 tx=0).stop 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 -- 172.21.15.87:0/1812462129 >> 172.21.15.87:0/1812462129 conn(0x55dc87120fb0 msgr2=0x55dc87122c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.650+0000 7f8106de9740 1 -- 172.21.15.87:0/1812462129 shutdown_connections 2024-04-22T13:34:11.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.651+0000 7f8106de9740 1 -- 172.21.15.87:0/1812462129 wait complete. 2024-04-22T13:34:11.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.651+0000 7f8106de9740 1 -- start start 2024-04-22T13:34:11.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.651+0000 7f8106de9740 1 Processor -- start 2024-04-22T13:34:11.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc87133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87134560 0x55dc871a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f80e000cd90 con 0x55dc8712fe10 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80e000e200 con 0x55dc87127290 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f80e000d460 con 0x55dc87134560 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8106de9740 10 client.0 _ll_register_callbacks cb 0x7f80f8006590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8101f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.652+0000 7f8101f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43856/0 (socket says 172.21.15.87:43856) 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.653+0000 7f8101f90640 1 -- 172.21.15.87:0/3613194736 learned_addr learned my addr 172.21.15.87:0/3613194736 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:11.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.653+0000 7f810178f640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc87133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.653+0000 7f810178f640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87134560 0x55dc871a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.653+0000 7f8101f90640 1 -- 172.21.15.87:0/3613194736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87134560 msgr2=0x55dc871a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.653+0000 7f8101f90640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87134560 0x55dc871a58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.654+0000 7f8101f90640 1 -- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 msgr2=0x55dc87133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.654+0000 7f8101f90640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc87133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:11.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.654+0000 7f8101f90640 1 -- 172.21.15.87:0/3613194736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc871259e0 con 0x55dc8712fe10 2024-04-22T13:34:11.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.654+0000 7f8101f90640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f80f402c750 tx=0x7f80f4034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2948a2a41d8861 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:11.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.654+0000 7f80e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:11.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.655+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f404d070 con 0x55dc8712fe10 2024-04-22T13:34:11.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.655+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f8016790 con 0x55dc8712fe10 2024-04-22T13:34:11.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.655+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f80f4032d30 con 0x55dc8712fe10 2024-04-22T13:34:11.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.655+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f80f4041040 con 0x55dc8712fe10 2024-04-22T13:34:11.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.659+0000 7f8106de9740 10 client.16440 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:11.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.659+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dc871a6d20 con 0x55dc8712fe10 2024-04-22T13:34:11.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.660+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f8016790 con 0x55dc8712fe10 2024-04-22T13:34:11.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.661+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f80f4050020 con 0x55dc8712fe10 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 10 client.16440 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 10 client.16440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 20 client.16440 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc871a7060 0x55dc871a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 4 client.16440 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:34:11.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.667+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dc87092790 con 0x55dc871a7060 2024-04-22T13:34:11.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.668+0000 7f810178f640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc871a7060 0x55dc871a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:11.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.669+0000 7f810178f640 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc871a7060 0x55dc871a9450 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30dc4963593bd6ae server_cookie=b93630d4c4a9e7be in_seq=0 out_seq=0 2024-04-22T13:34:11.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:11.669+0000 7f80e57fa640 10 client.16440 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:11.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:11 smithi087 ceph-mon[27999]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:11.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:11 smithi120 ceph-mon[31101]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:11.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:11 smithi132 ceph-mon[31531]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:12.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.046+0000 7f80e57fa640 1 -- 172.21.15.87:0/3613194736 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dc87092790 con 0x55dc871a7060 2024-04-22T13:34:12.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.046+0000 7f80e57fa640 10 client.16440 handle_command_reply: tid=0 2024-04-22T13:34:12.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.046+0000 7f8106de9740 1 client.16440 shutdown 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 msgr2=0x55dc87134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f80f402c750 tx=0x7f80f4034440 comp rx=0 tx=0).stop 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 shutdown_connections 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dc871a7060 0x55dc871a9450 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dc87134560 0x55dc871a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dc8712fe10 0x55dc87134020 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 --2- 172.21.15.87:0/3613194736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc87127290 0x55dc87133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 >> 172.21.15.87:0/3613194736 conn(0x55dc87120fb0 msgr2=0x55dc8711f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f80e57fa640 0 client.16440 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 shutdown_connections 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 1 -- 172.21.15.87:0/3613194736 wait complete. 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.047+0000 7f8106de9740 20 client.16440 trim_cache size 0 max 16384 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:34:12.044 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "54065d93-47b3-4495-ae96-164b25903ab8", 2024-04-22T13:34:12.045 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:34:12.045 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.053+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 msgr2=0x7f80e80a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.053+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 0x7f80e80a63e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0a5200 tx=0x7f80ec0c7bd0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.053+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 msgr2=0x7f80fc1abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.053+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e970 tx=0x7f80ec00ed70 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 shutdown_connections 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f80e80a3f20 0x7f80e80a63e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80fc11b730 0x7f80fc1abf10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80fc1107a0 0x7f80fc1a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 --2- 172.21.15.87:0/1381446826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80fc10e7d0 0x7f80fc1a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.054+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 >> 172.21.15.87:0/1381446826 conn(0x7f80fc06d060 msgr2=0x7f80fc11a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.055+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 shutdown_connections 2024-04-22T13:34:12.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.055+0000 7f81041f4640 1 -- 172.21.15.87:0/1381446826 wait complete. 2024-04-22T13:34:12.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:34:12.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 -- 172.21.15.87:0/2267850361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70109d20 msgr2=0x7fda7010a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 --2- 172.21.15.87:0/2267850361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70109d20 0x7fda7010a1d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda68009ca0 tx=0x7fda6802c1c0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 -- 172.21.15.87:0/2267850361 shutdown_connections 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 --2- 172.21.15.87:0/2267850361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70109d20 0x7fda7010a1d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 --2- 172.21.15.87:0/2267850361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda7010b2a0 0x7fda701097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 --2- 172.21.15.87:0/2267850361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda7010a980 0x7fda7010ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.546+0000 7fda7797e640 1 -- 172.21.15.87:0/2267850361 >> 172.21.15.87:0/2267850361 conn(0x7fda7006d070 msgr2=0x7fda7006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.551+0000 7fda7797e640 1 -- 172.21.15.87:0/2267850361 shutdown_connections 2024-04-22T13:34:12.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.551+0000 7fda7797e640 1 -- 172.21.15.87:0/2267850361 wait complete. 2024-04-22T13:34:12.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.552+0000 7fda7797e640 1 Processor -- start 2024-04-22T13:34:12.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.552+0000 7fda7797e640 1 -- start start 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda7010a980 0x7fda7011f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda70118a40 con 0x7fda7010a980 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda701188c0 con 0x7fda7010b2a0 2024-04-22T13:34:12.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7797e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda70118bc0 con 0x7fda70109d20 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.553+0000 7fda7571a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54908/0 (socket says 172.21.15.87:54908) 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56086/0 (socket says 172.21.15.87:56086) 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda75f1b640 1 -- 172.21.15.87:0/959305540 learned_addr learned my addr 172.21.15.87:0/959305540 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 -- 172.21.15.87:0/959305540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 msgr2=0x7fda7011a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 -- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda7010a980 msgr2=0x7fda7011f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda7010a980 0x7fda7011f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda7571a640 1 -- 172.21.15.87:0/959305540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7011a990 con 0x7fda70109d20 2024-04-22T13:34:12.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.554+0000 7fda75f1b640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:34:12.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.566+0000 7fda7571a640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00d6d0 tx=0x7fda6c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.570+0000 7fda667fc640 1 -- 172.21.15.87:0/959305540 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c017070 con 0x7fda70109d20 2024-04-22T13:34:12.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.570+0000 7fda667fc640 1 -- 172.21.15.87:0/959305540 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda6c009820 con 0x7fda70109d20 2024-04-22T13:34:12.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.570+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7011ac80 con 0x7fda70109d20 2024-04-22T13:34:12.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.570+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda70122ea0 con 0x7fda70109d20 2024-04-22T13:34:12.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.574+0000 7fda667fc640 1 -- 172.21.15.87:0/959305540 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c004c40 con 0x7fda70109d20 2024-04-22T13:34:12.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.574+0000 7fda7a56b740 1 Processor -- start 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 -- start start 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cc3ea0 0x560732ccc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560732c281e0 con 0x560732cc3ea0 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560732bc5390 con 0x560732ccca80 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.577+0000 7fda7a56b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560732c241a0 con 0x560732cc3580 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.578+0000 7fda74f19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.578+0000 7fda74f19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54912/0 (socket says 172.21.15.87:54912) 2024-04-22T13:34:12.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.578+0000 7fda74f19640 1 -- 172.21.15.87:0/1949624796 learned_addr learned my addr 172.21.15.87:0/1949624796 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:12.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.579+0000 7fda74f19640 1 -- 172.21.15.87:0/1949624796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 msgr2=0x560732cc3960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:12.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.579+0000 7fda74f19640 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc3960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.579+0000 7fda74f19640 1 -- 172.21.15.87:0/1949624796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cc3ea0 msgr2=0x560732ccc540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:34:12.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.579+0000 7fda74f19640 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cc3ea0 0x560732ccc540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.579+0000 7fda74f19640 1 -- 172.21.15.87:0/1949624796 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560732a047d0 con 0x560732ccca80 2024-04-22T13:34:12.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.584+0000 7fda74f19640 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda600097b0 tx=0x7fda60009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1accb93c01f79c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.588+0000 7fda667fc640 1 -- 172.21.15.87:0/959305540 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda6c005220 con 0x7fda70109d20 2024-04-22T13:34:12.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.588+0000 7fda5b7fe640 1 -- 172.21.15.87:0/1949624796 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60015020 con 0x560732ccca80 2024-04-22T13:34:12.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.589+0000 7fda5b7fe640 1 -- 172.21.15.87:0/1949624796 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda60011420 con 0x560732ccca80 2024-04-22T13:34:12.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.591+0000 7fda7a56b740 1 -- 172.21.15.87:0/1949624796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 msgr2=0x560732ccee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.591+0000 7fda7a56b740 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda600097b0 tx=0x7fda60009c70 comp rx=0 tx=0).stop 2024-04-22T13:34:12.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.591+0000 7fda667fc640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 0x7fda5c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.591+0000 7fda667fc640 1 -- 172.21.15.87:0/959305540 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fda6c0d0000 con 0x7fda70109d20 2024-04-22T13:34:12.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.593+0000 7fda75f1b640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 0x7fda5c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 -- 172.21.15.87:0/1949624796 shutdown_connections 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732ccee70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cc3ea0 0x560732ccc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda75f1b640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 0x7fda5c0a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda70071ce0 tx=0x7fda6802c6e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 --2- 172.21.15.87:0/1949624796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 -- 172.21.15.87:0/1949624796 >> 172.21.15.87:0/1949624796 conn(0x560732cbdcb0 msgr2=0x560732cbe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.594+0000 7fda7a56b740 1 -- 172.21.15.87:0/1949624796 shutdown_connections 2024-04-22T13:34:12.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.596+0000 7fda7a56b740 1 -- 172.21.15.87:0/1949624796 wait complete. 2024-04-22T13:34:12.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.596+0000 7fda7a56b740 1 -- start start 2024-04-22T13:34:12.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.596+0000 7fda7a56b740 1 Processor -- start 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560732ccf8c0 con 0x560732ccca80 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560732be12f0 con 0x560732cc3ea0 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560732ccfa40 con 0x560732cc3580 2024-04-22T13:34:12.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.597+0000 7fda7a56b740 10 client.0 _ll_register_callbacks cb 0x560732bee580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:12.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.598+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.598+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43880/0 (socket says 172.21.15.87:43880) 2024-04-22T13:34:12.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.598+0000 7fda75f1b640 1 -- 172.21.15.87:0/3976642954 learned_addr learned my addr 172.21.15.87:0/3976642954 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:12.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.598+0000 7fda7571a640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda74f19640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda7571a640 1 -- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 msgr2=0x560732cc9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda7571a640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda7571a640 1 -- 172.21.15.87:0/3976642954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 msgr2=0x560732cca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda7571a640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.600+0000 7fda7571a640 1 -- 172.21.15.87:0/3976642954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56073293a3f0 con 0x560732cc3580 2024-04-22T13:34:12.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.610+0000 7fda7571a640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc9a40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0cd760 tx=0x7fda6c09c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=837ab652ccc73dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.610+0000 7fda58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:12.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.611+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0b5060 con 0x560732cc3580 2024-04-22T13:34:12.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.612+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fda6c0986a0 con 0x560732cc3580 2024-04-22T13:34:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.612+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda60011420 con 0x560732cc3580 2024-04-22T13:34:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.613+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fda6c0ba050 con 0x560732cc3580 2024-04-22T13:34:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.613+0000 7fda7a56b740 10 client.25714 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.613+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5607328fa0f0 con 0x560732cc3580 2024-04-22T13:34:12.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.613+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda60011420 con 0x560732cc3580 2024-04-22T13:34:12.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.614+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fda6c097d80 con 0x560732cc3580 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.616+0000 7fda7a56b740 10 client.25714 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.616+0000 7fda7a56b740 10 client.25714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.616+0000 7fda7a56b740 20 client.25714 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.617+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732cdc9d0 0x560732cdedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.617+0000 7fda7a56b740 4 client.25714 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.617+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560732c35c00 con 0x560732cdc9d0 2024-04-22T13:34:12.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.617+0000 7fda74f19640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732cdc9d0 0x560732cdedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.621+0000 7fda74f19640 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732cdc9d0 0x560732cdedc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=327d2a1624b82372 server_cookie=bed29e23efd0f355 in_seq=0 out_seq=0 2024-04-22T13:34:12.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.621+0000 7fda58ff9640 10 client.25714 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:12.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.624+0000 7fda58ff9640 1 -- 172.21.15.87:0/3976642954 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560732c35c00 con 0x560732cdc9d0 2024-04-22T13:34:12.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.624+0000 7fda58ff9640 10 client.25714 handle_command_reply: tid=0 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 client.25714 shutdown 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 msgr2=0x560732cc9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc9a40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0cd760 tx=0x7fda6c09c0f0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 shutdown_connections 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732cdc9d0 0x560732cdedc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3976642954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc3580 0x560732cc9a40 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0cd760 tx=0x7fda6c09c0f0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 >> 172.21.15.87:0/3976642954 conn(0x560732cbdcb0 msgr2=0x560732cbfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.629+0000 7fda58ff9640 0 client.25714 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:12.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.631+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 shutdown_connections 2024-04-22T13:34:12.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.631+0000 7fda7a56b740 1 -- 172.21.15.87:0/3976642954 wait complete. 2024-04-22T13:34:12.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.631+0000 7fda7a56b740 20 client.25714 trim_cache size 0 max 16384 2024-04-22T13:34:12.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.732+0000 7fda7a56b740 1 Processor -- start 2024-04-22T13:34:12.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.734+0000 7fda7a56b740 1 -- start start 2024-04-22T13:34:12.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.734+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.734+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cd8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.735+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 0x560732ce46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.735+0000 7fda7a56b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda54010fb0 con 0x560732ccca80 2024-04-22T13:34:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.735+0000 7fda7a56b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400e880 con 0x560732cc3ea0 2024-04-22T13:34:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.735+0000 7fda7a56b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400c6d0 con 0x560732cca310 2024-04-22T13:34:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.737+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 0x560732ce46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.737+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 0x560732ce46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56120/0 (socket says 172.21.15.87:56120) 2024-04-22T13:34:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.737+0000 7fda75f1b640 1 -- 172.21.15.87:0/3102071990 learned_addr learned my addr 172.21.15.87:0/3102071990 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.737+0000 7fda7571a640 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.741+0000 7fda7571a640 1 -- 172.21.15.87:0/3102071990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 msgr2=0x560732ce46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.741+0000 7fda7571a640 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 0x560732ce46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.741+0000 7fda7571a640 1 -- 172.21.15.87:0/3102071990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 msgr2=0x560732cd8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.741+0000 7fda7571a640 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cd8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.741+0000 7fda7571a640 1 -- 172.21.15.87:0/3102071990 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560732cbdae0 con 0x560732cc3ea0 2024-04-22T13:34:12.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.753+0000 7fda7571a640 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc89a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda6c098490 tx=0x7fda6c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f1918dad4fbf0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.757+0000 7fda5affd640 1 -- 172.21.15.87:0/3102071990 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0a8020 con 0x560732cc3ea0 2024-04-22T13:34:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.757+0000 7fda5affd640 1 -- 172.21.15.87:0/3102071990 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda6c097d80 con 0x560732cc3ea0 2024-04-22T13:34:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.757+0000 7fda5affd640 1 -- 172.21.15.87:0/3102071990 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c014070 con 0x560732cc3ea0 2024-04-22T13:34:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.759+0000 7fda7a56b740 1 -- 172.21.15.87:0/3102071990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 msgr2=0x560732cc89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc89a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda6c098490 tx=0x7fda6c0057f0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 -- 172.21.15.87:0/3102071990 shutdown_connections 2024-04-22T13:34:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cca310 0x560732ce46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732ccca80 0x560732cd8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 --2- 172.21.15.87:0/3102071990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732cc3ea0 0x560732cc89a0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fda6c098490 tx=0x7fda6c0057f0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.760+0000 7fda7a56b740 1 -- 172.21.15.87:0/3102071990 >> 172.21.15.87:0/3102071990 conn(0x560732cbdcb0 msgr2=0x560732cbcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.765+0000 7fda7a56b740 1 -- 172.21.15.87:0/3102071990 shutdown_connections 2024-04-22T13:34:12.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.769+0000 7fda7a56b740 1 -- 172.21.15.87:0/3102071990 wait complete. 2024-04-22T13:34:12.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.770+0000 7fda7a56b740 1 -- start start 2024-04-22T13:34:12.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.770+0000 7fda7a56b740 1 Processor -- start 2024-04-22T13:34:12.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.771+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 0x560732cc6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.772+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732cc6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.772+0000 7fda7a56b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.772+0000 7fda7a56b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400cd40 con 0x560732cca310 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.772+0000 7fda7a56b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400e1b0 con 0x560732ccca80 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.772+0000 7fda7a56b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400d410 con 0x560732cc74f0 2024-04-22T13:34:12.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.773+0000 7fda7a56b740 10 client.0 _ll_register_callbacks cb 0x7fda6c0982d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:12.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.773+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.773+0000 7fda75f1b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56124/0 (socket says 172.21.15.87:56124) 2024-04-22T13:34:12.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.773+0000 7fda75f1b640 1 -- 172.21.15.87:0/4095742878 learned_addr learned my addr 172.21.15.87:0/4095742878 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda75f1b640 1 -- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 msgr2=0x560732cc6f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda75f1b640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732cc6f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda75f1b640 1 -- 172.21.15.87:0/4095742878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 msgr2=0x560732cc6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda7571a640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 0x560732cc6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda75f1b640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 0x560732cc6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda75f1b640 1 -- 172.21.15.87:0/4095742878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560732d42690 con 0x560732cc74f0 2024-04-22T13:34:12.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.774+0000 7fda7571a640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 0x560732cc6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.784+0000 7fda75f1b640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda6802c750 tx=0x7fda6802c1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1dbbecddc2b552d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.784+0000 7fda58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.784+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68044070 con 0x560732cc74f0 2024-04-22T13:34:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.785+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fda68056470 con 0x560732cc74f0 2024-04-22T13:34:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.785+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda6c097d80 con 0x560732cc74f0 2024-04-22T13:34:12.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.785+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fda68039040 con 0x560732cc74f0 2024-04-22T13:34:12.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.786+0000 7fda7a56b740 10 client.25720 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:12.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.786+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560732d43980 con 0x560732cc74f0 2024-04-22T13:34:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.789+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda68051070 con 0x560732cc74f0 2024-04-22T13:34:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.791+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fda6805e020 con 0x560732cc74f0 2024-04-22T13:34:12.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 10 client.25720 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:12.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 10 client.25720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:12.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 20 client.25720 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:12.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732d44030 0x560732d46450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:12.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 4 client.25720 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:34:12.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.793+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fda6c097d80 con 0x560732d44030 2024-04-22T13:34:12.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.800+0000 7fda7571a640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732d44030 0x560732d46450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.807+0000 7fda7571a640 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732d44030 0x560732d46450 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8189c0826f567bfe server_cookie=e192b94cc93f6f21 in_seq=0 out_seq=0 2024-04-22T13:34:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.807+0000 7fda58ff9640 10 client.25720 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda58ff9640 1 -- 172.21.15.87:0/4095742878 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7fda6c097d80 con 0x560732d44030 2024-04-22T13:34:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda58ff9640 10 client.25720 handle_command_reply: tid=0 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 client.25720 shutdown 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 msgr2=0x560732d41f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda6802c750 tx=0x7fda6802c1c0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 shutdown_connections 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560732d44030 0x560732d46450 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560732cc74f0 0x560732d41f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560732ccca80 0x560732cc6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 --2- 172.21.15.87:0/4095742878 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560732cca310 0x560732cc6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda58ff9640 0 client.25720 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.812+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 >> 172.21.15.87:0/4095742878 conn(0x560732cbdcb0 msgr2=0x560732cc92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.813+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 shutdown_connections 2024-04-22T13:34:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.813+0000 7fda7a56b740 1 -- 172.21.15.87:0/4095742878 wait complete. 2024-04-22T13:34:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.815+0000 7fda7a56b740 20 client.25720 trim_cache size 0 max 16384 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "scrub active (10 inodes in the stack)", 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "54065d93-47b3-4495-ae96-164b25903ab8": { 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "54065d93-47b3-4495-ae96-164b25903ab8", 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:34:12.811 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:34:12.812 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.820+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 msgr2=0x7fda5c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.821+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 0x7fda5c0a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda70071ce0 tx=0x7fda6802c6e0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.822+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 msgr2=0x7fda7011efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.822+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00d6d0 tx=0x7fda6c00db90 comp rx=0 tx=0).stop 2024-04-22T13:34:12.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 shutdown_connections 2024-04-22T13:34:12.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda5c0a3f20 0x7fda5c0a63e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda7010b2a0 0x7fda7011a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda7010a980 0x7fda7011f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 --2- 172.21.15.87:0/959305540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda70109d20 0x7fda7011efc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:12.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 >> 172.21.15.87:0/959305540 conn(0x7fda7006d070 msgr2=0x7fda7010d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:12.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 shutdown_connections 2024-04-22T13:34:12.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:12.828+0000 7fda7797e640 1 -- 172.21.15.87:0/959305540 wait complete. 2024-04-22T13:34:12.869 INFO:tasks.cephfs.filesystem:scrub status for tag:54065d93-47b3-4495-ae96-164b25903ab8 - {'path': '/', 'tag': '54065d93-47b3-4495-ae96-164b25903ab8', 'options': 'recursive,force'} 2024-04-22T13:34:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.231+0000 7f1c57fff640 1 -- 172.21.15.87:0/2808645797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58106100 msgr2=0x7f1c58106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.231+0000 7f1c57fff640 1 --2- 172.21.15.87:0/2808645797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58106100 0x7f1c58106590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009be0 tx=0x7f1c5002bfc0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.233+0000 7f1c57fff640 1 -- 172.21.15.87:0/2808645797 shutdown_connections 2024-04-22T13:34:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.233+0000 7f1c57fff640 1 --2- 172.21.15.87:0/2808645797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58106100 0x7f1c58106590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.233+0000 7f1c57fff640 1 --2- 172.21.15.87:0/2808645797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c58101620 0x7f1c58105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.233+0000 7f1c57fff640 1 --2- 172.21.15.87:0/2808645797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58100d00 0x7f1c581010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.233+0000 7f1c57fff640 1 -- 172.21.15.87:0/2808645797 >> 172.21.15.87:0/2808645797 conn(0x7f1c580fc770 msgr2=0x7f1c580feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:13.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.246+0000 7f1c57fff640 1 -- 172.21.15.87:0/2808645797 shutdown_connections 2024-04-22T13:34:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.249+0000 7f1c57fff640 1 -- 172.21.15.87:0/2808645797 wait complete. 2024-04-22T13:34:13.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.251+0000 7f1c57fff640 1 Processor -- start 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.251+0000 7f1c57fff640 1 -- start start 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c58100d00 0x7f1c581a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 0x7f1c581a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 0x7f1c5819e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581185e0 con 0x7f1c58100d00 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58118460 con 0x7f1c58101620 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.252+0000 7f1c57fff640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c58118760 con 0x7f1c58106100 2024-04-22T13:34:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.253+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 0x7f1c5819e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.253+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 0x7f1c5819e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56126/0 (socket says 172.21.15.87:56126) 2024-04-22T13:34:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.253+0000 7f1c577fe640 1 -- 172.21.15.87:0/133685899 learned_addr learned my addr 172.21.15.87:0/133685899 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.253+0000 7f1c567fc640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 0x7f1c581a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 -- 172.21.15.87:0/133685899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 msgr2=0x7f1c5819e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 0x7f1c5819e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 -- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c58100d00 msgr2=0x7f1c581a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c58100d00 0x7f1c581a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 -- 172.21.15.87:0/133685899 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5819f050 con 0x7f1c58101620 2024-04-22T13:34:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.254+0000 7f1c567fc640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 0x7f1c581a4c90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00e6f0 tx=0x7f1c4c00ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.256 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.260+0000 7f1c47fff640 1 -- 172.21.15.87:0/133685899 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00cab0 con 0x7f1c58101620 2024-04-22T13:34:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.260+0000 7f1c47fff640 1 -- 172.21.15.87:0/133685899 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c00cc50 con 0x7f1c58101620 2024-04-22T13:34:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.260+0000 7f1c47fff640 1 -- 172.21.15.87:0/133685899 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c018710 con 0x7f1c58101620 2024-04-22T13:34:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.260+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5819f340 con 0x7f1c58101620 2024-04-22T13:34:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.260+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c581070b0 con 0x7f1c58101620 2024-04-22T13:34:13.264 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.268+0000 7f1c601c8740 1 Processor -- start 2024-04-22T13:34:13.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.272+0000 7f1c47fff640 1 -- 172.21.15.87:0/133685899 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1c4c0188b0 con 0x7f1c58101620 2024-04-22T13:34:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.273+0000 7f1c601c8740 1 -- start start 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d7a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d7b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559989cdf8d0 con 0x559989d7ad50 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559989c7ce80 con 0x559989d7bac0 2024-04-22T13:34:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.274+0000 7f1c601c8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559989cdb890 con 0x559989d842d0 2024-04-22T13:34:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.275+0000 7f1c47fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 0x7f1c400a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.276+0000 7f1c47fff640 1 -- 172.21.15.87:0/133685899 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1c4c0d0780 con 0x7f1c58101620 2024-04-22T13:34:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.277+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d7b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.277+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d7a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.277+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.277+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56140/0 (socket says 172.21.15.87:56140) 2024-04-22T13:34:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.277+0000 7f1c567fc640 1 -- 172.21.15.87:0/3081263499 learned_addr learned my addr 172.21.15.87:0/3081263499 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.291+0000 7f1c577fe640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 0x7f1c400a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.292+0000 7f1c567fc640 1 -- 172.21.15.87:0/3081263499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 msgr2=0x559989d7a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.292+0000 7f1c567fc640 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d7a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.292+0000 7f1c567fc640 1 -- 172.21.15.87:0/3081263499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 msgr2=0x559989d7b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.292+0000 7f1c567fc640 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d7b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.292+0000 7f1c567fc640 1 -- 172.21.15.87:0/3081263499 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559989c9e320 con 0x559989d842d0 2024-04-22T13:34:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.297+0000 7f1c567fc640 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c09c430 tx=0x7f1c4c09c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fccf6bd5382741ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.298+0000 7f1c577fe640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 0x7f1c400a63e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c5002e010 tx=0x7f1c5002bfc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.298+0000 7f1c457fa640 1 -- 172.21.15.87:0/3081263499 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c00ce00 con 0x559989d842d0 2024-04-22T13:34:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.298+0000 7f1c457fa640 1 -- 172.21.15.87:0/3081263499 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c09aa30 con 0x559989d842d0 2024-04-22T13:34:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.298+0000 7f1c457fa640 1 -- 172.21.15.87:0/3081263499 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c018e00 con 0x559989d842d0 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.300+0000 7f1c601c8740 1 -- 172.21.15.87:0/3081263499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 msgr2=0x559989d866c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.300+0000 7f1c601c8740 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c09c430 tx=0x7f1c4c09c870 comp rx=0 tx=0).stop 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.301+0000 7f1c601c8740 1 -- 172.21.15.87:0/3081263499 shutdown_connections 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.301+0000 7f1c601c8740 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d866c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.301+0000 7f1c601c8740 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d7b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.301+0000 7f1c601c8740 1 --2- 172.21.15.87:0/3081263499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d7a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.302+0000 7f1c601c8740 1 -- 172.21.15.87:0/3081263499 >> 172.21.15.87:0/3081263499 conn(0x559989d41c90 msgr2=0x559989d75a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.302+0000 7f1c601c8740 1 -- 172.21.15.87:0/3081263499 shutdown_connections 2024-04-22T13:34:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.302+0000 7f1c601c8740 1 -- 172.21.15.87:0/3081263499 wait complete. 2024-04-22T13:34:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.304+0000 7f1c601c8740 1 -- start start 2024-04-22T13:34:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.304+0000 7f1c601c8740 1 Processor -- start 2024-04-22T13:34:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.306+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 0x559989d83b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.306+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d80f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.306+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.307+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 0x559989d83b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.307+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 0x559989d83b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54966/0 (socket says 172.21.15.87:54966) 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.307+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.307+0000 7f1c577fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43918/0 (socket says 172.21.15.87:43918) 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d80f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d80f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56142/0 (socket says 172.21.15.87:56142) 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c56ffd640 1 -- 172.21.15.87:0/1279064053 learned_addr learned my addr 172.21.15.87:0/1279064053 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559989d87050 con 0x559989d842d0 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559989c67eb0 con 0x559989d7ad50 2024-04-22T13:34:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.311+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559989d871d0 con 0x559989d7bac0 2024-04-22T13:34:13.308 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.312+0000 7f1c601c8740 10 client.0 _ll_register_callbacks cb 0x559989c94b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.313+0000 7f1c577fe640 1 -- 172.21.15.87:0/1279064053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 msgr2=0x559989d80f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.313+0000 7f1c577fe640 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d80f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.313+0000 7f1c577fe640 1 -- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 msgr2=0x559989d83b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.313+0000 7f1c577fe640 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 0x559989d83b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.313+0000 7f1c577fe640 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599899f2210 con 0x559989d842d0 2024-04-22T13:34:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.314+0000 7f1c577fe640 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1c50004af0 tx=0x7f1c50009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fa6bad537eabf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.314+0000 7f1c3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.317+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50051070 con 0x559989d842d0 2024-04-22T13:34:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.318+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c4c09aa30 con 0x559989d842d0 2024-04-22T13:34:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.318+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1c500045a0 con 0x559989d842d0 2024-04-22T13:34:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.324+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c5005d050 con 0x559989d842d0 2024-04-22T13:34:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.324+0000 7f1c601c8740 10 client.16452 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.324+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559989abe620 con 0x559989d842d0 2024-04-22T13:34:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.325+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c4c09aa30 con 0x559989d842d0 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1c5004c070 con 0x559989d842d0 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 10 client.16452 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 20 client.16452 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989d944e0 0x559989d96980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 4 client.16452 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.326+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559989cf3880 con 0x559989d944e0 2024-04-22T13:34:13.324 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.328+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989d944e0 0x559989d96980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.337+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989d944e0 0x559989d96980 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0378cf31d327d4f server_cookie=2df1e9e0129bed2d in_seq=0 out_seq=0 2024-04-22T13:34:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.338+0000 7f1c3e7fc640 10 client.16452 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.355+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/1279064053 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559989cf3880 con 0x559989d944e0 2024-04-22T13:34:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.355+0000 7f1c3e7fc640 10 client.16452 handle_command_reply: tid=0 2024-04-22T13:34:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.355+0000 7f1c601c8740 1 client.16452 shutdown 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 msgr2=0x559989d81490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1c50004af0 tx=0x7f1c50009500 comp rx=0 tx=0).stop 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 shutdown_connections 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989d944e0 0x559989d96980 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d842d0 0x559989d81490 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d80f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 --2- 172.21.15.87:0/1279064053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7ad50 0x559989d83b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 >> 172.21.15.87:0/1279064053 conn(0x559989d41c90 msgr2=0x559989d77790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c3e7fc640 0 client.16452 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 shutdown_connections 2024-04-22T13:34:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 1 -- 172.21.15.87:0/1279064053 wait complete. 2024-04-22T13:34:13.361 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.363+0000 7f1c601c8740 20 client.16452 trim_cache size 0 max 16384 2024-04-22T13:34:13.432 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.437+0000 7f1c601c8740 1 Processor -- start 2024-04-22T13:34:13.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.437+0000 7f1c601c8740 1 -- start start 2024-04-22T13:34:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.439+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.440+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d70970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.440+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d842d0 0x559989d9c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.440+0000 7f1c601c8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c38010fa0 con 0x559989d7ad50 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.440+0000 7f1c601c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3800e870 con 0x559989d842d0 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.440+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d70970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.441+0000 7f1c567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d70970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:56158/0 (socket says 172.21.15.87:56158) 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.441+0000 7f1c567fc640 1 -- 172.21.15.87:0/2832738895 learned_addr learned my addr 172.21.15.87:0/2832738895 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.441+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.441+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3800c6c0 con 0x559989d7bac0 2024-04-22T13:34:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.446+0000 7f1c56ffd640 1 -- 172.21.15.87:0/2832738895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 msgr2=0x559989d70970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.446+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d70970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.446+0000 7f1c56ffd640 1 -- 172.21.15.87:0/2832738895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d842d0 msgr2=0x559989d9c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.446+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d842d0 0x559989d9c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.446+0000 7f1c56ffd640 1 -- 172.21.15.87:0/2832738895 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559989d7f4c0 con 0x559989d7ad50 2024-04-22T13:34:13.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.454+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8ff80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x559989d97200 tx=0x7f1c4800b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82447c66b376ba4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.456+0000 7f1c44ff9640 1 -- 172.21.15.87:0/2832738895 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48021070 con 0x559989d7ad50 2024-04-22T13:34:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.456+0000 7f1c44ff9640 1 -- 172.21.15.87:0/2832738895 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c48015990 con 0x559989d7ad50 2024-04-22T13:34:13.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.457+0000 7f1c44ff9640 1 -- 172.21.15.87:0/2832738895 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48024a60 con 0x559989d7ad50 2024-04-22T13:34:13.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.458+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 msgr2=0x559989d8ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:13.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.459+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8ff80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x559989d97200 tx=0x7f1c4800b720 comp rx=0 tx=0).stop 2024-04-22T13:34:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.460+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 shutdown_connections 2024-04-22T13:34:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.460+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d842d0 0x559989d9c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.460+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d7bac0 0x559989d70970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.460+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2832738895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8ff80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.457 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.460+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 >> 172.21.15.87:0/2832738895 conn(0x559989d41c90 msgr2=0x559989d7c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:13.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.467+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 shutdown_connections 2024-04-22T13:34:13.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.467+0000 7f1c601c8740 1 -- 172.21.15.87:0/2832738895 wait complete. 2024-04-22T13:34:13.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.468+0000 7f1c601c8740 1 -- start start 2024-04-22T13:34:13.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.468+0000 7f1c601c8740 1 Processor -- start 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d8c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d7d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3800cd30 con 0x559989d7ad50 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3800e1a0 con 0x559989d7bac0 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3800d400 con 0x559989d842d0 2024-04-22T13:34:13.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.469+0000 7f1c601c8740 10 client.0 _ll_register_callbacks cb 0x7f1c5002c960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.473+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.473+0000 7f1c56ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43938/0 (socket says 172.21.15.87:43938) 2024-04-22T13:34:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.473+0000 7f1c56ffd640 1 -- 172.21.15.87:0/2398304520 learned_addr learned my addr 172.21.15.87:0/2398304520 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.477+0000 7f1c567fc640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d8c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.477+0000 7f1c577fe640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d7d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.486+0000 7f1c577fe640 1 -- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 msgr2=0x559989d8c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.486+0000 7f1c577fe640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d8c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.486+0000 7f1c577fe640 1 -- 172.21.15.87:0/2398304520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 msgr2=0x559989d8c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.486+0000 7f1c577fe640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.486+0000 7f1c577fe640 1 -- 172.21.15.87:0/2398304520 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559989d7d720 con 0x559989d842d0 2024-04-22T13:34:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.495+0000 7f1c577fe640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d7d1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c50036040 tx=0x7f1c50043680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bbd35dcd74206fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:13.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.496+0000 7f1c3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.501+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50051040 con 0x559989d842d0 2024-04-22T13:34:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.502+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1c50004520 con 0x559989d842d0 2024-04-22T13:34:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.507+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c48015990 con 0x559989d842d0 2024-04-22T13:34:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.514+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c5002c640 con 0x559989d842d0 2024-04-22T13:34:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.519+0000 7f1c601c8740 10 client.25735 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.519+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559989dfa2f0 con 0x559989d842d0 2024-04-22T13:34:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.524+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c50047070 con 0x559989d842d0 2024-04-22T13:34:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.524+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1c5004c040 con 0x559989d842d0 2024-04-22T13:34:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 10 client.25735 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 20 client.25735 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989dfa9b0 0x559989dfcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 4 client.25735 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:34:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f1c48015990 con 0x559989dfa9b0 2024-04-22T13:34:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.528+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989dfa9b0 0x559989dfcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.530+0000 7f1c56ffd640 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989dfa9b0 0x559989dfcda0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76756be1d4279715 server_cookie=edc0c901e099e697 in_seq=0 out_seq=0 2024-04-22T13:34:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:13.533+0000 7f1c3e7fc640 10 client.25735 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:13.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:13 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:34:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:13 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:13 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:34:13.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:13 smithi120 ceph-mon[31101]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:13.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:13 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:34:13.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:13 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:13.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:13 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:34:13.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:13 smithi132 ceph-mon[31531]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:13.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:13 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:34:13.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:13 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:13.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:13 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:34:13.991 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:13 smithi087 ceph-mon[27999]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 704 MiB/s wr, 1.31k op/s 2024-04-22T13:34:15.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:15 smithi120 ceph-mon[31101]: pgmap v74: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.57k op/s 2024-04-22T13:34:15.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:15 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:15.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:15 smithi132 ceph-mon[31531]: pgmap v74: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.57k op/s 2024-04-22T13:34:15.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:15 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:15.995 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:15 smithi087 ceph-mon[27999]: pgmap v74: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.57k op/s 2024-04-22T13:34:15.995 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:15 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:16.884 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.889+0000 7f1c3e7fc640 1 -- 172.21.15.87:0/2398304520 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f1c48015990 con 0x559989dfa9b0 2024-04-22T13:34:16.885 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.889+0000 7f1c3e7fc640 10 client.25735 handle_command_reply: tid=0 2024-04-22T13:34:16.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.890+0000 7f1c601c8740 1 client.25735 shutdown 2024-04-22T13:34:16.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 msgr2=0x559989d7d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:16.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d7d1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c50036040 tx=0x7f1c50043680 comp rx=0 tx=0).stop 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 shutdown_connections 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559989dfa9b0 0x559989dfcda0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559989d842d0 0x559989d7d1e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559989d7bac0 0x559989d8c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 --2- 172.21.15.87:0/2398304520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559989d7ad50 0x559989d8c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 >> 172.21.15.87:0/2398304520 conn(0x559989d41c90 msgr2=0x559989d74250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c3e7fc640 0 client.25735 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.892+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 shutdown_connections 2024-04-22T13:34:16.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.893+0000 7f1c601c8740 1 -- 172.21.15.87:0/2398304520 wait complete. 2024-04-22T13:34:16.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.899+0000 7f1c601c8740 20 client.25735 trim_cache size 0 max 16384 2024-04-22T13:34:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.908+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 msgr2=0x7f1c400a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.909+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 0x7f1c400a63e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c5002e010 tx=0x7f1c5002bfc0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.909+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 msgr2=0x7f1c581a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.909+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 0x7f1c581a4c90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00e6f0 tx=0x7f1c4c00ebb0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 shutdown_connections 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c400a3f20 0x7f1c400a63e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c58106100 0x7f1c5819e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c58101620 0x7f1c581a4c90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 --2- 172.21.15.87:0/133685899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c58100d00 0x7f1c581a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.911+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 >> 172.21.15.87:0/133685899 conn(0x7f1c580fc770 msgr2=0x7f1c580fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.921+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 shutdown_connections 2024-04-22T13:34:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:16.921+0000 7f1c57fff640 1 -- 172.21.15.87:0/133685899 wait complete. 2024-04-22T13:34:16.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:16 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:34:16.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:16 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:34:16.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:16 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:34:17.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:17 smithi120 ceph-mon[31101]: pgmap v75: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 677 MiB/s wr, 2.31k op/s 2024-04-22T13:34:17.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:17 smithi132 ceph-mon[31531]: pgmap v75: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 677 MiB/s wr, 2.31k op/s 2024-04-22T13:34:17.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:17 smithi087 ceph-mon[27999]: pgmap v75: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 677 MiB/s wr, 2.31k op/s 2024-04-22T13:34:19.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:19 smithi120 ceph-mon[31101]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 895 MiB/s wr, 2.75k op/s 2024-04-22T13:34:19.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:19 smithi132 ceph-mon[31531]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 895 MiB/s wr, 2.75k op/s 2024-04-22T13:34:19.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:19 smithi087 ceph-mon[27999]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 895 MiB/s wr, 2.75k op/s 2024-04-22T13:34:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:21 smithi120 ceph-mon[31101]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 829 MiB/s wr, 2.41k op/s 2024-04-22T13:34:21.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:21 smithi132 ceph-mon[31531]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 829 MiB/s wr, 2.41k op/s 2024-04-22T13:34:21.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:21 smithi087 ceph-mon[27999]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 829 MiB/s wr, 2.41k op/s 2024-04-22T13:34:23.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:23 smithi120 ceph-mon[31101]: pgmap v78: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 701 MiB/s wr, 2.00k op/s 2024-04-22T13:34:23.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:23 smithi132 ceph-mon[31531]: pgmap v78: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 701 MiB/s wr, 2.00k op/s 2024-04-22T13:34:23.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:23 smithi087 ceph-mon[27999]: pgmap v78: 97 pgs: 97 active+clean; 16 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 701 MiB/s wr, 2.00k op/s 2024-04-22T13:34:25.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:25 smithi120 ceph-mon[31101]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.98k op/s 2024-04-22T13:34:25.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:25 smithi132 ceph-mon[31531]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.98k op/s 2024-04-22T13:34:25.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:25 smithi087 ceph-mon[27999]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.98k op/s 2024-04-22T13:34:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:27 smithi120 ceph-mon[31101]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.72k op/s 2024-04-22T13:34:27.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:27 smithi132 ceph-mon[31531]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.72k op/s 2024-04-22T13:34:27.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:27 smithi087 ceph-mon[27999]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.72k op/s 2024-04-22T13:34:29.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:29 smithi120 ceph-mon[31101]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 966 MiB/s wr, 2.41k op/s 2024-04-22T13:34:29.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:29 smithi132 ceph-mon[31531]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 966 MiB/s wr, 2.41k op/s 2024-04-22T13:34:29.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:29 smithi087 ceph-mon[27999]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 966 MiB/s wr, 2.41k op/s 2024-04-22T13:34:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.314+0000 7fd71b462640 1 -- 172.21.15.87:0/2218093928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71411b730 msgr2=0x7fd71411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.314+0000 7fd71b462640 1 --2- 172.21.15.87:0/2218093928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71411b730 0x7fd71411db20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd704009930 tx=0x7fd70402bd10 comp rx=0 tx=0).stop 2024-04-22T13:34:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.320+0000 7fd71b462640 1 -- 172.21.15.87:0/2218093928 shutdown_connections 2024-04-22T13:34:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.320+0000 7fd71b462640 1 --2- 172.21.15.87:0/2218093928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71411b730 0x7fd71411db20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.320+0000 7fd71b462640 1 --2- 172.21.15.87:0/2218093928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714110b20 0x7fd714109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.320+0000 7fd71b462640 1 --2- 172.21.15.87:0/2218093928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd714110740 0x7fd714108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.320+0000 7fd71b462640 1 -- 172.21.15.87:0/2218093928 >> 172.21.15.87:0/2218093928 conn(0x7fd71406d020 msgr2=0x7fd71406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.327+0000 7fd71b462640 1 -- 172.21.15.87:0/2218093928 shutdown_connections 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.328+0000 7fd71b462640 1 -- 172.21.15.87:0/2218093928 wait complete. 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.329+0000 7fd71b462640 1 Processor -- start 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.329+0000 7fd71b462640 1 -- start start 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd714110740 0x7fd71411ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714114c20 0x7fd7141150a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141113e0 con 0x7fd714110740 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111260 con 0x7fd714114c20 2024-04-22T13:34:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd71b462640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111560 con 0x7fd71411b730 2024-04-22T13:34:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.330+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:50250/0 (socket says 172.21.15.87:50250) 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 -- 172.21.15.87:0/1795402803 learned_addr learned my addr 172.21.15.87:0/1795402803 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7199ff640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714114c20 0x7fd7141150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 -- 172.21.15.87:0/1795402803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714114c20 msgr2=0x7fd7141150a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714114c20 0x7fd7141150a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 -- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd714110740 msgr2=0x7fd71411ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd714110740 0x7fd71411ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.331+0000 7fd7189fd640 1 -- 172.21.15.87:0/1795402803 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd714115960 con 0x7fd71411b730 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.332+0000 7fd7189fd640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00b9c0 tx=0x7fd70c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.333+0000 7fd70a7fc640 1 -- 172.21.15.87:0/1795402803 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c019070 con 0x7fd71411b730 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.333+0000 7fd70a7fc640 1 -- 172.21.15.87:0/1795402803 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c00cba0 con 0x7fd71411b730 2024-04-22T13:34:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.333+0000 7fd70a7fc640 1 -- 172.21.15.87:0/1795402803 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0125a0 con 0x7fd71411b730 2024-04-22T13:34:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.338+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71407a210 con 0x7fd71411b730 2024-04-22T13:34:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.338+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71407a6e0 con 0x7fd71411b730 2024-04-22T13:34:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.346+0000 7fd71e04b740 1 Processor -- start 2024-04-22T13:34:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.349+0000 7fd70a7fc640 1 -- 172.21.15.87:0/1795402803 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd70c014440 con 0x7fd71411b730 2024-04-22T13:34:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.351+0000 7fd70a7fc640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 0x7fd7000a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 -- start start 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd70a7fc640 1 -- 172.21.15.87:0/1795402803 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd70c0cfc00 con 0x7fd71411b730 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a20128d0 0x5611a2011620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5611a1f768d0 con 0x5611a2011b60 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5611a1f13e80 con 0x5611a20128d0 2024-04-22T13:34:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.352+0000 7fd71e04b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5611a1f72890 con 0x5611a201b0e0 2024-04-22T13:34:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.354+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.354+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44770/0 (socket says 172.21.15.87:44770) 2024-04-22T13:34:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.354+0000 7fd7189fd640 1 -- 172.21.15.87:0/3343948909 learned_addr learned my addr 172.21.15.87:0/3343948909 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.354+0000 7fd7199ff640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 0x7fd7000a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.354 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7199ff640 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.355 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7189fd640 1 -- 172.21.15.87:0/3343948909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 msgr2=0x5611a201d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7189fd640 1 -- 172.21.15.87:0/3343948909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a20128d0 msgr2=0x5611a2011620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a20128d0 0x5611a2011620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.358+0000 7fd7189fd640 1 -- 172.21.15.87:0/3343948909 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a1f35320 con 0x5611a2011b60 2024-04-22T13:34:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.359+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd70c098120 tx=0x7fd70c09b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8f6a1c2ec30ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.360+0000 7fd6ff7fe640 1 -- 172.21.15.87:0/3343948909 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0b5220 con 0x5611a2011b60 2024-04-22T13:34:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.361+0000 7fd6ff7fe640 1 -- 172.21.15.87:0/3343948909 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c0a0070 con 0x5611a2011b60 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.362+0000 7fd71e04b740 1 -- 172.21.15.87:0/3343948909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 msgr2=0x5611a2011fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.362+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd70c098120 tx=0x7fd70c09b930 comp rx=0 tx=0).stop 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd7199ff640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 0x7fd7000a64b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd704009850 tx=0x7fd70402c770 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd71e04b740 1 -- 172.21.15.87:0/3343948909 shutdown_connections 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a2011b60 0x5611a2011fe0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3343948909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a20128d0 0x5611a2011620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.363+0000 7fd71e04b740 1 -- 172.21.15.87:0/3343948909 >> 172.21.15.87:0/3343948909 conn(0x5611a1fd8c90 msgr2=0x5611a200c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.365+0000 7fd71e04b740 1 -- 172.21.15.87:0/3343948909 shutdown_connections 2024-04-22T13:34:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.368+0000 7fd71e04b740 1 -- 172.21.15.87:0/3343948909 wait complete. 2024-04-22T13:34:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.368+0000 7fd71e04b740 1 -- start start 2024-04-22T13:34:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 Processor -- start 2024-04-22T13:34:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a2018460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5611a201de60 con 0x5611a20128d0 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5611a1efeeb0 con 0x5611a201b0e0 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.369+0000 7fd71e04b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5611a201dfe0 con 0x5611a2011b60 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.370+0000 7fd71e04b740 10 client.0 _ll_register_callbacks cb 0x5611a1f2bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.370+0000 7fd7191fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.370+0000 7fd7191fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:50272/0 (socket says 172.21.15.87:50272) 2024-04-22T13:34:30.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.370+0000 7fd7191fe640 1 -- 172.21.15.87:0/4225592577 learned_addr learned my addr 172.21.15.87:0/4225592577 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:30.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.370+0000 7fd7199ff640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a2018460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7189fd640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7191fe640 1 -- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 msgr2=0x5611a2018460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7191fe640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a2018460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7191fe640 1 -- 172.21.15.87:0/4225592577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 msgr2=0x5611a2017e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7191fe640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.371+0000 7fd7191fe640 1 -- 172.21.15.87:0/4225592577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a1c89210 con 0x5611a2011b60 2024-04-22T13:34:30.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.372+0000 7fd7191fe640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd710009ee0 tx=0x7fd71000b4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=116148404e1837a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.372+0000 7fd6fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.373+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710015020 con 0x5611a2011b60 2024-04-22T13:34:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.373+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5611a1f8a880 con 0x5611a2011b60 2024-04-22T13:34:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.374+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd710011620 con 0x5611a2011b60 2024-04-22T13:34:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.374+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd710020050 con 0x5611a2011b60 2024-04-22T13:34:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.374+0000 7fd71e04b740 10 client.25741 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5611a1d55620 con 0x5611a2011b60 2024-04-22T13:34:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5611a1f8a880 con 0x5611a2011b60 2024-04-22T13:34:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd710019a60 con 0x5611a2011b60 2024-04-22T13:34:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 10 client.25741 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:30.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 10 client.25741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:30.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 20 client.25741 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:30.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a202b6f0 0x5611a202db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 4 client.25741 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:30.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.375+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611a1f163d0 con 0x5611a202b6f0 2024-04-22T13:34:30.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.379+0000 7fd7189fd640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a202b6f0 0x5611a202db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.384 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.389+0000 7fd7189fd640 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a202b6f0 0x5611a202db60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a166c0ce3e36ab server_cookie=877db2cb7cd627d9 in_seq=0 out_seq=0 2024-04-22T13:34:30.384 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.389+0000 7fd6fcff9640 10 client.25741 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.393+0000 7fd6fcff9640 1 -- 172.21.15.87:0/4225592577 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611a1f163d0 con 0x5611a202b6f0 2024-04-22T13:34:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.393+0000 7fd6fcff9640 10 client.25741 handle_command_reply: tid=0 2024-04-22T13:34:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.393+0000 7fd71e04b740 1 client.25741 shutdown 2024-04-22T13:34:30.393 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 msgr2=0x5611a201aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd710009ee0 tx=0x7fd71000b4b0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 shutdown_connections 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a202b6f0 0x5611a202db60 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a2018460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 --2- 172.21.15.87:0/4225592577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a201aac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.398+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 >> 172.21.15.87:0/4225592577 conn(0x5611a1fd8c90 msgr2=0x5611a200e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.399+0000 7fd6fcff9640 0 client.25741 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.403+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 shutdown_connections 2024-04-22T13:34:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.403+0000 7fd71e04b740 1 -- 172.21.15.87:0/4225592577 wait complete. 2024-04-22T13:34:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.406+0000 7fd71e04b740 20 client.25741 trim_cache size 0 max 16384 2024-04-22T13:34:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.440+0000 7fd71e04b740 1 Processor -- start 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.440+0000 7fd71e04b740 1 -- start start 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a2026db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a201a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141d8ad0 con 0x5611a20128d0 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd714082ba0 con 0x5611a201b0e0 2024-04-22T13:34:30.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd71e04b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd71406e580 con 0x5611a2011b60 2024-04-22T13:34:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44782/0 (socket says 172.21.15.87:44782) 2024-04-22T13:34:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 -- 172.21.15.87:0/3310908442 learned_addr learned my addr 172.21.15.87:0/3310908442 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 -- 172.21.15.87:0/3310908442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 msgr2=0x5611a2026db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a2026db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 -- 172.21.15.87:0/3310908442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 msgr2=0x5611a201a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a201a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.441+0000 7fd7189fd640 1 -- 172.21.15.87:0/3310908442 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611a20187d0 con 0x5611a20128d0 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.442+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd70c09bc50 tx=0x7fd70c0cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b9511948154a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.442+0000 7fd6feffd640 1 -- 172.21.15.87:0/3310908442 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0a4930 con 0x5611a20128d0 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.442+0000 7fd6feffd640 1 -- 172.21.15.87:0/3310908442 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c0a0070 con 0x5611a20128d0 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.442+0000 7fd6feffd640 1 -- 172.21.15.87:0/3310908442 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0a44a0 con 0x5611a20128d0 2024-04-22T13:34:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- 172.21.15.87:0/3310908442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 msgr2=0x5611a2017df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd70c09bc50 tx=0x7fd70c0cf040 comp rx=0 tx=0).stop 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- 172.21.15.87:0/3310908442 shutdown_connections 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a201b0e0 0x5611a201a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a2017df0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3310908442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a2011b60 0x5611a2026db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- 172.21.15.87:0/3310908442 >> 172.21.15.87:0/3310908442 conn(0x5611a1fd8c90 msgr2=0x5611a20176e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- 172.21.15.87:0/3310908442 shutdown_connections 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- 172.21.15.87:0/3310908442 wait complete. 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 -- start start 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.443+0000 7fd71e04b740 1 Processor -- start 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a2011b60 0x5611a201e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd714081060 con 0x5611a20128d0 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7140824d0 con 0x5611a2011b60 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd714081730 con 0x5611a201b0e0 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd71e04b740 10 client.0 _ll_register_callbacks cb 0x7fd710009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd7189fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44790/0 (socket says 172.21.15.87:44790) 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.444+0000 7fd7189fd640 1 -- 172.21.15.87:0/3836325407 learned_addr learned my addr 172.21.15.87:0/3836325407 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 -- 172.21.15.87:0/3836325407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 msgr2=0x5611a201ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 -- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a2011b60 msgr2=0x5611a201e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a2011b60 0x5611a201e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 -- 172.21.15.87:0/3836325407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611a208fcd0 con 0x5611a20128d0 2024-04-22T13:34:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd7189fd640 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd70c004b90 tx=0x7fd70c0a4a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f35d39b50e5d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd6fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0bc1b0 con 0x5611a20128d0 2024-04-22T13:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd70c0a44a0 con 0x5611a20128d0 2024-04-22T13:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70c0a0070 con 0x5611a20128d0 2024-04-22T13:34:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd70c0a46a0 con 0x5611a20128d0 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd71e04b740 10 client.16473 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.445+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5611a2090f80 con 0x5611a20128d0 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd70c0a0070 con 0x5611a20128d0 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd70c097a10 con 0x5611a20128d0 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 10 client.16473 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 10 client.16473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 20 client.16473 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:30.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a20911b0 0x5611a20935a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 4 client.16473 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:34:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.446+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5611a1f842f0 con 0x5611a20911b0 2024-04-22T13:34:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.450+0000 7fd7191fe640 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a20911b0 0x5611a20935a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.452+0000 7fd6fcff9640 10 client.16473 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:30.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:30.452+0000 7fd7191fe640 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a20911b0 0x5611a20935a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab9ad1d1d421d9eb server_cookie=2609583e46dbe63c in_seq=0 out_seq=0 2024-04-22T13:34:31.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:31 smithi120 ceph-mon[31101]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 897 MiB/s wr, 2.42k op/s 2024-04-22T13:34:31.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:31 smithi132 ceph-mon[31531]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 897 MiB/s wr, 2.42k op/s 2024-04-22T13:34:31.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:31 smithi087 ceph-mon[27999]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 897 MiB/s wr, 2.42k op/s 2024-04-22T13:34:33.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:33 smithi132 ceph-mon[31531]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 750 MiB/s wr, 2.13k op/s 2024-04-22T13:34:33.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:33 smithi087 ceph-mon[27999]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 750 MiB/s wr, 2.13k op/s 2024-04-22T13:34:34.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:33 smithi120 ceph-mon[31101]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 750 MiB/s wr, 2.13k op/s 2024-04-22T13:34:35.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:35 smithi132 ceph-mon[31531]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.84k op/s 2024-04-22T13:34:35.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:35 smithi087 ceph-mon[27999]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.84k op/s 2024-04-22T13:34:36.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:35 smithi120 ceph-mon[31101]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.84k op/s 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.373+0000 7fd6fcff9640 1 -- 172.21.15.87:0/3836325407 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5611a1f842f0 con 0x5611a20911b0 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.373+0000 7fd6fcff9640 10 client.16473 handle_command_reply: tid=0 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.373+0000 7fd71e04b740 1 client.16473 shutdown 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 msgr2=0x5611a201e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd70c004b90 tx=0x7fd70c0a4a90 comp rx=0 tx=0).stop 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 shutdown_connections 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611a20911b0 0x5611a20935a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611a201b0e0 0x5611a201ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611a20128d0 0x5611a201e7c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 --2- 172.21.15.87:0/3836325407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611a2011b60 0x5611a201e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 >> 172.21.15.87:0/3836325407 conn(0x5611a1fd8c90 msgr2=0x5611a200b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd6fcff9640 0 client.16473 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 shutdown_connections 2024-04-22T13:34:36.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.374+0000 7fd71e04b740 1 -- 172.21.15.87:0/3836325407 wait complete. 2024-04-22T13:34:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.375+0000 7fd71e04b740 20 client.16473 trim_cache size 0 max 16384 2024-04-22T13:34:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:34:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:34:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:34:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 msgr2=0x7fd7000a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 0x7fd7000a64b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd704009850 tx=0x7fd70402c770 comp rx=0 tx=0).stop 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 msgr2=0x7fd71411b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00b9c0 tx=0x7fd70c00be80 comp rx=0 tx=0).stop 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 shutdown_connections 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7000a3ff0 0x7fd7000a64b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd714114c20 0x7fd7141150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd71411b730 0x7fd71411b050 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 --2- 172.21.15.87:0/1795402803 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd714110740 0x7fd71411ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 >> 172.21.15.87:0/1795402803 conn(0x7fd71406d020 msgr2=0x7fd71410d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 shutdown_connections 2024-04-22T13:34:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:36.376+0000 7fd71b462640 1 -- 172.21.15.87:0/1795402803 wait complete. 2024-04-22T13:34:37.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:37 smithi087 ceph-mon[27999]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 741 MiB/s wr, 1.86k op/s 2024-04-22T13:34:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:37 smithi120 ceph-mon[31101]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 741 MiB/s wr, 1.86k op/s 2024-04-22T13:34:38.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:37 smithi132 ceph-mon[31531]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 741 MiB/s wr, 1.86k op/s 2024-04-22T13:34:40.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:39 smithi120 ceph-mon[31101]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 958 MiB/s wr, 2.49k op/s 2024-04-22T13:34:40.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:39 smithi132 ceph-mon[31531]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 958 MiB/s wr, 2.49k op/s 2024-04-22T13:34:40.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:39 smithi087 ceph-mon[27999]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 958 MiB/s wr, 2.49k op/s 2024-04-22T13:34:42.053 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:41 smithi087 ceph-mon[27999]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.14k op/s 2024-04-22T13:34:42.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:41 smithi120 ceph-mon[31101]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.14k op/s 2024-04-22T13:34:42.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:41 smithi132 ceph-mon[31531]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.14k op/s 2024-04-22T13:34:42.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:34:42.870 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:34:43.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.377+0000 7f015f59e640 1 -- 172.21.15.87:0/1206035916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 msgr2=0x7f0160110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.377+0000 7f015f59e640 1 --2- 172.21.15.87:0/1206035916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f0160110940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0150009c10 tx=0x7f015002bef0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.380+0000 7f015f59e640 1 -- 172.21.15.87:0/1206035916 shutdown_connections 2024-04-22T13:34:43.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.380+0000 7f015f59e640 1 --2- 172.21.15.87:0/1206035916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010ebe0 0x7f016011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.380+0000 7f015f59e640 1 --2- 172.21.15.87:0/1206035916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010e030 0x7f016010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.380+0000 7f015f59e640 1 --2- 172.21.15.87:0/1206035916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f0160110940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.380+0000 7f015f59e640 1 -- 172.21.15.87:0/1206035916 >> 172.21.15.87:0/1206035916 conn(0x7f016006ce30 msgr2=0x7f016006d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.381+0000 7f015f59e640 1 -- 172.21.15.87:0/1206035916 shutdown_connections 2024-04-22T13:34:43.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.381+0000 7f015f59e640 1 -- 172.21.15.87:0/1206035916 wait complete. 2024-04-22T13:34:43.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.383+0000 7f015f59e640 1 Processor -- start 2024-04-22T13:34:43.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.383+0000 7f015f59e640 1 -- start start 2024-04-22T13:34:43.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010ebe0 0x7f01601a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f01601abb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0160120de0 con 0x7f016010e030 2024-04-22T13:34:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0160120c60 con 0x7f016010ebe0 2024-04-22T13:34:43.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.385+0000 7f015f59e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0160120f60 con 0x7f0160110560 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.388+0000 7f015e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.388+0000 7f015e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33672/0 (socket says 172.21.15.87:33672) 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.388+0000 7f015e59c640 1 -- 172.21.15.87:0/2369612885 learned_addr learned my addr 172.21.15.87:0/2369612885 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.390+0000 7f015dd9b640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010ebe0 0x7f01601a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.390+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f01601abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.394+0000 7f015e59c640 1 -- 172.21.15.87:0/2369612885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 msgr2=0x7f01601abb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.394+0000 7f015e59c640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f01601abb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.394+0000 7f015e59c640 1 -- 172.21.15.87:0/2369612885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010ebe0 msgr2=0x7f01601a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.394+0000 7f015e59c640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010ebe0 0x7f01601a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.394+0000 7f015e59c640 1 -- 172.21.15.87:0/2369612885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01601ac270 con 0x7f016010e030 2024-04-22T13:34:43.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.398+0000 7f015e59c640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0150009be0 tx=0x7f0150003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.399+0000 7f01577fe640 1 -- 172.21.15.87:0/2369612885 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150043020 con 0x7f016010e030 2024-04-22T13:34:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.400+0000 7f01577fe640 1 -- 172.21.15.87:0/2369612885 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0150003e30 con 0x7f016010e030 2024-04-22T13:34:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.400+0000 7f01577fe640 1 -- 172.21.15.87:0/2369612885 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150034ad0 con 0x7f016010e030 2024-04-22T13:34:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.401+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01601ac500 con 0x7f016010e030 2024-04-22T13:34:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.401+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01601b3df0 con 0x7f016010e030 2024-04-22T13:34:43.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.408+0000 7f0167a6c740 1 Processor -- start 2024-04-22T13:34:43.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.409+0000 7f0167a6c740 1 -- start start 2024-04-22T13:34:43.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.411+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.412+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.412+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 0x55f1948b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.412+0000 7f0167a6c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f19480b1e0 con 0x55f1948a6710 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.412+0000 7f0167a6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1947a8390 con 0x55f1948afc10 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.412+0000 7f0167a6c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1948071a0 con 0x55f1948a7030 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.413+0000 7f015e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.413+0000 7f015e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34258/0 (socket says 172.21.15.87:34258) 2024-04-22T13:34:43.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.413+0000 7f015e59c640 1 -- 172.21.15.87:0/2563248107 learned_addr learned my addr 172.21.15.87:0/2563248107 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.413+0000 7f01577fe640 1 -- 172.21.15.87:0/2369612885 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0150046020 con 0x7f016010e030 2024-04-22T13:34:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.420+0000 7f015e59c640 1 -- 172.21.15.87:0/2563248107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 msgr2=0x55f1948b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.420+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015dd9b640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 0x55f1948b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015e59c640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 0x55f1948b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015e59c640 1 -- 172.21.15.87:0/2563248107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 msgr2=0x55f1948a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015e59c640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015e59c640 1 -- 172.21.15.87:0/2563248107 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1945e77d0 con 0x55f1948a7030 2024-04-22T13:34:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015dd9b640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 0x55f1948b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:43.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.421+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:43.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.424+0000 7f01577fe640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 0x7f014c0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.424+0000 7f015e59c640 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f01500f0710 tx=0x7f01500f6ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b099cec7c148e027 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.425+0000 7f01577fe640 1 -- 172.21.15.87:0/2369612885 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f01500f2f60 con 0x7f016010e030 2024-04-22T13:34:43.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.429+0000 7f0154ff9640 1 -- 172.21.15.87:0/2563248107 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01500bb2c0 con 0x55f1948a7030 2024-04-22T13:34:43.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.429+0000 7f0154ff9640 1 -- 172.21.15.87:0/2563248107 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0150100070 con 0x55f1948a7030 2024-04-22T13:34:43.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.429+0000 7f0154ff9640 1 -- 172.21.15.87:0/2563248107 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150033710 con 0x55f1948a7030 2024-04-22T13:34:43.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.430+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 0x7f014c0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.434+0000 7f0167a6c740 1 -- 172.21.15.87:0/2563248107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 msgr2=0x55f1948af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.434+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f01500f0710 tx=0x7f01500f6ef0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.441+0000 7f0167a6c740 1 -- 172.21.15.87:0/2563248107 shutdown_connections 2024-04-22T13:34:43.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.441+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948afc10 0x55f1948b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.441+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948af6d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.441+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2563248107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.441+0000 7f0167a6c740 1 -- 172.21.15.87:0/2563248107 >> 172.21.15.87:0/2563248107 conn(0x55f1948a0e40 msgr2=0x55f1948a1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.448+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 0x7f014c0a63f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f015800bbc0 tx=0x7f0158005f30 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.448+0000 7f0167a6c740 1 -- 172.21.15.87:0/2563248107 shutdown_connections 2024-04-22T13:34:43.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.448+0000 7f0167a6c740 1 -- 172.21.15.87:0/2563248107 wait complete. 2024-04-22T13:34:43.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.451+0000 7f0167a6c740 1 -- start start 2024-04-22T13:34:43.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.452+0000 7f0167a6c740 1 Processor -- start 2024-04-22T13:34:43.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948afc10 0x55f1948ad700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f1948071a0 con 0x55f1948afc10 2024-04-22T13:34:43.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1947c42f0 con 0x55f1948a6710 2024-04-22T13:34:43.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1948b27c0 con 0x55f1948a7030 2024-04-22T13:34:43.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.456+0000 7f0167a6c740 10 client.0 _ll_register_callbacks cb 0x55f1947d1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:43.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.458+0000 7f015dd9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.458+0000 7f015e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.458+0000 7f015e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54624/0 (socket says 172.21.15.87:54624) 2024-04-22T13:34:43.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.458+0000 7f015e59c640 1 -- 172.21.15.87:0/4097232861 learned_addr learned my addr 172.21.15.87:0/4097232861 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:43.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.460+0000 7f015e59c640 1 -- 172.21.15.87:0/4097232861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 msgr2=0x55f1948ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.460+0000 7f015e59c640 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.460+0000 7f015e59c640 1 -- 172.21.15.87:0/4097232861 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948afc10 msgr2=0x55f1948ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.460+0000 7f015e59c640 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948afc10 0x55f1948ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.460+0000 7f015e59c640 1 -- 172.21.15.87:0/4097232861 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f19451d3f0 con 0x55f1948a6710 2024-04-22T13:34:43.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.465+0000 7f015e59c640 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f01500c7760 tx=0x7f01500f0740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b661ac52b38320 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.466+0000 7f0145ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:43.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.467+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150034c70 con 0x55f1948a6710 2024-04-22T13:34:43.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.467+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f01500bd320 con 0x55f1948a6710 2024-04-22T13:34:43.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.470+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0150100070 con 0x55f1948a6710 2024-04-22T13:34:43.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.474+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f01500be980 con 0x55f1948a6710 2024-04-22T13:34:43.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.475+0000 7f0167a6c740 10 client.25742 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:43.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.475+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f1944dd0f0 con 0x55f1948a6710 2024-04-22T13:34:43.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.478+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0150100070 con 0x55f1948a6710 2024-04-22T13:34:43.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.479+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0150050060 con 0x55f1948a6710 2024-04-22T13:34:43.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.479+0000 7f0167a6c740 10 client.25742 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:43.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.479+0000 7f0167a6c740 10 client.25742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:43.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.480+0000 7f0167a6c740 20 client.25742 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:43.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.480+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f1948bf600 0x55f1948c19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.480+0000 7f0167a6c740 4 client.25742 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:43.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.480+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f194818c00 con 0x55f1948bf600 2024-04-22T13:34:43.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.482+0000 7f015dd9b640 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f1948bf600 0x55f1948c19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.486+0000 7f015dd9b640 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f1948bf600 0x55f1948c19f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=327ee71d88f39e1f server_cookie=d6fcb848f75e0d1d in_seq=0 out_seq=0 2024-04-22T13:34:43.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.487+0000 7f0145ffb640 10 client.25742 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:43.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.492+0000 7f0145ffb640 1 -- 172.21.15.87:0/4097232861 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f194818c00 con 0x55f1948bf600 2024-04-22T13:34:43.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.492+0000 7f0145ffb640 10 client.25742 handle_command_reply: tid=0 2024-04-22T13:34:43.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.492+0000 7f0167a6c740 1 client.25742 shutdown 2024-04-22T13:34:43.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.494+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 msgr2=0x55f1948acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.494+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f01500c7760 tx=0x7f01500f0740 comp rx=0 tx=0).stop 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 shutdown_connections 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f1948bf600 0x55f1948c19f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948afc10 0x55f1948ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 --2- 172.21.15.87:0/4097232861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a6710 0x55f1948acc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.495+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 >> 172.21.15.87:0/4097232861 conn(0x55f1948a0e40 msgr2=0x55f1948a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.499+0000 7f0145ffb640 0 client.25742 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:43.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.503+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 shutdown_connections 2024-04-22T13:34:43.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.504+0000 7f0167a6c740 1 -- 172.21.15.87:0/4097232861 wait complete. 2024-04-22T13:34:43.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.504+0000 7f0167a6c740 20 client.25742 trim_cache size 0 max 16384 2024-04-22T13:34:43.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.629+0000 7f0167a6c740 1 Processor -- start 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.631+0000 7f0167a6c740 1 -- start start 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a7030 0x55f1948ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f01601bef60 con 0x55f1948a6710 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f01601bc830 con 0x55f1948a7030 2024-04-22T13:34:43.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.632+0000 7f0167a6c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f01601ba680 con 0x55f1948afc10 2024-04-22T13:34:43.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.634+0000 7f015ed9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.634+0000 7f015ed9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34268/0 (socket says 172.21.15.87:34268) 2024-04-22T13:34:43.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.634+0000 7f015ed9d640 1 -- 172.21.15.87:0/2049338575 learned_addr learned my addr 172.21.15.87:0/2049338575 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:43.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.634+0000 7f015dd9b640 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a7030 0x55f1948ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 -- 172.21.15.87:0/2049338575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a7030 msgr2=0x55f1948ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a7030 0x55f1948ad520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 -- 172.21.15.87:0/2049338575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 msgr2=0x55f1948af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 -- 172.21.15.87:0/2049338575 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1948a7470 con 0x55f1948afc10 2024-04-22T13:34:43.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.636+0000 7f015ed9d640 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0158010e70 tx=0x7f01580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9a3e865c950cf38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.637+0000 7f0147fff640 1 -- 172.21.15.87:0/2049338575 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158023070 con 0x55f1948afc10 2024-04-22T13:34:43.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.637+0000 7f0147fff640 1 -- 172.21.15.87:0/2049338575 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f015801f900 con 0x55f1948afc10 2024-04-22T13:34:43.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 -- 172.21.15.87:0/2049338575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 msgr2=0x55f1948c7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0158010e70 tx=0x7f01580046c0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 -- 172.21.15.87:0/2049338575 shutdown_connections 2024-04-22T13:34:43.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948afc10 0x55f1948c7110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948a7030 0x55f1948ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 --2- 172.21.15.87:0/2049338575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.639+0000 7f0167a6c740 1 -- 172.21.15.87:0/2049338575 >> 172.21.15.87:0/2049338575 conn(0x55f1948a0e40 msgr2=0x55f1948a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.643+0000 7f0167a6c740 1 -- 172.21.15.87:0/2049338575 shutdown_connections 2024-04-22T13:34:43.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.643+0000 7f0167a6c740 1 -- 172.21.15.87:0/2049338575 wait complete. 2024-04-22T13:34:43.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.644+0000 7f0167a6c740 1 -- start start 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.644+0000 7f0167a6c740 1 Processor -- start 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948b3fc0 0x55f194924980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f01601bacf0 con 0x55f1948a6710 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f01601bc160 con 0x55f1948b3fc0 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f01601bb3c0 con 0x55f1948a7030 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.645+0000 7f0167a6c740 10 client.0 _ll_register_callbacks cb 0x7f01500f6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33714/0 (socket says 172.21.15.87:33714) 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 -- 172.21.15.87:0/568013610 learned_addr learned my addr 172.21.15.87:0/568013610 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 -- 172.21.15.87:0/568013610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 msgr2=0x55f1948b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 -- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948b3fc0 msgr2=0x55f194924980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948b3fc0 0x55f194924980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.646+0000 7f015e59c640 1 -- 172.21.15.87:0/568013610 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f194925090 con 0x55f1948a6710 2024-04-22T13:34:43.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.647+0000 7f015e59c640 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f01500fb570 tx=0x7f01500c33d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14309ae6b5ade6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:43.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.647+0000 7f0145ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:43.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.647+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01500beb60 con 0x55f1948a6710 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.647+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f194812a70 con 0x55f1948a6710 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.647+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f01500c30b0 con 0x55f1948a6710 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.648+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f01500bedf0 con 0x55f1948a6710 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.648+0000 7f0167a6c740 10 client.16482 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.648+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f194925090 con 0x55f1948a6710 2024-04-22T13:34:43.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.648+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f194812a70 con 0x55f1948a6710 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f01500c7800 con 0x55f1948a6710 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 10 client.16482 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 20 client.16482 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f194926740 0x55f194928b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 4 client.16482 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:34:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.650+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f19478e590 con 0x55f194926740 2024-04-22T13:34:43.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.656+0000 7f015dd9b640 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f194926740 0x55f194928b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.664+0000 7f015dd9b640 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f194926740 0x55f194928b70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fb9ffa239952334 server_cookie=ea40abf24eb24c5f in_seq=0 out_seq=0 2024-04-22T13:34:43.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.664+0000 7f0145ffb640 10 client.16482 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.669+0000 7f0145ffb640 1 -- 172.21.15.87:0/568013610 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f19478e590 con 0x55f194926740 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.669+0000 7f0145ffb640 10 client.16482 handle_command_reply: tid=0 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.669+0000 7f0167a6c740 1 client.16482 shutdown 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.669+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 msgr2=0x55f1948b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.669+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f01500fb570 tx=0x7f01500c33d0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 shutdown_connections 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f194926740 0x55f194928b70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1948b3fc0 0x55f194924980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1948a7030 0x55f1948b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 --2- 172.21.15.87:0/568013610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f1948a6710 0x55f1948b34f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 >> 172.21.15.87:0/568013610 conn(0x55f1948a0e40 msgr2=0x55f1948a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0145ffb640 0 client.16482 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:43.668 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:43.668 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:34:43.668 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:34:43.668 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:43.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 shutdown_connections 2024-04-22T13:34:43.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 1 -- 172.21.15.87:0/568013610 wait complete. 2024-04-22T13:34:43.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.670+0000 7f0167a6c740 20 client.16482 trim_cache size 0 max 16384 2024-04-22T13:34:43.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.675+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 msgr2=0x7f014c0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.675+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 0x7f014c0a63f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f015800bbc0 tx=0x7f0158005f30 comp rx=0 tx=0).stop 2024-04-22T13:34:43.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.675+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 msgr2=0x7f01601a7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:43.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.675+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0150009be0 tx=0x7f0150003970 comp rx=0 tx=0).stop 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 shutdown_connections 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f014c0a3f30 0x7f014c0a63f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0160110560 0x7f01601abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f016010ebe0 0x7f01601a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 --2- 172.21.15.87:0/2369612885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f016010e030 0x7f01601a7280 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:43.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.681+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 >> 172.21.15.87:0/2369612885 conn(0x7f016006ce30 msgr2=0x7f0160071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:43.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.685+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 shutdown_connections 2024-04-22T13:34:43.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:43.685+0000 7f015f59e640 1 -- 172.21.15.87:0/2369612885 wait complete. 2024-04-22T13:34:43.730 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:34:43.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:34:43.996 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:43 smithi087 ceph-mon[27999]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 727 MiB/s wr, 1.69k op/s 2024-04-22T13:34:44.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.147+0000 7f7028ded640 1 -- 172.21.15.87:0/3568676949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7024110b30 msgr2=0x7f7024109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:44.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.147+0000 7f7028ded640 1 --2- 172.21.15.87:0/3568676949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7024110b30 0x7f7024109490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7018009900 tx=0x7f701802bc60 comp rx=0 tx=0).stop 2024-04-22T13:34:44.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.148+0000 7f7028ded640 1 -- 172.21.15.87:0/3568676949 shutdown_connections 2024-04-22T13:34:44.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.148+0000 7f7028ded640 1 --2- 172.21.15.87:0/3568676949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7024109b70 0x7f702411d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:44.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.148+0000 7f7028ded640 1 --2- 172.21.15.87:0/3568676949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7024110b30 0x7f7024109490 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:44.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.148+0000 7f7028ded640 1 --2- 172.21.15.87:0/3568676949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70241106c0 0x7f7024108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:44.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.148+0000 7f7028ded640 1 -- 172.21.15.87:0/3568676949 >> 172.21.15.87:0/3568676949 conn(0x7f702406e410 msgr2=0x7f70240719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:44.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.149+0000 7f7028ded640 1 -- 172.21.15.87:0/3568676949 shutdown_connections 2024-04-22T13:34:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.154+0000 7f7028ded640 1 -- 172.21.15.87:0/3568676949 wait complete. 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 Processor -- start 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 -- start start 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70241106c0 0x7f702411ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7024110b30 0x7f7024114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7024111810 con 0x7f70241153d0 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7024111690 con 0x7f7024110b30 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.156+0000 7f7028ded640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7024111990 con 0x7f70241106c0 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.158+0000 7f7022d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.158+0000 7f7022d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33722/0 (socket says 172.21.15.87:33722) 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.158+0000 7f7022d9d640 1 -- 172.21.15.87:0/3275027092 learned_addr learned my addr 172.21.15.87:0/3275027092 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f702259c640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70241106c0 0x7f702411ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f7022d9d640 1 -- 172.21.15.87:0/3275027092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70241106c0 msgr2=0x7f702411ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:44.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f7022d9d640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70241106c0 0x7f702411ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:44.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f7022d9d640 1 -- 172.21.15.87:0/3275027092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7024110b30 msgr2=0x7f7024114e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:44.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f7022d9d640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7024110b30 0x7f7024114e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:44.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.159+0000 7f7022d9d640 1 -- 172.21.15.87:0/3275027092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70241bc420 con 0x7f70241153d0 2024-04-22T13:34:44.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.161+0000 7f7022d9d640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f701400e9a0 tx=0x7f701400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.166+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701400cc60 con 0x7f70241153d0 2024-04-22T13:34:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.166+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701400ce00 con 0x7f70241153d0 2024-04-22T13:34:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.166+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014010430 con 0x7f70241153d0 2024-04-22T13:34:44.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.166+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70241bc710 con 0x7f70241153d0 2024-04-22T13:34:44.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.166+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241bcc10 con 0x7f70241153d0 2024-04-22T13:34:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.171+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7004005290 con 0x7f70241153d0 2024-04-22T13:34:44.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:43 smithi120 ceph-mon[31101]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 727 MiB/s wr, 1.69k op/s 2024-04-22T13:34:44.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.179+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70140046c0 con 0x7f70241153d0 2024-04-22T13:34:44.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.181+0000 7f70137fe640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 0x7f700c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:44.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.181+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f70140d0ab0 con 0x7f70241153d0 2024-04-22T13:34:44.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.192+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f701409d160 con 0x7f70241153d0 2024-04-22T13:34:44.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.192+0000 7f702259c640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 0x7f700c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:44.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.198+0000 7f702259c640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 0x7f700c0a63e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f701c005fd0 tx=0x7f701c005f60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:43 smithi132 ceph-mon[31531]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 727 MiB/s wr, 1.69k op/s 2024-04-22T13:34:44.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:44.998+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7004005850 con 0x7f70241153d0 2024-04-22T13:34:45.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.003+0000 7f70137fe640 1 -- 172.21.15.87:0/3275027092 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f701409ca60 con 0x7f70241153d0 2024-04-22T13:34:45.003 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:34:45.004 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:34:45.004 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:34:45.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.029+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 msgr2=0x7f700c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.029+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 0x7f700c0a63e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f701c005fd0 tx=0x7f701c005f60 comp rx=0 tx=0).stop 2024-04-22T13:34:45.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.029+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 msgr2=0x7f7024115880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.029+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f701400e9a0 tx=0x7f701400eda0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 shutdown_connections 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f700c0a3f20 0x7f700c0a63e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70241153d0 0x7f7024115880 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7024110b30 0x7f7024114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 --2- 172.21.15.87:0/3275027092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70241106c0 0x7f702411ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.038+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 >> 172.21.15.87:0/3275027092 conn(0x7f702406e410 msgr2=0x7f702411ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.042+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 shutdown_connections 2024-04-22T13:34:45.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.048+0000 7f7028ded640 1 -- 172.21.15.87:0/3275027092 wait complete. 2024-04-22T13:34:45.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:34:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.523+0000 7f7053a43640 1 -- 172.21.15.87:0/322799154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c1107a0 msgr2=0x7f704c10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.523+0000 7f7053a43640 1 --2- 172.21.15.87:0/322799154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c1107a0 0x7f704c10e290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7044009930 tx=0x7f704402bc20 comp rx=0 tx=0).stop 2024-04-22T13:34:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.526+0000 7f7053a43640 1 -- 172.21.15.87:0/322799154 shutdown_connections 2024-04-22T13:34:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.526+0000 7f7053a43640 1 --2- 172.21.15.87:0/322799154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f704c11b730 0x7f704c11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.526+0000 7f7053a43640 1 --2- 172.21.15.87:0/322799154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c10e7d0 0x7f704c10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.526+0000 7f7053a43640 1 --2- 172.21.15.87:0/322799154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c1107a0 0x7f704c10e290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.526+0000 7f7053a43640 1 -- 172.21.15.87:0/322799154 >> 172.21.15.87:0/322799154 conn(0x7f704c06d060 msgr2=0x7f704c06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.528+0000 7f7053a43640 1 -- 172.21.15.87:0/322799154 shutdown_connections 2024-04-22T13:34:45.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.528+0000 7f7053a43640 1 -- 172.21.15.87:0/322799154 wait complete. 2024-04-22T13:34:45.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 Processor -- start 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 -- start start 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f704c10e7d0 0x7f704c1a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c11b730 0x7f704c1abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f704c120fb0 con 0x7f704c1107a0 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f704c120e30 con 0x7f704c11b730 2024-04-22T13:34:45.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.530+0000 7f7053a43640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f704c121130 con 0x7f704c10e7d0 2024-04-22T13:34:45.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.531+0000 7f7050fde640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.531+0000 7f7050fde640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33738/0 (socket says 172.21.15.87:33738) 2024-04-22T13:34:45.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.531+0000 7f7050fde640 1 -- 172.21.15.87:0/2590726265 learned_addr learned my addr 172.21.15.87:0/2590726265 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.532+0000 7f7050fde640 1 -- 172.21.15.87:0/2590726265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f704c10e7d0 msgr2=0x7f704c1a7590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.532+0000 7f7050fde640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f704c10e7d0 0x7f704c1a7590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.532+0000 7f7050fde640 1 -- 172.21.15.87:0/2590726265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c11b730 msgr2=0x7f704c1abe50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.532+0000 7f7050fde640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c11b730 0x7f704c1abe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.532+0000 7f7050fde640 1 -- 172.21.15.87:0/2590726265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c1ac5d0 con 0x7f704c1107a0 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.533+0000 7f7050fde640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f703c00ebc0 tx=0x7f703c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.534+0000 7f704a7fc640 1 -- 172.21.15.87:0/2590726265 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c010630 con 0x7f704c1107a0 2024-04-22T13:34:45.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.534+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704c1ac8c0 con 0x7f704c1107a0 2024-04-22T13:34:45.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.534+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704c110f10 con 0x7f704c1107a0 2024-04-22T13:34:45.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.540+0000 7f7056639740 1 Processor -- start 2024-04-22T13:34:45.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.541+0000 7f7056639740 1 -- start start 2024-04-22T13:34:45.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.541+0000 7f7056639740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315ba4d0 0x5642315ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f7056639740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f7056639740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315c3a00 0x5642315c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f7056639740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56423151f1f0 con 0x5642315c3a00 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f7056639740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642314bc4b0 con 0x5642315ba4d0 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f7056639740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56423151b1b0 con 0x5642315bae80 2024-04-22T13:34:45.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.542+0000 7f704a7fc640 1 -- 172.21.15.87:0/2590726265 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f703c010c80 con 0x7f704c1107a0 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f70517df640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f70517df640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34320/0 (socket says 172.21.15.87:34320) 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f70517df640 1 -- 172.21.15.87:0/2583162578 learned_addr learned my addr 172.21.15.87:0/2583162578 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f7051fe0640 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315ba4d0 0x5642315ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f7050fde640 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315c3a00 0x5642315c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.544+0000 7f704a7fc640 1 -- 172.21.15.87:0/2590726265 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c026060 con 0x7f704c1107a0 2024-04-22T13:34:45.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.545+0000 7f70517df640 1 -- 172.21.15.87:0/2583162578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315ba4d0 msgr2=0x5642315ba8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.545+0000 7f70517df640 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315ba4d0 0x5642315ba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.545+0000 7f70517df640 1 -- 172.21.15.87:0/2583162578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315c3a00 msgr2=0x5642315c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.545+0000 7f70517df640 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315c3a00 0x5642315c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.545+0000 7f70517df640 1 -- 172.21.15.87:0/2583162578 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642314ae1b0 con 0x5642315bae80 2024-04-22T13:34:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.549+0000 7f70517df640 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7044009230 tx=0x7f70440056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc1184d75436c45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.551+0000 7f704a7fc640 1 -- 172.21.15.87:0/2590726265 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f703c0107d0 con 0x7f704c1107a0 2024-04-22T13:34:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.552+0000 7f70337fe640 1 -- 172.21.15.87:0/2583162578 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044042020 con 0x5642315bae80 2024-04-22T13:34:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.552+0000 7f70337fe640 1 -- 172.21.15.87:0/2583162578 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7044009520 con 0x5642315bae80 2024-04-22T13:34:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.553+0000 7f7056639740 1 -- 172.21.15.87:0/2583162578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 msgr2=0x5642315c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.553+0000 7f7056639740 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7044009230 tx=0x7f70440056d0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.553+0000 7f704a7fc640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.554+0000 7f7051fe0640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.555+0000 7f704a7fc640 1 -- 172.21.15.87:0/2590726265 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f703c0d12f0 con 0x7f704c1107a0 2024-04-22T13:34:45.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.558+0000 7f7051fe0640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7040005d80 tx=0x7f7040005c90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.560+0000 7f7056639740 1 -- 172.21.15.87:0/2583162578 shutdown_connections 2024-04-22T13:34:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.560+0000 7f7056639740 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315c3a00 0x5642315c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.560+0000 7f7056639740 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x5642315c3430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.560+0000 7f7056639740 1 --2- 172.21.15.87:0/2583162578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315ba4d0 0x5642315ba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.560+0000 7f7056639740 1 -- 172.21.15.87:0/2583162578 >> 172.21.15.87:0/2583162578 conn(0x5642315b4ba0 msgr2=0x5642315b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.563+0000 7f7056639740 1 -- 172.21.15.87:0/2583162578 shutdown_connections 2024-04-22T13:34:45.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.565+0000 7f7056639740 1 -- 172.21.15.87:0/2583162578 wait complete. 2024-04-22T13:34:45.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.566+0000 7f7056639740 1 -- start start 2024-04-22T13:34:45.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.566+0000 7f7056639740 1 Processor -- start 2024-04-22T13:34:45.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.567+0000 7f7056639740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.567+0000 7f7056639740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.568+0000 7f7056639740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 0x5642315c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.568+0000 7f7056639740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5642315c6960 con 0x5642315ba4d0 2024-04-22T13:34:45.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.568+0000 7f7050fde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.568+0000 7f7050fde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54698/0 (socket says 172.21.15.87:54698) 2024-04-22T13:34:45.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.568+0000 7f7050fde640 1 -- 172.21.15.87:0/3790365953 learned_addr learned my addr 172.21.15.87:0/3790365953 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:45.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.569+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642314d8500 con 0x5642315bae80 2024-04-22T13:34:45.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.569+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5642315c6ae0 con 0x5642315c3a00 2024-04-22T13:34:45.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.569+0000 7f7051fe0640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 0x5642315c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.569+0000 7f7056639740 10 client.0 _ll_register_callbacks cb 0x5642314a6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f70517df640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f7051fe0640 1 -- 172.21.15.87:0/3790365953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 msgr2=0x5642315c0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f7051fe0640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f7051fe0640 1 -- 172.21.15.87:0/3790365953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 msgr2=0x5642315c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f7051fe0640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.570+0000 7f7051fe0640 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642312313f0 con 0x5642315c3a00 2024-04-22T13:34:45.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.573+0000 7f7051fe0640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 0x5642315c0ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f704000e6c0 tx=0x7f70400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ed1555b0732ed67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.573+0000 7f7030ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.573+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040022030 con 0x5642315c3a00 2024-04-22T13:34:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.574+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7044009520 con 0x5642315c3a00 2024-04-22T13:34:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.574+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7040004020 con 0x5642315c3a00 2024-04-22T13:34:45.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.578+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f704002c050 con 0x5642315c3a00 2024-04-22T13:34:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.582+0000 7f7056639740 10 client.25768 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.582+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5642312fb7d0 con 0x5642315c3a00 2024-04-22T13:34:45.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.584+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7044009520 con 0x5642315c3a00 2024-04-22T13:34:45.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.584+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f704000f040 con 0x5642315c3a00 2024-04-22T13:34:45.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.585+0000 7f7056639740 10 client.25768 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.585+0000 7f7056639740 10 client.25768 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:34:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.586+0000 7f7056639740 20 client.25768 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:45.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.587+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315d3cb0 0x5642315d60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.587+0000 7f7056639740 4 client.25768 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:45.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.587+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642314be930 con 0x5642315d3cb0 2024-04-22T13:34:45.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.589+0000 7f70517df640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315d3cb0 0x5642315d60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.595+0000 7f70517df640 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315d3cb0 0x5642315d60a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c70e274cc836de48 server_cookie=7821f4b91a0d2bf2 in_seq=0 out_seq=0 2024-04-22T13:34:45.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.595+0000 7f7030ff9640 10 client.25768 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:45.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.599+0000 7f7030ff9640 1 -- 172.21.15.87:0/3790365953 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642314be930 con 0x5642315d3cb0 2024-04-22T13:34:45.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.599+0000 7f7030ff9640 10 client.25768 handle_command_reply: tid=0 2024-04-22T13:34:45.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.602+0000 7f7056639740 1 client.25768 shutdown 2024-04-22T13:34:45.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.604+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 msgr2=0x5642315c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 0x5642315c0ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f704000e6c0 tx=0x7f70400046c0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 shutdown_connections 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315d3cb0 0x5642315d60a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315c3a00 0x5642315c0ad0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 --2- 172.21.15.87:0/3790365953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.605+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 >> 172.21.15.87:0/3790365953 conn(0x5642315b4ba0 msgr2=0x5642315b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.608+0000 7f7030ff9640 0 client.25768 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:45.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.608+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 shutdown_connections 2024-04-22T13:34:45.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.608+0000 7f7056639740 1 -- 172.21.15.87:0/3790365953 wait complete. 2024-04-22T13:34:45.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.608+0000 7f7056639740 20 client.25768 trim_cache size 0 max 16384 2024-04-22T13:34:45.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.651+0000 7f7056639740 1 Processor -- start 2024-04-22T13:34:45.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 -- start start 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315b4ba0 0x5642315cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7038010ff0 con 0x5642315ba4d0 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f703800e8c0 con 0x5642315bae80 2024-04-22T13:34:45.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.653+0000 7f7056639740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f703800c710 con 0x5642315b4ba0 2024-04-22T13:34:45.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.655+0000 7f7051fe0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.656+0000 7f70517df640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315b4ba0 0x5642315cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.656+0000 7f7050fde640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.656+0000 7f7050fde640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33754/0 (socket says 172.21.15.87:33754) 2024-04-22T13:34:45.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.656+0000 7f7050fde640 1 -- 172.21.15.87:0/1235178189 learned_addr learned my addr 172.21.15.87:0/1235178189 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.657+0000 7f7050fde640 1 -- 172.21.15.87:0/1235178189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315b4ba0 msgr2=0x5642315cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.657+0000 7f7050fde640 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315b4ba0 0x5642315cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.657+0000 7f7050fde640 1 -- 172.21.15.87:0/1235178189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 msgr2=0x5642315c2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.657+0000 7f7050fde640 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.657+0000 7f7050fde640 1 -- 172.21.15.87:0/1235178189 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642315bac20 con 0x5642315ba4d0 2024-04-22T13:34:45.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.658+0000 7f7050fde640 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f703c09d230 tx=0x7f703c009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8705e7198cf92fea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.658+0000 7f7032ffd640 1 -- 172.21.15.87:0/1235178189 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c0b6070 con 0x5642315ba4d0 2024-04-22T13:34:45.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.659+0000 7f7032ffd640 1 -- 172.21.15.87:0/1235178189 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f703c0a1650 con 0x5642315ba4d0 2024-04-22T13:34:45.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.659+0000 7f7032ffd640 1 -- 172.21.15.87:0/1235178189 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c099a80 con 0x5642315ba4d0 2024-04-22T13:34:45.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.660+0000 7f7056639740 1 -- 172.21.15.87:0/1235178189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 msgr2=0x5642315b66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.660+0000 7f7056639740 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f703c09d230 tx=0x7f703c009200 comp rx=0 tx=0).stop 2024-04-22T13:34:45.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 -- 172.21.15.87:0/1235178189 shutdown_connections 2024-04-22T13:34:45.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315bae80 0x5642315c2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315b66c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 --2- 172.21.15.87:0/1235178189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315b4ba0 0x5642315cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 -- 172.21.15.87:0/1235178189 >> 172.21.15.87:0/1235178189 conn(0x5642315c0c30 msgr2=0x5642315b5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.661+0000 7f7056639740 1 -- 172.21.15.87:0/1235178189 shutdown_connections 2024-04-22T13:34:45.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.662+0000 7f7056639740 1 -- 172.21.15.87:0/1235178189 wait complete. 2024-04-22T13:34:45.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.662+0000 7f7056639740 1 -- start start 2024-04-22T13:34:45.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.662+0000 7f7056639740 1 Processor -- start 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x564231639220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f703800cd80 con 0x5642315ba4d0 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f703800e1f0 con 0x5642315b4ba0 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f703800d450 con 0x5642315bae80 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.663+0000 7f7056639740 10 client.0 _ll_register_callbacks cb 0x7f704000ede0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.664+0000 7f70517df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.664+0000 7f70517df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54708/0 (socket says 172.21.15.87:54708) 2024-04-22T13:34:45.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.664+0000 7f70517df640 1 -- 172.21.15.87:0/4056914736 learned_addr learned my addr 172.21.15.87:0/4056914736 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:45.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.664+0000 7f7051fe0640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x564231639220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 -- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 msgr2=0x564231639220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x564231639220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 -- 172.21.15.87:0/4056914736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 msgr2=0x5642315c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:45.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 -- 172.21.15.87:0/4056914736 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564231639760 con 0x5642315b4ba0 2024-04-22T13:34:45.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.665+0000 7f70517df640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7044005ab0 tx=0x7f704402bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1208e02a4146dff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:45.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.668+0000 7f7030ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:45.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.668+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044056060 con 0x5642315b4ba0 2024-04-22T13:34:45.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.668+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f704402c7a0 con 0x5642315b4ba0 2024-04-22T13:34:45.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.668+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f703c0a1650 con 0x5642315b4ba0 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.668+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7044042040 con 0x5642315b4ba0 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7056639740 10 client.25757 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56423163aa60 con 0x5642315b4ba0 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f703c0a1650 con 0x5642315b4ba0 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7044059020 con 0x5642315b4ba0 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7056639740 10 client.25757 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:45.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7056639740 10 client.25757 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:34:45.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.669+0000 7f7056639740 20 client.25757 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:45.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.670+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315c3a00 0x56423163cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:45.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.670+0000 7f7056639740 4 client.25757 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:45.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.670+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56423152cc10 con 0x5642315c3a00 2024-04-22T13:34:45.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.672+0000 7f7050fde640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315c3a00 0x56423163cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.676+0000 7f7050fde640 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315c3a00 0x56423163cdd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=270991afae519692 server_cookie=8204c73d87586130 in_seq=0 out_seq=0 2024-04-22T13:34:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.676+0000 7f7030ff9640 10 client.25757 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.677+0000 7f7030ff9640 1 -- 172.21.15.87:0/4056914736 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56423152cc10 con 0x5642315c3a00 2024-04-22T13:34:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.677+0000 7f7030ff9640 10 client.25757 handle_command_reply: tid=0 2024-04-22T13:34:45.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.677+0000 7f7056639740 1 client.25757 shutdown 2024-04-22T13:34:45.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.681+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 msgr2=0x5642315c7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.681+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7044005ab0 tx=0x7f704402bc20 comp rx=0 tx=0).stop 2024-04-22T13:34:45.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 shutdown_connections 2024-04-22T13:34:45.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5642315c3a00 0x56423163cdd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642315bae80 0x564231639220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642315ba4d0 0x5642315c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 --2- 172.21.15.87:0/4056914736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642315b4ba0 0x5642315c7f00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 >> 172.21.15.87:0/4056914736 conn(0x5642315c0c30 msgr2=0x5642315b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.682+0000 7f7030ff9640 0 client.25757 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:34:45.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.683+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 shutdown_connections 2024-04-22T13:34:45.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.683+0000 7f7056639740 1 -- 172.21.15.87:0/4056914736 wait complete. 2024-04-22T13:34:45.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.684+0000 7f7056639740 20 client.25757 trim_cache size 0 max 16384 2024-04-22T13:34:45.682 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:45.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.694+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 msgr2=0x7f70340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.694+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7040005d80 tx=0x7f7040005c90 comp rx=0 tx=0).stop 2024-04-22T13:34:45.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.694+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 msgr2=0x7f704c1a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:45.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.694+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f703c00ebc0 tx=0x7f703c00c6a0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 shutdown_connections 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f704c11b730 0x7f704c1abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f704c1107a0 0x7f704c1a7ad0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 --2- 172.21.15.87:0/2590726265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f704c10e7d0 0x7f704c1a7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.695+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 >> 172.21.15.87:0/2590726265 conn(0x7f704c06d060 msgr2=0x7f704c11a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.699+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 shutdown_connections 2024-04-22T13:34:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:45.702+0000 7f7053a43640 1 -- 172.21.15.87:0/2590726265 wait complete. 2024-04-22T13:34:45.751 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:34:46.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:45 smithi120 ceph-mon[31101]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 GiB/s wr, 2.54k op/s 2024-04-22T13:34:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:45 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3275027092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:46.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.198+0000 7f1adaffd640 1 -- 172.21.15.87:0/542359761 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4032820 con 0x7f1adc1150a0 2024-04-22T13:34:46.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.199+0000 7f1ae2915640 1 -- 172.21.15.87:0/542359761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 msgr2=0x7f1adc117490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.199+0000 7f1ae2915640 1 --2- 172.21.15.87:0/542359761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc117490 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009cd0 tx=0x7f1ad402c120 comp rx=0 tx=0).stop 2024-04-22T13:34:46.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.201+0000 7f1ae2915640 1 -- 172.21.15.87:0/542359761 shutdown_connections 2024-04-22T13:34:46.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.201+0000 7f1ae2915640 1 --2- 172.21.15.87:0/542359761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc117490 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.201+0000 7f1ae2915640 1 --2- 172.21.15.87:0/542359761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1adc109870 0x7f1adc114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.201+0000 7f1ae2915640 1 --2- 172.21.15.87:0/542359761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1adc108f50 0x7f1adc109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.201+0000 7f1ae2915640 1 -- 172.21.15.87:0/542359761 >> 172.21.15.87:0/542359761 conn(0x7f1adc06d4c0 msgr2=0x7f1adc06d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.203+0000 7f1ae2915640 1 -- 172.21.15.87:0/542359761 shutdown_connections 2024-04-22T13:34:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.203+0000 7f1ae2915640 1 -- 172.21.15.87:0/542359761 wait complete. 2024-04-22T13:34:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.205+0000 7f1ae2915640 1 Processor -- start 2024-04-22T13:34:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.205+0000 7f1ae2915640 1 -- start start 2024-04-22T13:34:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1adc108f50 0x7f1adc1146e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1adc109870 0x7f1adc10be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc11aa50 con 0x7f1adc1150a0 2024-04-22T13:34:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc11a8d0 con 0x7f1adc109870 2024-04-22T13:34:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae2915640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc11abd0 con 0x7f1adc108f50 2024-04-22T13:34:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.206+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33768/0 (socket says 172.21.15.87:33768) 2024-04-22T13:34:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/858935466 learned_addr learned my addr 172.21.15.87:0/858935466 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/858935466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1adc108f50 msgr2=0x7f1adc1146e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1adc108f50 0x7f1adc1146e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/858935466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1adc109870 msgr2=0x7f1adc10be90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1adc109870 0x7f1adc10be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.207+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/858935466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc10ca10 con 0x7f1adc1150a0 2024-04-22T13:34:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.208+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4005cc0 tx=0x7f1ad4034d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.209+0000 7f1ad97fa640 1 -- 172.21.15.87:0/858935466 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4041020 con 0x7f1adc1150a0 2024-04-22T13:34:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.209+0000 7f1ad97fa640 1 -- 172.21.15.87:0/858935466 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad403d3f0 con 0x7f1adc1150a0 2024-04-22T13:34:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.209+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1adc10cca0 con 0x7f1adc1150a0 2024-04-22T13:34:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.209+0000 7f1ad97fa640 1 -- 172.21.15.87:0/858935466 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4039040 con 0x7f1adc1150a0 2024-04-22T13:34:46.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.209+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1adc110530 con 0x7f1adc1150a0 2024-04-22T13:34:46.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.214+0000 7f1ad97fa640 1 -- 172.21.15.87:0/858935466 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ad4044020 con 0x7f1adc1150a0 2024-04-22T13:34:46.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.214+0000 7f1ae55e5740 1 Processor -- start 2024-04-22T13:34:46.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:45 smithi132 ceph-mon[31531]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 GiB/s wr, 2.54k op/s 2024-04-22T13:34:46.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:45 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3275027092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:46.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.215+0000 7f1ad97fa640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 0x7f1ac40a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.216+0000 7f1ad97fa640 1 -- 172.21.15.87:0/858935466 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1ad40f2660 con 0x7f1adc1150a0 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 -- start start 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7ea4d0 0x555a0e7ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555a0e74f1f0 con 0x555a0e7f3a00 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555a0e6ec4b0 con 0x555a0e7eae80 2024-04-22T13:34:46.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.217+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a0e74b1b0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.219+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 0x7f1ac40a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.220+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.220+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33780/0 (socket says 172.21.15.87:33780) 2024-04-22T13:34:46.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.220+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/2019538142 learned_addr learned my addr 172.21.15.87:0/2019538142 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.221+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 0x7f1ac40a62e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0006fd0 tx=0x7f1ad0009b20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.222+0000 7f1adbfff640 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7ea4d0 0x555a0e7ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.223+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.223+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/2019538142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7ea4d0 msgr2=0x555a0e7ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.224+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7ea4d0 0x555a0e7ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.224+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/2019538142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 msgr2=0x555a0e7f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.224+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.224+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/2019538142 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0e6de1b0 con 0x555a0e7f3a00 2024-04-22T13:34:46.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.225+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40c6e60 tx=0x7f1ad40be080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68bfcd406daed6e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.226+0000 7f1ac27fc640 1 -- 172.21.15.87:0/2019538142 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad40bee20 con 0x555a0e7f3a00 2024-04-22T13:34:46.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.226+0000 7f1ac27fc640 1 -- 172.21.15.87:0/2019538142 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad40a90e0 con 0x555a0e7f3a00 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 -- 172.21.15.87:0/2019538142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 msgr2=0x555a0e7f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40c6e60 tx=0x7f1ad40be080 comp rx=0 tx=0).stop 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 -- 172.21.15.87:0/2019538142 shutdown_connections 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f5e50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/2019538142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7ea4d0 0x555a0e7ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.228+0000 7f1ae55e5740 1 -- 172.21.15.87:0/2019538142 >> 172.21.15.87:0/2019538142 conn(0x555a0e7e4ba0 msgr2=0x555a0e7e4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.231+0000 7f1ae55e5740 1 -- 172.21.15.87:0/2019538142 shutdown_connections 2024-04-22T13:34:46.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.232+0000 7f1ae55e5740 1 -- 172.21.15.87:0/2019538142 wait complete. 2024-04-22T13:34:46.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.232+0000 7f1ae55e5740 1 -- start start 2024-04-22T13:34:46.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.232+0000 7f1ae55e5740 1 Processor -- start 2024-04-22T13:34:46.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.233+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 0x555a0e7f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.233+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7f0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.233+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.233+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555a0e7f6730 con 0x555a0e7f3a00 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.234+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555a0e708500 con 0x555a0e7ea4d0 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.234+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a0e7f68b0 con 0x555a0e7eae80 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.234+0000 7f1ae55e5740 10 client.0 _ll_register_callbacks cb 0x555a0e6d6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.235+0000 7f1adb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.235+0000 7f1adbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 0x555a0e7f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.236+0000 7f1adb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7f0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34398/0 (socket says 172.21.15.87:34398) 2024-04-22T13:34:46.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.236+0000 7f1adb7fe640 1 -- 172.21.15.87:0/390480199 learned_addr learned my addr 172.21.15.87:0/390480199 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.237+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 -- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 msgr2=0x555a0e7f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 -- 172.21.15.87:0/390480199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 msgr2=0x555a0e7f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 -- 172.21.15.87:0/390480199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a0e4613f0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.238+0000 7f1adbfff640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 0x555a0e7f32d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00bf50 tx=0x7f1acc00d5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2abeb02def472671 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.239+0000 7f1abb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:46.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.239+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc014430 con 0x555a0e7ea4d0 2024-04-22T13:34:46.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.239+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad40a90e0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.239+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1acc0145d0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.240+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1acc022050 con 0x555a0e7ea4d0 2024-04-22T13:34:46.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.240+0000 7f1ae55e5740 10 client.25763 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:46.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.240+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555a0e52b7d0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:45 smithi087 ceph-mon[27999]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 GiB/s wr, 2.54k op/s 2024-04-22T13:34:46.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:45 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3275027092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:34:46.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.245+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad40a90e0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.246+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1acc013070 con 0x555a0e7ea4d0 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 10 client.25763 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 10 client.25763 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 20 client.25763 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e803750 0x555a0e805ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 4 client.25763 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.247+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a0e6ee930 con 0x555a0e803750 2024-04-22T13:34:46.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.248+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e803750 0x555a0e805ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.251+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e803750 0x555a0e805ba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa741508ac5429fd server_cookie=ccb4a0b91a7882f1 in_seq=0 out_seq=0 2024-04-22T13:34:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.251+0000 7f1abb7fe640 10 client.25763 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.252+0000 7f1abb7fe640 1 -- 172.21.15.87:0/390480199 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a0e6ee930 con 0x555a0e803750 2024-04-22T13:34:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.252+0000 7f1abb7fe640 10 client.25763 handle_command_reply: tid=0 2024-04-22T13:34:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.253+0000 7f1ae55e5740 1 client.25763 shutdown 2024-04-22T13:34:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.255+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 msgr2=0x555a0e7f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.255+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 0x555a0e7f32d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00bf50 tx=0x7f1acc00d5f0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 shutdown_connections 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e803750 0x555a0e805ba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7f3a00 0x555a0e7f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/390480199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7ea4d0 0x555a0e7f32d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 >> 172.21.15.87:0/390480199 conn(0x555a0e7e4ba0 msgr2=0x555a0e7e68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1abb7fe640 0 client.25763 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 shutdown_connections 2024-04-22T13:34:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.256+0000 7f1ae55e5740 1 -- 172.21.15.87:0/390480199 wait complete. 2024-04-22T13:34:46.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.257+0000 7f1ae55e5740 20 client.25763 trim_cache size 0 max 16384 2024-04-22T13:34:46.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.284+0000 7f1ae55e5740 1 Processor -- start 2024-04-22T13:34:46.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.284+0000 7f1ae55e5740 1 -- start start 2024-04-22T13:34:46.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c7910 con 0x555a0e7ea4d0 2024-04-22T13:34:46.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c51e0 con 0x555a0e7eae80 2024-04-22T13:34:46.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.285+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c30e0 con 0x555a0e7f3a00 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.286+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.286+0000 7f1ae0eb2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34400/0 (socket says 172.21.15.87:34400) 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.286+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/1714012112 learned_addr learned my addr 172.21.15.87:0/1714012112 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/1714012112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 msgr2=0x555a0e7f2d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f2d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/1714012112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 msgr2=0x555a0e7e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 -- 172.21.15.87:0/1714012112 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0e802350 con 0x555a0e7f3a00 2024-04-22T13:34:46.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.287+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40bb0f0 tx=0x7f1ad402cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673d94c15aed74ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.288+0000 7f1ac1ffb640 1 -- 172.21.15.87:0/1714012112 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad40bee20 con 0x555a0e7f3a00 2024-04-22T13:34:46.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.289+0000 7f1ac1ffb640 1 -- 172.21.15.87:0/1714012112 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad40a90b0 con 0x555a0e7f3a00 2024-04-22T13:34:46.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.289+0000 7f1ac1ffb640 1 -- 172.21.15.87:0/1714012112 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad403d8a0 con 0x555a0e7f3a00 2024-04-22T13:34:46.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.292+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1714012112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 msgr2=0x555a0e7f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.292+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40bb0f0 tx=0x7f1ad402cee0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.292+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1714012112 shutdown_connections 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.293+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7f3a00 0x555a0e7f32d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.293+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7eae80 0x555a0e7f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.293+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1714012112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7e6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.293+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1714012112 >> 172.21.15.87:0/1714012112 conn(0x555a0e7e4ba0 msgr2=0x555a0e7e5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.293+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1714012112 shutdown_connections 2024-04-22T13:34:46.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.294+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1714012112 wait complete. 2024-04-22T13:34:46.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.294+0000 7f1ae55e5740 1 -- start start 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.294+0000 7f1ae55e5740 1 Processor -- start 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7fbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 0x555a0e7ed640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c36a0 con 0x555a0e7ea4d0 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c4b10 con 0x555a0e7f3a00 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1c3d70 con 0x555a0e7eae80 2024-04-22T13:34:46.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae55e5740 10 client.0 _ll_register_callbacks cb 0x7f1acc00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1adbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7fbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1adb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1adbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7fbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33810/0 (socket says 172.21.15.87:33810) 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1adb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34402/0 (socket says 172.21.15.87:34402) 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1adbfff640 1 -- 172.21.15.87:0/1192873195 learned_addr learned my addr 172.21.15.87:0/1192873195 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.295+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 0x555a0e7ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1adb7fe640 1 -- 172.21.15.87:0/1192873195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 msgr2=0x555a0e7ed640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 0x555a0e7ed640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1adb7fe640 1 -- 172.21.15.87:0/1192873195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 msgr2=0x555a0e7fbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7fbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1adb7fe640 1 -- 172.21.15.87:0/1192873195 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a0e7ede70 con 0x555a0e7eae80 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.296+0000 7f1ae0eb2640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 0x555a0e7ed640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.297+0000 7f1adb7fe640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x555a0e8063e0 tx=0x7f1ad0004820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eecfd200e08cfe63 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.297+0000 7f1abb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:46.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.297+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000f040 con 0x555a0e7eae80 2024-04-22T13:34:46.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.297+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f1ad0024500 con 0x555a0e7eae80 2024-04-22T13:34:46.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.298+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad40a90b0 con 0x555a0e7eae80 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.298+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ad0024740 con 0x555a0e7eae80 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.298+0000 7f1ae55e5740 10 client.25786 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.298+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555a0e869db0 con 0x555a0e7eae80 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad40a90b0 con 0x555a0e7eae80 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1ad002e450 con 0x555a0e7eae80 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 10 client.25786 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 10 client.25786 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 20 client.25786 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e86a530 0x555a0e86c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 4 client.25786 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:46.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.299+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a0e75cc10 con 0x555a0e86a530 2024-04-22T13:34:46.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.304+0000 7f1adbfff640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e86a530 0x555a0e86c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.345+0000 7f1adbfff640 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e86a530 0x555a0e86c950 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cbe8464ccd20d942 server_cookie=bc0eea7bfe5009ef in_seq=0 out_seq=0 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1abb7fe640 10 client.25786 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1abb7fe640 1 -- 172.21.15.87:0/1192873195 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a0e75cc10 con 0x555a0e86a530 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1abb7fe640 10 client.25786 handle_command_reply: tid=0 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1ae55e5740 1 client.25786 shutdown 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 msgr2=0x555a0e7ed0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.346+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x555a0e8063e0 tx=0x7f1ad0004820 comp rx=0 tx=0).stop 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 shutdown_connections 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x555a0e86a530 0x555a0e86c950 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a0e7f3a00 0x555a0e7ed640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a0e7eae80 0x555a0e7ed0a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 --2- 172.21.15.87:0/1192873195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555a0e7ea4d0 0x555a0e7fbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 >> 172.21.15.87:0/1192873195 conn(0x555a0e7e4ba0 msgr2=0x555a0e7e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.347+0000 7f1abb7fe640 0 client.25786 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:34:46.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.350+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 shutdown_connections 2024-04-22T13:34:46.352 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:46.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.352+0000 7f1ae55e5740 1 -- 172.21.15.87:0/1192873195 wait complete. 2024-04-22T13:34:46.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.353+0000 7f1ae55e5740 20 client.25786 trim_cache size 0 max 16384 2024-04-22T13:34:46.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 msgr2=0x7f1ac40a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 0x7f1ac40a62e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0006fd0 tx=0x7f1ad0009b20 comp rx=0 tx=0).stop 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 msgr2=0x7f1adc10c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4005cc0 tx=0x7f1ad4034d50 comp rx=0 tx=0).stop 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 shutdown_connections 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1ac40a3e20 0x7f1ac40a62e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1adc1150a0 0x7f1adc10c3d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1adc109870 0x7f1adc10be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 --2- 172.21.15.87:0/858935466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1adc108f50 0x7f1adc1146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.399+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 >> 172.21.15.87:0/858935466 conn(0x7f1adc06d4c0 msgr2=0x7f1adc10a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.400+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 shutdown_connections 2024-04-22T13:34:46.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.400+0000 7f1ae2915640 1 -- 172.21.15.87:0/858935466 wait complete. 2024-04-22T13:34:46.463 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:34:46.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.918+0000 7f7a5d992640 1 -- 172.21.15.87:0/176178422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581061c0 msgr2=0x7f7a58106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.918+0000 7f7a5d992640 1 --2- 172.21.15.87:0/176178422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581061c0 0x7f7a58106650 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009860 tx=0x7f7a4802bc80 comp rx=0 tx=0).stop 2024-04-22T13:34:46.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.923+0000 7f7a5d992640 1 -- 172.21.15.87:0/176178422 shutdown_connections 2024-04-22T13:34:46.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.923+0000 7f7a5d992640 1 --2- 172.21.15.87:0/176178422 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581061c0 0x7f7a58106650 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.923+0000 7f7a5d992640 1 --2- 172.21.15.87:0/176178422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a581016e0 0x7f7a58105ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.923+0000 7f7a5d992640 1 --2- 172.21.15.87:0/176178422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a58100dc0 0x7f7a581011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.923+0000 7f7a5d992640 1 -- 172.21.15.87:0/176178422 >> 172.21.15.87:0/176178422 conn(0x7f7a580fc7d0 msgr2=0x7f7a580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.927+0000 7f7a5d992640 1 -- 172.21.15.87:0/176178422 shutdown_connections 2024-04-22T13:34:46.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.928+0000 7f7a5d992640 1 -- 172.21.15.87:0/176178422 wait complete. 2024-04-22T13:34:46.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.929+0000 7f7a5d992640 1 Processor -- start 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 -- start start 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a58100dc0 0x7f7a58114fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a581061c0 0x7f7a5810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58118710 con 0x7f7a581016e0 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58118590 con 0x7f7a58100dc0 2024-04-22T13:34:46.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.930+0000 7f7a5d992640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58118890 con 0x7f7a581061c0 2024-04-22T13:34:46.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.931+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.931+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33822/0 (socket says 172.21.15.87:33822) 2024-04-22T13:34:46.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.931+0000 7f7a57fff640 1 -- 172.21.15.87:0/1161994644 learned_addr learned my addr 172.21.15.87:0/1161994644 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.932+0000 7f7a57fff640 1 -- 172.21.15.87:0/1161994644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a581061c0 msgr2=0x7f7a5810e5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:46.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.933+0000 7f7a57fff640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a581061c0 0x7f7a5810e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.933+0000 7f7a57fff640 1 -- 172.21.15.87:0/1161994644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a58100dc0 msgr2=0x7f7a58114fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.933+0000 7f7a57fff640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a58100dc0 0x7f7a58114fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.933+0000 7f7a57fff640 1 -- 172.21.15.87:0/1161994644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5810eb90 con 0x7f7a581016e0 2024-04-22T13:34:46.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.933+0000 7f7a57fff640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000eea0 tx=0x7f7a5000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.934+0000 7f7a55ffb640 1 -- 172.21.15.87:0/1161994644 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5001b070 con 0x7f7a581016e0 2024-04-22T13:34:46.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.935+0000 7f7a55ffb640 1 -- 172.21.15.87:0/1161994644 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a5000ba20 con 0x7f7a581016e0 2024-04-22T13:34:46.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.935+0000 7f7a55ffb640 1 -- 172.21.15.87:0/1161994644 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50016af0 con 0x7f7a581016e0 2024-04-22T13:34:46.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.935+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5810ee80 con 0x7f7a581016e0 2024-04-22T13:34:46.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.935+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a58112e80 con 0x7f7a581016e0 2024-04-22T13:34:46.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.944+0000 7f7a617dd740 1 Processor -- start 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 -- start start 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9fa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e9561f0 con 0x55bc4e9fac00 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e8f34b0 con 0x55bc4e9f16d0 2024-04-22T13:34:46.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.950+0000 7f7a617dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e9521b0 con 0x55bc4e9f2080 2024-04-22T13:34:46.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.951+0000 7f7a55ffb640 1 -- 172.21.15.87:0/1161994644 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7a5001e070 con 0x7f7a581016e0 2024-04-22T13:34:46.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.953+0000 7f7a55ffb640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 0x7f7a400a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.956+0000 7f7a5c990640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9fa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.956+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.956+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33826/0 (socket says 172.21.15.87:33826) 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.956+0000 7f7a57fff640 1 -- 172.21.15.87:0/2554890440 learned_addr learned my addr 172.21.15.87:0/2554890440 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.956+0000 7f7a55ffb640 1 -- 172.21.15.87:0/1161994644 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7a5009a550 con 0x7f7a581016e0 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.958+0000 7f7a57fff640 1 -- 172.21.15.87:0/2554890440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 msgr2=0x55bc4e9fa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.958+0000 7f7a57fff640 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9fa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.958+0000 7f7a57fff640 1 -- 172.21.15.87:0/2554890440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 msgr2=0x55bc4e9f1ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.958+0000 7f7a57fff640 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9f1ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.958+0000 7f7a57fff640 1 -- 172.21.15.87:0/2554890440 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4e8e51b0 con 0x55bc4e9fac00 2024-04-22T13:34:46.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.959+0000 7f7a57fff640 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7a50000f80 tx=0x7f7a5009c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca873f19c09dd32 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.961+0000 7f7a3effd640 1 -- 172.21.15.87:0/2554890440 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a500b4070 con 0x55bc4e9fac00 2024-04-22T13:34:46.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.961+0000 7f7a3effd640 1 -- 172.21.15.87:0/2554890440 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a5009c4b0 con 0x55bc4e9fac00 2024-04-22T13:34:46.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.963+0000 7f7a617dd740 1 -- 172.21.15.87:0/2554890440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 msgr2=0x55bc4e9fd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:46.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.963+0000 7f7a617dd740 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7a50000f80 tx=0x7f7a5009c380 comp rx=0 tx=0).stop 2024-04-22T13:34:46.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.969+0000 7f7a5d191640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 0x7f7a400a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 -- 172.21.15.87:0/2554890440 shutdown_connections 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fd050 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9fa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 --2- 172.21.15.87:0/2554890440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 -- 172.21.15.87:0/2554890440 >> 172.21.15.87:0/2554890440 conn(0x55bc4e9ebda0 msgr2=0x55bc4e9ec180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:46.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 -- 172.21.15.87:0/2554890440 shutdown_connections 2024-04-22T13:34:46.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.977+0000 7f7a617dd740 1 -- 172.21.15.87:0/2554890440 wait complete. 2024-04-22T13:34:46.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.978+0000 7f7a617dd740 1 -- start start 2024-04-22T13:34:46.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.978+0000 7f7a617dd740 1 Processor -- start 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9fa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e9fdb60 con 0x55bc4e9fac00 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e90f500 con 0x55bc4e9f16d0 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bc4e9fdce0 con 0x55bc4e9f2080 2024-04-22T13:34:46.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.980+0000 7f7a617dd740 10 client.0 _ll_register_callbacks cb 0x55bc4e8ddb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:46.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.981+0000 7f7a5c990640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9fa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.984+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:46.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.984+0000 7f7a57fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34436/0 (socket says 172.21.15.87:34436) 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.984+0000 7f7a57fff640 1 -- 172.21.15.87:0/3292429567 learned_addr learned my addr 172.21.15.87:0/3292429567 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.986+0000 7f7a57fff640 1 -- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 msgr2=0x55bc4e9fa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.986+0000 7f7a57fff640 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9fa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.986+0000 7f7a57fff640 1 -- 172.21.15.87:0/3292429567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 msgr2=0x55bc4e9f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.986+0000 7f7a57fff640 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:46.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.986+0000 7f7a57fff640 1 -- 172.21.15.87:0/3292429567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc4e6683f0 con 0x55bc4e9f2080 2024-04-22T13:34:46.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.991+0000 7f7a57fff640 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a5009c1b0 tx=0x7f7a500cdcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea98e4200d28b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:46.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.991+0000 7f7a37fff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:46.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.994+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a500b4070 con 0x55bc4e9f2080 2024-04-22T13:34:46.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:46.994+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7a5000b570 con 0x55bc4e9f2080 2024-04-22T13:34:46.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.001+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a5009c4b0 con 0x55bc4e9f2080 2024-04-22T13:34:46.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.001+0000 7f7a5d191640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 0x7f7a400a6360 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a480095f0 tx=0x7f7a4802c1a0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:47.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.005+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7a500a5120 con 0x55bc4e9f2080 2024-04-22T13:34:47.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.005+0000 7f7a617dd740 10 client.25792 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:47.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.005+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bc4e7327d0 con 0x55bc4e9f2080 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a5009c4b0 con 0x55bc4e9f2080 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7a5000b040 con 0x55bc4e9f2080 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 10 client.25792 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 20 client.25792 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea1bc90 0x55bc4ea1e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 4 client.25792 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:47.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.006+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc4e8f5930 con 0x55bc4ea1bc90 2024-04-22T13:34:47.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.016+0000 7f7a5c990640 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea1bc90 0x55bc4ea1e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.020+0000 7f7a37fff640 10 client.25792 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:47.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.020+0000 7f7a5c990640 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea1bc90 0x55bc4ea1e080 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92d02ed8f828f7dd server_cookie=3349348ec90f163e in_seq=0 out_seq=0 2024-04-22T13:34:47.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.025+0000 7f7a37fff640 1 -- 172.21.15.87:0/3292429567 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc4e8f5930 con 0x55bc4ea1bc90 2024-04-22T13:34:47.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.025+0000 7f7a37fff640 10 client.25792 handle_command_reply: tid=0 2024-04-22T13:34:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.025+0000 7f7a617dd740 1 client.25792 shutdown 2024-04-22T13:34:47.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.027+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 msgr2=0x55bc4e9f77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:47.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.027+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a5009c1b0 tx=0x7f7a500cdcd0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 shutdown_connections 2024-04-22T13:34:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea1bc90 0x55bc4ea1e080 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4e9f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f77f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 --2- 172.21.15.87:0/3292429567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4e9fa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a37fff640 0 client.25792 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:47.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.030+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 >> 172.21.15.87:0/3292429567 conn(0x55bc4e9ebda0 msgr2=0x55bc4e9edaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:47.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.031+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 shutdown_connections 2024-04-22T13:34:47.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.032+0000 7f7a617dd740 1 -- 172.21.15.87:0/3292429567 wait complete. 2024-04-22T13:34:47.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.032+0000 7f7a617dd740 20 client.25792 trim_cache size 0 max 16384 2024-04-22T13:34:47.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.092+0000 7f7a617dd740 1 Processor -- start 2024-04-22T13:34:47.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 -- start start 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea06850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581b69a0 con 0x55bc4e9f2080 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7a580737d0 con 0x55bc4e9f16d0 2024-04-22T13:34:47.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.093+0000 7f7a617dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58071620 con 0x55bc4e9fac00 2024-04-22T13:34:47.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.097+0000 7f7a5d191640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.097+0000 7f7a5d191640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34442/0 (socket says 172.21.15.87:34442) 2024-04-22T13:34:47.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.097+0000 7f7a5d191640 1 -- 172.21.15.87:0/1525196312 learned_addr learned my addr 172.21.15.87:0/1525196312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:47.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.097+0000 7f7a5c990640 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea06850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.100+0000 7f7a5d191640 1 -- 172.21.15.87:0/1525196312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 msgr2=0x55bc4ea06850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:47.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.100+0000 7f7a5d191640 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea06850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.100+0000 7f7a5d191640 1 -- 172.21.15.87:0/1525196312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9f2080 msgr2=0x55bc4e9f9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:47.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.100+0000 7f7a5d191640 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.100+0000 7f7a5d191640 1 -- 172.21.15.87:0/1525196312 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4e9f8090 con 0x55bc4e9fac00 2024-04-22T13:34:47.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.103+0000 7f7a5d191640 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009860 tx=0x7f7a4803e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e346647eb821bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:47.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.104+0000 7f7a3e7fc640 1 -- 172.21.15.87:0/1525196312 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a48050070 con 0x55bc4e9fac00 2024-04-22T13:34:47.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.104+0000 7f7a3e7fc640 1 -- 172.21.15.87:0/1525196312 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a4803a040 con 0x55bc4e9fac00 2024-04-22T13:34:47.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.104+0000 7f7a3e7fc640 1 -- 172.21.15.87:0/1525196312 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a480406c0 con 0x55bc4e9fac00 2024-04-22T13:34:47.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.106+0000 7f7a617dd740 1 -- 172.21.15.87:0/1525196312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 msgr2=0x55bc4e9fa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:47.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.106+0000 7f7a617dd740 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009860 tx=0x7f7a4803e000 comp rx=0 tx=0).stop 2024-04-22T13:34:47.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.108+0000 7f7a617dd740 1 -- 172.21.15.87:0/1525196312 shutdown_connections 2024-04-22T13:34:47.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.108+0000 7f7a617dd740 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9fac00 0x55bc4e9fa460 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.108+0000 7f7a617dd740 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9f2080 0x55bc4e9f9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.108+0000 7f7a617dd740 1 --2- 172.21.15.87:0/1525196312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea06850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.108+0000 7f7a617dd740 1 -- 172.21.15.87:0/1525196312 >> 172.21.15.87:0/1525196312 conn(0x55bc4e9ebda0 msgr2=0x55bc4e9eda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:47.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.113+0000 7f7a617dd740 1 -- 172.21.15.87:0/1525196312 shutdown_connections 2024-04-22T13:34:47.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.113+0000 7f7a617dd740 1 -- 172.21.15.87:0/1525196312 wait complete. 2024-04-22T13:34:47.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.114+0000 7f7a617dd740 1 -- start start 2024-04-22T13:34:47.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.114+0000 7f7a617dd740 1 Processor -- start 2024-04-22T13:34:47.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f2080 0x55bc4ea1e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4ea1ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58071c90 con 0x55bc4e9fac00 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58073100 con 0x55bc4e9f2080 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58072360 con 0x55bc4e9f16d0 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.118+0000 7f7a617dd740 10 client.0 _ll_register_callbacks cb 0x7f7a5009c280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.119+0000 7f7a5c990640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.119+0000 7f7a5c990640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34452/0 (socket says 172.21.15.87:34452) 2024-04-22T13:34:47.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.119+0000 7f7a5c990640 1 -- 172.21.15.87:0/870496650 learned_addr learned my addr 172.21.15.87:0/870496650 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.120+0000 7f7a57fff640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f2080 0x55bc4ea1e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5d191640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4ea1ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 -- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f2080 msgr2=0x55bc4ea1e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f2080 0x55bc4ea1e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 -- 172.21.15.87:0/870496650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 msgr2=0x55bc4ea1ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:47.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4ea1ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 -- 172.21.15.87:0/870496650 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc4ea1f4e0 con 0x55bc4e9f16d0 2024-04-22T13:34:47.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5d191640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4ea1ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:47.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.121+0000 7f7a5c990640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c004790 tx=0x7f7a4c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba3ec8e7f6f7974b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:47.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.124+0000 7f7a37fff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:47.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.124+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c021440 con 0x55bc4e9f16d0 2024-04-22T13:34:47.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.124+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7a4c015410 con 0x55bc4e9f16d0 2024-04-22T13:34:47.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.125+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4803a040 con 0x55bc4e9f16d0 2024-04-22T13:34:47.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.129+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7a4c015610 con 0x55bc4e9f16d0 2024-04-22T13:34:47.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.130+0000 7f7a617dd740 10 client.25798 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:47.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.130+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bc4ea81e20 con 0x55bc4e9f16d0 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a4803a040 con 0x55bc4e9f16d0 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7a4c01f070 con 0x55bc4e9f16d0 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 10 client.25798 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 10 client.25798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 20 client.25798 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea82500 0x55bc4ea84920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 4 client.25798 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:34:47.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.132+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc4e963c10 con 0x55bc4ea82500 2024-04-22T13:34:47.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.134+0000 7f7a57fff640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea82500 0x55bc4ea84920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:47.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.136+0000 7f7a57fff640 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea82500 0x55bc4ea84920 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afa95dc771f4f38b server_cookie=620ecd63eb62f616 in_seq=0 out_seq=0 2024-04-22T13:34:47.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.136+0000 7f7a37fff640 10 client.25798 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:47.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.143+0000 7f7a37fff640 1 -- 172.21.15.87:0/870496650 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc4e963c10 con 0x55bc4ea82500 2024-04-22T13:34:47.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.143+0000 7f7a37fff640 10 client.25798 handle_command_reply: tid=0 2024-04-22T13:34:47.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.143+0000 7f7a617dd740 1 client.25798 shutdown 2024-04-22T13:34:47.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 msgr2=0x55bc4ea213c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c004790 tx=0x7f7a4c00be80 comp rx=0 tx=0).stop 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 shutdown_connections 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bc4ea82500 0x55bc4ea84920 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bc4e9fac00 0x55bc4ea1ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bc4e9f2080 0x55bc4ea1e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 --2- 172.21.15.87:0/870496650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc4e9f16d0 0x55bc4ea213c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a37fff640 0 client.25798 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:47.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.145+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 >> 172.21.15.87:0/870496650 conn(0x55bc4e9ebda0 msgr2=0x55bc4e9f7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:47.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.152+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 shutdown_connections 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.155+0000 7f7a617dd740 1 -- 172.21.15.87:0/870496650 wait complete. 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.155+0000 7f7a617dd740 20 client.25798 trim_cache size 0 max 16384 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 msgr2=0x7f7a400a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 0x7f7a400a6360 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7a480095f0 tx=0x7f7a4802c1a0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 msgr2=0x7f7a5810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:47.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000eea0 tx=0x7f7a5000ef90 comp rx=0 tx=0).stop 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 shutdown_connections 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7a400a3ea0 0x7f7a400a6360 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a581061c0 0x7f7a5810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a581016e0 0x7f7a5810e060 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 --2- 172.21.15.87:0/1161994644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a58100dc0 0x7f7a58114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:47.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.158+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 >> 172.21.15.87:0/1161994644 conn(0x7f7a580fc7d0 msgr2=0x7f7a580fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:47.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.161+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 shutdown_connections 2024-04-22T13:34:47.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:47.161+0000 7f7a5d992640 1 -- 172.21.15.87:0/1161994644 wait complete. 2024-04-22T13:34:48.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:47 smithi120 ceph-mon[31101]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 738 MiB/s wr, 1.83k op/s 2024-04-22T13:34:48.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:47 smithi132 ceph-mon[31531]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 738 MiB/s wr, 1.83k op/s 2024-04-22T13:34:48.216 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:34:48.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:34:48.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:34:48.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:47 smithi087 ceph-mon[27999]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 738 MiB/s wr, 1.83k op/s 2024-04-22T13:34:48.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.575+0000 7fed60a25640 1 -- 172.21.15.87:0/127611793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 msgr2=0x7fed5c110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.575+0000 7fed60a25640 1 --2- 172.21.15.87:0/127611793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 0x7fed5c110940 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed50009890 tx=0x7fed5002bbb0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.578+0000 7fed60a25640 1 -- 172.21.15.87:0/127611793 shutdown_connections 2024-04-22T13:34:48.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.578+0000 7fed60a25640 1 --2- 172.21.15.87:0/127611793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c10ebe0 0x7fed5c11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.578+0000 7fed60a25640 1 --2- 172.21.15.87:0/127611793 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.578+0000 7fed60a25640 1 --2- 172.21.15.87:0/127611793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 0x7fed5c110940 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.578+0000 7fed60a25640 1 -- 172.21.15.87:0/127611793 >> 172.21.15.87:0/127611793 conn(0x7fed5c06ce30 msgr2=0x7fed5c06d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.589+0000 7fed60a25640 1 -- 172.21.15.87:0/127611793 shutdown_connections 2024-04-22T13:34:48.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.590+0000 7fed60a25640 1 -- 172.21.15.87:0/127611793 wait complete. 2024-04-22T13:34:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.591+0000 7fed60a25640 1 Processor -- start 2024-04-22T13:34:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.593+0000 7fed60a25640 1 -- start start 2024-04-22T13:34:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.593+0000 7fed60a25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.593+0000 7fed60a25640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c10ebe0 0x7fed5c1a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.594+0000 7fed60a25640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 0x7fed5c1abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.594+0000 7fed60a25640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fed5c120e60 con 0x7fed5c10e030 2024-04-22T13:34:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.594+0000 7fed60a25640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fed5c120ce0 con 0x7fed5c10ebe0 2024-04-22T13:34:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.594+0000 7fed60a25640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed5c120fe0 con 0x7fed5c110560 2024-04-22T13:34:48.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.596+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.596+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33858/0 (socket says 172.21.15.87:33858) 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.596+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2598553067 learned_addr learned my addr 172.21.15.87:0/2598553067 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.597+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2598553067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 msgr2=0x7fed5c1abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.597+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 0x7fed5c1abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.597+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2598553067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c10ebe0 msgr2=0x7fed5c1a78a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.597+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c10ebe0 0x7fed5c1a78a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.597+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2598553067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c1ac350 con 0x7fed5c10e030 2024-04-22T13:34:48.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.598+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fed5002c1c0 tx=0x7fed500045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.599+0000 7fed43fff640 1 -- 172.21.15.87:0/2598553067 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50039050 con 0x7fed5c10e030 2024-04-22T13:34:48.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.599+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5c1ac5e0 con 0x7fed5c10e030 2024-04-22T13:34:48.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.599+0000 7fed43fff640 1 -- 172.21.15.87:0/2598553067 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed50045070 con 0x7fed5c10e030 2024-04-22T13:34:48.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.599+0000 7fed43fff640 1 -- 172.21.15.87:0/2598553067 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50040720 con 0x7fed5c10e030 2024-04-22T13:34:48.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.599+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5c07a1b0 con 0x7fed5c10e030 2024-04-22T13:34:48.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.604+0000 7fed63e10740 1 Processor -- start 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.616+0000 7fed63e10740 1 -- start start 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e57e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e63420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bb32dbbf00 con 0x55bb32e57aa0 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb32d59590 con 0x55bb32e60fd0 2024-04-22T13:34:48.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.617+0000 7fed63e10740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bb32db7ec0 con 0x55bb32e58450 2024-04-22T13:34:48.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.624+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.624+0000 7fed5a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e63420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.625+0000 7fed43fff640 1 -- 172.21.15.87:0/2598553067 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fed500035f0 con 0x7fed5c10e030 2024-04-22T13:34:48.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.625+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34470/0 (socket says 172.21.15.87:34470) 2024-04-22T13:34:48.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.625+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2025435536 learned_addr learned my addr 172.21.15.87:0/2025435536 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:48.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.632+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e57e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.632+0000 7fed43fff640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 0x7fed440a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.633+0000 7fed43fff640 1 -- 172.21.15.87:0/2598553067 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fed500f6060 con 0x7fed5c10e030 2024-04-22T13:34:48.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 0x7fed440a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2025435536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 msgr2=0x55bb32e63420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e63420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2025435536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 msgr2=0x55bb32e57e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e57e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.637+0000 7fed5ad9d640 1 -- 172.21.15.87:0/2025435536 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb32b967e0 con 0x55bb32e58450 2024-04-22T13:34:48.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.640+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 0x7fed440a62a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed54005fd0 tx=0x7fed540095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.642+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed500ef620 tx=0x7fed5002bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb9df6ceb0f8648a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.645+0000 7fed417fa640 1 -- 172.21.15.87:0/2025435536 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500bd7b0 con 0x55bb32e58450 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.645+0000 7fed417fa640 1 -- 172.21.15.87:0/2025435536 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed500f5020 con 0x55bb32e58450 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.645+0000 7fed417fa640 1 -- 172.21.15.87:0/2025435536 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500c6a30 con 0x55bb32e58450 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.647+0000 7fed63e10740 1 -- 172.21.15.87:0/2025435536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 msgr2=0x55bb32e60a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.647+0000 7fed63e10740 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed500ef620 tx=0x7fed5002bbb0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.648+0000 7fed63e10740 1 -- 172.21.15.87:0/2025435536 shutdown_connections 2024-04-22T13:34:48.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.648+0000 7fed63e10740 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e63420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.648+0000 7fed63e10740 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e58450 0x55bb32e60a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.648+0000 7fed63e10740 1 --2- 172.21.15.87:0/2025435536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e57e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.648+0000 7fed63e10740 1 -- 172.21.15.87:0/2025435536 >> 172.21.15.87:0/2025435536 conn(0x55bb32e52170 msgr2=0x55bb32e52550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.656+0000 7fed63e10740 1 -- 172.21.15.87:0/2025435536 shutdown_connections 2024-04-22T13:34:48.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.656+0000 7fed63e10740 1 -- 172.21.15.87:0/2025435536 wait complete. 2024-04-22T13:34:48.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.657+0000 7fed63e10740 1 -- start start 2024-04-22T13:34:48.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.657+0000 7fed63e10740 1 Processor -- start 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 0x55bb32e5df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e58450 0x55bb32e5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e5e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bb32e63d00 con 0x55bb32e58450 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb32d75580 con 0x55bb32e60fd0 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bb32e63e80 con 0x55bb32e57aa0 2024-04-22T13:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.658+0000 7fed63e10740 10 client.0 _ll_register_callbacks cb 0x55bb32d7b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:48.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.660+0000 7fed5b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e5e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.660+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 0x55bb32e5df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.661+0000 7fed5b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e5e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54856/0 (socket says 172.21.15.87:54856) 2024-04-22T13:34:48.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.661+0000 7fed5b59e640 1 -- 172.21.15.87:0/3168209620 learned_addr learned my addr 172.21.15.87:0/3168209620 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:48.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.665+0000 7fed5a59c640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e58450 0x55bb32e5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.666+0000 7fed5ad9d640 1 -- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 msgr2=0x55bb32e5e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.666+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e5e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.666+0000 7fed5ad9d640 1 -- 172.21.15.87:0/3168209620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e58450 msgr2=0x55bb32e5e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.666+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e58450 0x55bb32e5e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.666+0000 7fed5ad9d640 1 -- 172.21.15.87:0/3168209620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb32bf6f70 con 0x55bb32e57aa0 2024-04-22T13:34:48.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.669+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 0x55bb32e5df20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed500ba820 tx=0x7fed500f4870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef2365a23d410e42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.669+0000 7fed3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:48.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.670+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500d6070 con 0x55bb32e57aa0 2024-04-22T13:34:48.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.671+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed500f5020 con 0x55bb32e57aa0 2024-04-22T13:34:48.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.671+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fed50034ba0 con 0x55bb32e57aa0 2024-04-22T13:34:48.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.674+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fed500bd7b0 con 0x55bb32e57aa0 2024-04-22T13:34:48.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.674+0000 7fed63e10740 10 client.25807 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:48.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.674+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb32a8e170 con 0x55bb32e57aa0 2024-04-22T13:34:48.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.676+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed500f5020 con 0x55bb32e57aa0 2024-04-22T13:34:48.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.677+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fed500ef6e0 con 0x55bb32e57aa0 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 10 client.25807 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 10 client.25807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 20 client.25807 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e71040 0x55bb32e73430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 4 client.25807 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:48.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.680+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb32dc9920 con 0x55bb32e71040 2024-04-22T13:34:48.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.682+0000 7fed5a59c640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e71040 0x55bb32e73430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.708+0000 7fed3eff5640 10 client.25807 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:48.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.708+0000 7fed5a59c640 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e71040 0x55bb32e73430 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144889847b56e442 server_cookie=2e8e73155a11f7c5 in_seq=0 out_seq=0 2024-04-22T13:34:48.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.712+0000 7fed3eff5640 1 -- 172.21.15.87:0/3168209620 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb32dc9920 con 0x55bb32e71040 2024-04-22T13:34:48.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.712+0000 7fed3eff5640 10 client.25807 handle_command_reply: tid=0 2024-04-22T13:34:48.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.712+0000 7fed63e10740 1 client.25807 shutdown 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 msgr2=0x55bb32e5df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 0x55bb32e5df20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed500ba820 tx=0x7fed500f4870 comp rx=0 tx=0).stop 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 shutdown_connections 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e71040 0x55bb32e73430 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e60fd0 0x55bb32e5e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e58450 0x55bb32e5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 --2- 172.21.15.87:0/3168209620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e57aa0 0x55bb32e5df20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.717+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 >> 172.21.15.87:0/3168209620 conn(0x55bb32e52170 msgr2=0x55bb32e53ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.725+0000 7fed3eff5640 0 client.25807 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:48.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.725+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 shutdown_connections 2024-04-22T13:34:48.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.729+0000 7fed63e10740 1 -- 172.21.15.87:0/3168209620 wait complete. 2024-04-22T13:34:48.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.734+0000 7fed63e10740 20 client.25807 trim_cache size 0 max 16384 2024-04-22T13:34:48.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.811+0000 7fed63e10740 1 Processor -- start 2024-04-22T13:34:48.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.812+0000 7fed63e10740 1 -- start start 2024-04-22T13:34:48.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed63e10740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed63e10740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e6ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed63e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e78be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed63e10740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fed48010f90 con 0x55bb32e57aa0 2024-04-22T13:34:48.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed5ad9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:34482/0 (socket says 172.21.15.87:34482) 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed5ad9d640 1 -- 172.21.15.87:0/1996341220 learned_addr learned my addr 172.21.15.87:0/1996341220 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.813+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fed4800e860 con 0x55bb32e58450 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.814+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed4800c6b0 con 0x55bb32e52170 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.814+0000 7fed5b59e640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e78be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.814+0000 7fed5a59c640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e6ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.814+0000 7fed5a59c640 1 -- 172.21.15.87:0/1996341220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 msgr2=0x55bb32e60650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.815+0000 7fed5a59c640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.815+0000 7fed5a59c640 1 -- 172.21.15.87:0/1996341220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 msgr2=0x55bb32e78be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.815+0000 7fed5a59c640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e78be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.815+0000 7fed5a59c640 1 -- 172.21.15.87:0/1996341220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb32e57550 con 0x55bb32e57aa0 2024-04-22T13:34:48.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.815+0000 7fed5a59c640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e6ca70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x55bb32e73b50 tx=0x7fed4c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6755a60bb2bb9410 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.817+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:48.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.817+0000 7fed40ff9640 1 -- 172.21.15.87:0/1996341220 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c005a50 con 0x55bb32e57aa0 2024-04-22T13:34:48.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.817+0000 7fed40ff9640 1 -- 172.21.15.87:0/1996341220 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4c01f030 con 0x55bb32e57aa0 2024-04-22T13:34:48.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.817+0000 7fed40ff9640 1 -- 172.21.15.87:0/1996341220 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0064b0 con 0x55bb32e57aa0 2024-04-22T13:34:48.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.819+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 msgr2=0x55bb32e6ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.819+0000 7fed63e10740 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e6ca70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x55bb32e73b50 tx=0x7fed4c009060 comp rx=0 tx=0).stop 2024-04-22T13:34:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.821+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 shutdown_connections 2024-04-22T13:34:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.821+0000 7fed63e10740 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e78be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.821+0000 7fed63e10740 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e6ca70 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.821+0000 7fed63e10740 1 --2- 172.21.15.87:0/1996341220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e60650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.822+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 >> 172.21.15.87:0/1996341220 conn(0x55bb32e5cb60 msgr2=0x55bb32e51390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.822+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 shutdown_connections 2024-04-22T13:34:48.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.822+0000 7fed63e10740 1 -- 172.21.15.87:0/1996341220 wait complete. 2024-04-22T13:34:48.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.823+0000 7fed63e10740 1 -- start start 2024-04-22T13:34:48.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.823+0000 7fed63e10740 1 Processor -- start 2024-04-22T13:34:48.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e65f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e66490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fed4800cd20 con 0x55bb32e57aa0 2024-04-22T13:34:48.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fed4800e190 con 0x55bb32e58450 2024-04-22T13:34:48.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed4800d3f0 con 0x55bb32e52170 2024-04-22T13:34:48.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.824+0000 7fed63e10740 10 client.0 _ll_register_callbacks cb 0x7fed54009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:48.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.825+0000 7fed5b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.825+0000 7fed5b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54876/0 (socket says 172.21.15.87:54876) 2024-04-22T13:34:48.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.825+0000 7fed5b59e640 1 -- 172.21.15.87:0/2562533166 learned_addr learned my addr 172.21.15.87:0/2562533166 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:48.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.826+0000 7fed5a59c640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e66490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.826+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e65f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.832+0000 7fed5b59e640 1 -- 172.21.15.87:0/2562533166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 msgr2=0x55bb32e65f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.832+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e65f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.832+0000 7fed5b59e640 1 -- 172.21.15.87:0/2562533166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 msgr2=0x55bb32e66490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.832+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e66490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.832+0000 7fed5b59e640 1 -- 172.21.15.87:0/2562533166 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb32ed7460 con 0x55bb32e58450 2024-04-22T13:34:48.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.834+0000 7fed5b59e640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed5400ce50 tx=0x7fed54004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7de9ac36fb4bca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:48.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.834+0000 7fed3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:48.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.835+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54004b20 con 0x55bb32e58450 2024-04-22T13:34:48.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.835+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fed54019070 con 0x55bb32e58450 2024-04-22T13:34:48.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.836+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4c01f030 con 0x55bb32e58450 2024-04-22T13:34:48.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.838+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fed54004cc0 con 0x55bb32e58450 2024-04-22T13:34:48.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.841+0000 7fed63e10740 10 client.25784 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:48.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.841+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb32ed8650 con 0x55bb32e58450 2024-04-22T13:34:48.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.845+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4c01f030 con 0x55bb32e58450 2024-04-22T13:34:48.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fed5400b040 con 0x55bb32e58450 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 10 client.25784 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 20 client.25784 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e60fd0 0x55bb32edae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 4 client.25784 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:34:48.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.847+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bb32dc3790 con 0x55bb32e60fd0 2024-04-22T13:34:48.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.849+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e60fd0 0x55bb32edae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:48.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.860+0000 7fed5ad9d640 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e60fd0 0x55bb32edae10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99deeff90f40325a server_cookie=92b7c95375284ead in_seq=0 out_seq=0 2024-04-22T13:34:48.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.860+0000 7fed3eff5640 10 client.25784 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:48.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.864+0000 7fed3eff5640 1 -- 172.21.15.87:0/2562533166 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fed500f3080 con 0x55bb32e60fd0 2024-04-22T13:34:48.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.865+0000 7fed3eff5640 10 client.25784 handle_command_reply: tid=0 2024-04-22T13:34:48.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.865+0000 7fed63e10740 1 client.25784 shutdown 2024-04-22T13:34:48.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 msgr2=0x55bb32e669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed5400ce50 tx=0x7fed54004a40 comp rx=0 tx=0).stop 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 shutdown_connections 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bb32e60fd0 0x55bb32edae10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb32e58450 0x55bb32e669d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bb32e57aa0 0x55bb32e66490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 --2- 172.21.15.87:0/2562533166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bb32e52170 0x55bb32e65f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 >> 172.21.15.87:0/2562533166 conn(0x55bb32e5cb60 msgr2=0x55bb32e5dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed3eff5640 0 client.25784 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:48.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.866+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 shutdown_connections 2024-04-22T13:34:48.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.867+0000 7fed63e10740 1 -- 172.21.15.87:0/2562533166 wait complete. 2024-04-22T13:34:48.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.867+0000 7fed63e10740 20 client.25784 trim_cache size 0 max 16384 2024-04-22T13:34:48.867 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:48.867 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:34:48.867 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "06786535-0ca9-434f-8710-bba5b62a8fa2", 2024-04-22T13:34:48.867 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:34:48.867 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:48.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.872+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 msgr2=0x7fed440a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.872+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 0x7fed440a62a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed54005fd0 tx=0x7fed540095c0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.872+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 msgr2=0x7fed5c1a7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:48.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.872+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fed5002c1c0 tx=0x7fed500045e0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 shutdown_connections 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fed440a3de0 0x7fed440a62a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed5c110560 0x7fed5c1abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed5c10ebe0 0x7fed5c1a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 --2- 172.21.15.87:0/2598553067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed5c10e030 0x7fed5c1a7360 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:48.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.874+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 >> 172.21.15.87:0/2598553067 conn(0x7fed5c06ce30 msgr2=0x7fed5c0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:48.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.876+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 shutdown_connections 2024-04-22T13:34:48.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:48.877+0000 7fed60a25640 1 -- 172.21.15.87:0/2598553067 wait complete. 2024-04-22T13:34:48.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:34:48.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 -- 172.21.15.87:0/2413547112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 msgr2=0x7f2658110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 --2- 172.21.15.87:0/2413547112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f2658110940 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2648009930 tx=0x7f264802bc30 comp rx=0 tx=0).stop 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 -- 172.21.15.87:0/2413547112 shutdown_connections 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 --2- 172.21.15.87:0/2413547112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f265810ebe0 0x7f265811d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 --2- 172.21.15.87:0/2413547112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f265810e030 0x7f265810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 --2- 172.21.15.87:0/2413547112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f2658110940 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 -- 172.21.15.87:0/2413547112 >> 172.21.15.87:0/2413547112 conn(0x7f265806ce30 msgr2=0x7f265806d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 -- 172.21.15.87:0/2413547112 shutdown_connections 2024-04-22T13:34:49.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.349+0000 7f265759e640 1 -- 172.21.15.87:0/2413547112 wait complete. 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 Processor -- start 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 -- start start 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f265810e030 0x7f26581a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f265810ebe0 0x7f26581a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2658120df0 con 0x7f2658110560 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2658120c70 con 0x7f265810ebe0 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.351+0000 7f265759e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2658120f70 con 0x7f265810e030 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33922/0 (socket says 172.21.15.87:33922) 2024-04-22T13:34:49.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 -- 172.21.15.87:0/3617310154 learned_addr learned my addr 172.21.15.87:0/3617310154 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 -- 172.21.15.87:0/3617310154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f265810e030 msgr2=0x7f26581a7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f265810e030 0x7f26581a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 -- 172.21.15.87:0/3617310154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f265810ebe0 msgr2=0x7f26581a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f265810ebe0 0x7f26581a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 -- 172.21.15.87:0/3617310154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26581ac340 con 0x7f2658110560 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.352+0000 7f2656d9d640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f265000eb10 tx=0x7f265000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.353+0000 7f26477fe640 1 -- 172.21.15.87:0/3617310154 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650004b20 con 0x7f2658110560 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.353+0000 7f26477fe640 1 -- 172.21.15.87:0/3617310154 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2650004cc0 con 0x7f2658110560 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.353+0000 7f26477fe640 1 -- 172.21.15.87:0/3617310154 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650010670 con 0x7f2658110560 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.353+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26581ac630 con 0x7f2658110560 2024-04-22T13:34:49.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.353+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26581b3ec0 con 0x7f2658110560 2024-04-22T13:34:49.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.359+0000 7f265f96d740 1 Processor -- start 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 -- start start 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 0x55b81387aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b8138836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 0x55b813886000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b8137df1e0 con 0x55b813883c10 2024-04-22T13:34:49.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b81377c390 con 0x55b81387a710 2024-04-22T13:34:49.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.361+0000 7f265f96d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b8137db1a0 con 0x55b81387b030 2024-04-22T13:34:49.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.362+0000 7f2655d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 0x55b81387aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.362+0000 7f2655d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 0x55b81387aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:54902/0 (socket says 172.21.15.87:54902) 2024-04-22T13:34:49.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.362+0000 7f2655d9b640 1 -- 172.21.15.87:0/802174644 learned_addr learned my addr 172.21.15.87:0/802174644 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.363+0000 7f2655d9b640 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 0x55b813886000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.363+0000 7f26477fe640 1 -- 172.21.15.87:0/3617310154 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2650005020 con 0x7f2658110560 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 -- 172.21.15.87:0/802174644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 msgr2=0x55b8138836d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b8138836d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 -- 172.21.15.87:0/802174644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 msgr2=0x55b81387aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 0x55b81387aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 -- 172.21.15.87:0/802174644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8135bb7d0 con 0x55b813883c10 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f26477fe640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 0x7f26400a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.365+0000 7f2655d9b640 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 0x55b813886000 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f264c015270 tx=0x7f264c001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91de0186c52af5e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.366+0000 7f26477fe640 1 -- 172.21.15.87:0/3617310154 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f26500d0e50 con 0x7f2658110560 2024-04-22T13:34:49.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.366+0000 7f2644ff9640 1 -- 172.21.15.87:0/802174644 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c008980 con 0x55b813883c10 2024-04-22T13:34:49.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.366+0000 7f2644ff9640 1 -- 172.21.15.87:0/802174644 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f264c0070f0 con 0x55b813883c10 2024-04-22T13:34:49.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.367+0000 7f2644ff9640 1 -- 172.21.15.87:0/802174644 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c004b50 con 0x55b813883c10 2024-04-22T13:34:49.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.367+0000 7f265f96d740 1 -- 172.21.15.87:0/802174644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 msgr2=0x55b813886000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.367+0000 7f265f96d740 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 0x55b813886000 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f264c015270 tx=0x7f264c001040 comp rx=0 tx=0).stop 2024-04-22T13:34:49.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 -- 172.21.15.87:0/802174644 shutdown_connections 2024-04-22T13:34:49.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b813883c10 0x55b813886000 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b8138836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 --2- 172.21.15.87:0/802174644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387a710 0x55b81387aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 -- 172.21.15.87:0/802174644 >> 172.21.15.87:0/802174644 conn(0x55b813874e40 msgr2=0x55b813875220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265659c640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 0x7f26400a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 -- 172.21.15.87:0/802174644 shutdown_connections 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.368+0000 7f265f96d740 1 -- 172.21.15.87:0/802174644 wait complete. 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 -- start start 2024-04-22T13:34:49.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 Processor -- start 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b813881150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813881690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b813886a50 con 0x55b81387a710 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b8137982f0 con 0x55b81387b030 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b813886bd0 con 0x55b813883c10 2024-04-22T13:34:49.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.369+0000 7f265f96d740 10 client.0 _ll_register_callbacks cb 0x55b8137a5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:49.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.370+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.370+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33944/0 (socket says 172.21.15.87:33944) 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.370+0000 7f265659c640 1 -- 172.21.15.87:0/1675580258 learned_addr learned my addr 172.21.15.87:0/1675580258 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.370+0000 7f2656d9d640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813881690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 -- 172.21.15.87:0/1675580258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 msgr2=0x55b813881690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813881690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 -- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 msgr2=0x55b813881150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b813881150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 -- 172.21.15.87:0/1675580258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8134f13f0 con 0x55b81387a710 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f264802c9a0 tx=0x7f2648004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785395e70de04d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f263dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f265659c640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 0x7f26400a62e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2648036040 tx=0x7f264802e000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.371+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648050060 con 0x55b81387a710 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.372+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f264c0070f0 con 0x55b81387a710 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.372+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2648043450 con 0x55b81387a710 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.372+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2648043650 con 0x55b81387a710 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.372+0000 7f265f96d740 10 client.16554 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:49.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.372+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b8134b10f0 con 0x55b81387a710 2024-04-22T13:34:49.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f264c0070f0 con 0x55b81387a710 2024-04-22T13:34:49.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2648053020 con 0x55b81387a710 2024-04-22T13:34:49.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 10 client.16554 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:49.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 10 client.16554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:49.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 20 client.16554 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:49.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b813893cf0 0x55b813896170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 4 client.16554 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:49.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.373+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8137ecc00 con 0x55b813893cf0 2024-04-22T13:34:49.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.374+0000 7f2655d9b640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b813893cf0 0x55b813896170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.377+0000 7f2655d9b640 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b813893cf0 0x55b813896170 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9f44b5f233221fb server_cookie=551d7e9fb253d828 in_seq=0 out_seq=0 2024-04-22T13:34:49.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.377+0000 7f263dffb640 10 client.16554 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.381+0000 7f263dffb640 1 -- 172.21.15.87:0/1675580258 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8137ecc00 con 0x55b813893cf0 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.381+0000 7f263dffb640 10 client.16554 handle_command_reply: tid=0 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 client.16554 shutdown 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 msgr2=0x55b813880c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f264802c9a0 tx=0x7f2648004360 comp rx=0 tx=0).stop 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 shutdown_connections 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b813893cf0 0x55b813896170 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813881690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b813881150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 --2- 172.21.15.87:0/1675580258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813880c10 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 >> 172.21.15.87:0/1675580258 conn(0x55b813874e40 msgr2=0x55b813874120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f263dffb640 0 client.16554 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 shutdown_connections 2024-04-22T13:34:49.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.382+0000 7f265f96d740 1 -- 172.21.15.87:0/1675580258 wait complete. 2024-04-22T13:34:49.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.383+0000 7f265f96d740 20 client.16554 trim_cache size 0 max 16384 2024-04-22T13:34:49.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.437+0000 7f265f96d740 1 Processor -- start 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.456+0000 7f265f96d740 1 -- start start 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b81388f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813873fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f26581bef60 con 0x55b81387a710 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26581bc830 con 0x55b81387b030 2024-04-22T13:34:49.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.458+0000 7f265f96d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f26581ba680 con 0x55b813883c10 2024-04-22T13:34:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.462+0000 7f2655d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b81388f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.464+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.464+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33958/0 (socket says 172.21.15.87:33958) 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.464+0000 7f265659c640 1 -- 172.21.15.87:0/2477495366 learned_addr learned my addr 172.21.15.87:0/2477495366 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.465+0000 7f265659c640 1 -- 172.21.15.87:0/2477495366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 msgr2=0x55b813873fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.465+0000 7f265659c640 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813873fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.465+0000 7f265659c640 1 -- 172.21.15.87:0/2477495366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 msgr2=0x55b81388f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.465+0000 7f265659c640 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b81388f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.465+0000 7f265659c640 1 -- 172.21.15.87:0/2477495366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b813886000 con 0x55b81387a710 2024-04-22T13:34:49.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.466+0000 7f265659c640 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2648034800 tx=0x7f2648005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c7afc9c4ae8e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.468+0000 7f263ffff640 1 -- 172.21.15.87:0/2477495366 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648050040 con 0x55b81387a710 2024-04-22T13:34:49.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.468+0000 7f263ffff640 1 -- 172.21.15.87:0/2477495366 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2648003e30 con 0x55b81387a710 2024-04-22T13:34:49.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.469+0000 7f265f96d740 1 -- 172.21.15.87:0/2477495366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 msgr2=0x55b813883340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.469+0000 7f265f96d740 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2648034800 tx=0x7f2648005fd0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.476+0000 7f265f96d740 1 -- 172.21.15.87:0/2477495366 shutdown_connections 2024-04-22T13:34:49.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.476+0000 7f265f96d740 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b813883c10 0x55b813873fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.476+0000 7f265f96d740 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b81387b030 0x55b81388f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.476+0000 7f265f96d740 1 --2- 172.21.15.87:0/2477495366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b813883340 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.476+0000 7f265f96d740 1 -- 172.21.15.87:0/2477495366 >> 172.21.15.87:0/2477495366 conn(0x55b813874e40 msgr2=0x55b813873890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.487+0000 7f265f96d740 1 -- 172.21.15.87:0/2477495366 shutdown_connections 2024-04-22T13:34:49.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.488+0000 7f265f96d740 1 -- 172.21.15.87:0/2477495366 wait complete. 2024-04-22T13:34:49.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.488+0000 7f265f96d740 1 -- start start 2024-04-22T13:34:49.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.488+0000 7f265f96d740 1 Processor -- start 2024-04-22T13:34:49.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b81387de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b813883c10 0x55b81387e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f26581bacf0 con 0x55b81387a710 2024-04-22T13:34:49.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26581bc160 con 0x55b813883c10 2024-04-22T13:34:49.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f26581bb3c0 con 0x55b81387b030 2024-04-22T13:34:49.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.494+0000 7f265f96d740 10 client.0 _ll_register_callbacks cb 0x7f26500093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:49.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.496+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.496+0000 7f265659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33966/0 (socket says 172.21.15.87:33966) 2024-04-22T13:34:49.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.496+0000 7f265659c640 1 -- 172.21.15.87:0/3540486833 learned_addr learned my addr 172.21.15.87:0/3540486833 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.497+0000 7f2655d9b640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b81387de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.504+0000 7f265659c640 1 -- 172.21.15.87:0/3540486833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 msgr2=0x55b81387de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.504+0000 7f265659c640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b81387de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.504+0000 7f265659c640 1 -- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b813883c10 msgr2=0x55b81387e390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:49.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.504+0000 7f265659c640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b813883c10 0x55b81387e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.504+0000 7f265659c640 1 -- 172.21.15.87:0/3540486833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8138f92b0 con 0x55b81387a710 2024-04-22T13:34:49.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.507+0000 7f265659c640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f26480044a0 tx=0x7f26480041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597d6382082babf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.508+0000 7f263dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:34:49.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.530+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648050040 con 0x55b81387a710 2024-04-22T13:34:49.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.530+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b8137e6a70 con 0x55b81387a710 2024-04-22T13:34:49.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.530+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f2648034440 con 0x55b81387a710 2024-04-22T13:34:49.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.534+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2648034a70 con 0x55b81387a710 2024-04-22T13:34:49.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.534+0000 7f265f96d740 10 client.16560 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:49.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.534+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b8138fa5f0 con 0x55b81387a710 2024-04-22T13:34:49.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.551+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b8137e6a70 con 0x55b81387a710 2024-04-22T13:34:49.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.551+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f264806e030 con 0x55b81387a710 2024-04-22T13:34:49.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 10 client.16560 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:49.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:49.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 20 client.16560 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:49.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8138fabf0 0x55b8138fd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 4 client.16560 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:34:49.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.555+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b813762590 con 0x55b8138fabf0 2024-04-22T13:34:49.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.561+0000 7f2655d9b640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8138fabf0 0x55b8138fd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.579+0000 7f263dffb640 10 client.16560 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.579+0000 7f2655d9b640 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8138fabf0 0x55b8138fd010 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6ea8434e1741f23 server_cookie=7d33716383e679ee in_seq=0 out_seq=0 2024-04-22T13:34:49.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.584+0000 7f263dffb640 1 -- 172.21.15.87:0/3540486833 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55b813762590 con 0x55b8138fabf0 2024-04-22T13:34:49.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.584+0000 7f263dffb640 10 client.16560 handle_command_reply: tid=0 2024-04-22T13:34:49.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.586+0000 7f265f96d740 1 client.16560 shutdown 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 msgr2=0x55b81387d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f26480044a0 tx=0x7f26480041c0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 shutdown_connections 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8138fabf0 0x55b8138fd010 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b813883c10 0x55b81387e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b81387b030 0x55b81387de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 --2- 172.21.15.87:0/3540486833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b81387a710 0x55b81387d910 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 >> 172.21.15.87:0/3540486833 conn(0x55b813874e40 msgr2=0x55b813874120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.587+0000 7f263dffb640 0 client.16560 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.590+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 shutdown_connections 2024-04-22T13:34:49.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.590+0000 7f265f96d740 1 -- 172.21.15.87:0/3540486833 wait complete. 2024-04-22T13:34:49.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.592+0000 7f265f96d740 20 client.16560 trim_cache size 0 max 16384 2024-04-22T13:34:49.588 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:34:49.588 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: "06786535-0ca9-434f-8710-bba5b62a8fa2": { 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "06786535-0ca9-434f-8710-bba5b62a8fa2", 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.594+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 msgr2=0x7f26400a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.594+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 0x7f26400a62e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2648036040 tx=0x7f264802e000 comp rx=0 tx=0).stop 2024-04-22T13:34:49.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.594+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 msgr2=0x7f26581abc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.594+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f265000eb10 tx=0x7f265000c2d0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.596+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 shutdown_connections 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.597+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f26400a3e20 0x7f26400a62e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.597+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658110560 0x7f26581abc10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.597+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f265810ebe0 0x7f26581a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.598+0000 7f265759e640 1 --2- 172.21.15.87:0/3617310154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f265810e030 0x7f26581a7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.598+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 >> 172.21.15.87:0/3617310154 conn(0x7f265806ce30 msgr2=0x7f2658071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.604+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 shutdown_connections 2024-04-22T13:34:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:34:49.605+0000 7f265759e640 1 -- 172.21.15.87:0/3617310154 wait complete. 2024-04-22T13:34:49.701 INFO:tasks.cephfs.filesystem:scrub status for tag:06786535-0ca9-434f-8710-bba5b62a8fa2 - {'path': '/', 'tag': '06786535-0ca9-434f-8710-bba5b62a8fa2', 'options': 'recursive,force'} 2024-04-22T13:34:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.839+0000 7fcf767fc640 1 -- 172.21.15.87:0/3635313218 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70032950 con 0x7fcf78109520 2024-04-22T13:34:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.840+0000 7fcf7cf34640 1 -- 172.21.15.87:0/3635313218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf78109520 msgr2=0x7fcf781099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.840+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/3635313218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf78109520 0x7fcf781099a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ca0 tx=0x7fcf7002c140 comp rx=0 tx=0).stop 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.841+0000 7fcf7cf34640 1 -- 172.21.15.87:0/3635313218 shutdown_connections 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.841+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/3635313218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78114c90 0x7fcf78117080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.841+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/3635313218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf78109520 0x7fcf781099a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.841+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/3635313218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf7806e240 0x7fcf78108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.841+0000 7fcf7cf34640 1 -- 172.21.15.87:0/3635313218 >> 172.21.15.87:0/3635313218 conn(0x7fcf7806ce90 msgr2=0x7fcf7806d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.842+0000 7fcf7cf34640 1 -- 172.21.15.87:0/3635313218 shutdown_connections 2024-04-22T13:34:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.843+0000 7fcf7cf34640 1 -- 172.21.15.87:0/3635313218 wait complete. 2024-04-22T13:34:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.844+0000 7fcf7cf34640 1 Processor -- start 2024-04-22T13:34:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.844+0000 7fcf7cf34640 1 -- start start 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.844+0000 7fcf7cf34640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 0x7fcf7811ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.844+0000 7fcf7cf34640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.845+0000 7fcf7cf34640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf78114c90 0x7fcf78119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.845+0000 7fcf7cf34640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78118760 con 0x7fcf7806e240 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.845+0000 7fcf7cf34640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf781185e0 con 0x7fcf78109520 2024-04-22T13:34:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.845+0000 7fcf7cf34640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf781188e0 con 0x7fcf78114c90 2024-04-22T13:34:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.846+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.846+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:59524/0 (socket says 172.21.15.87:59524) 2024-04-22T13:34:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.846+0000 7fcf76ffd640 1 -- 172.21.15.87:0/623402308 learned_addr learned my addr 172.21.15.87:0/623402308 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.847+0000 7fcf76ffd640 1 -- 172.21.15.87:0/623402308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf78114c90 msgr2=0x7fcf78119f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:34:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.847+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf78114c90 0x7fcf78119f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.847+0000 7fcf76ffd640 1 -- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 msgr2=0x7fcf7811ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.847+0000 7fcf777fe640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 0x7fcf7811ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.848+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 0x7fcf7811ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.848+0000 7fcf76ffd640 1 -- 172.21.15.87:0/623402308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7811a840 con 0x7fcf78109520 2024-04-22T13:34:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.848+0000 7fcf777fe640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 0x7fcf7811ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:34:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.849+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf70037040 tx=0x7fcf70035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.850+0000 7fcf74ff9640 1 -- 172.21.15.87:0/623402308 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70039040 con 0x7fcf78109520 2024-04-22T13:34:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.850+0000 7fcf74ff9640 1 -- 172.21.15.87:0/623402308 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf70044070 con 0x7fcf78109520 2024-04-22T13:34:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.850+0000 7fcf74ff9640 1 -- 172.21.15.87:0/623402308 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70004550 con 0x7fcf78109520 2024-04-22T13:34:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.850+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf7811aad0 con 0x7fcf78109520 2024-04-22T13:34:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.851+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf781bcb00 con 0x7fcf78109520 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.856+0000 7fcf80d89740 1 Processor -- start 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.856+0000 7fcf80d89740 1 -- start start 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482ac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 0x557f482b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557f482118d0 con 0x557f482adac0 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f481aee80 con 0x557f482b62d0 2024-04-22T13:34:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.857+0000 7fcf80d89740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557f4820d890 con 0x557f482acd50 2024-04-22T13:34:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.860+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 0x557f482b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.860+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 0x557f482b86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:59532/0 (socket says 172.21.15.87:59532) 2024-04-22T13:34:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.860+0000 7fcf76ffd640 1 -- 172.21.15.87:0/558362267 learned_addr learned my addr 172.21.15.87:0/558362267 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.861+0000 7fcf77fff640 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.861+0000 7fcf777fe640 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 -- 172.21.15.87:0/558362267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 msgr2=0x557f482ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 -- 172.21.15.87:0/558362267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 msgr2=0x557f482b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 0x557f482b86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 -- 172.21.15.87:0/558362267 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f481d0320 con 0x557f482adac0 2024-04-22T13:34:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.862+0000 7fcf77fff640 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482ac810 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcf7806b1a0 tx=0x7fcf6c009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c47e51a770527d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.867+0000 7fcf74ff9640 1 -- 172.21.15.87:0/623402308 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcf700046f0 con 0x7fcf78109520 2024-04-22T13:34:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.867+0000 7fcf5dffb640 1 -- 172.21.15.87:0/558362267 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c015020 con 0x557f482adac0 2024-04-22T13:34:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.867+0000 7fcf5dffb640 1 -- 172.21.15.87:0/558362267 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf6c011ce0 con 0x557f482adac0 2024-04-22T13:34:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.867+0000 7fcf5dffb640 1 -- 172.21.15.87:0/558362267 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00bde0 con 0x557f482adac0 2024-04-22T13:34:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.868+0000 7fcf74ff9640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 0x7fcf600a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.868+0000 7fcf74ff9640 1 -- 172.21.15.87:0/623402308 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf700f2960 con 0x7fcf78109520 2024-04-22T13:34:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.870+0000 7fcf80d89740 1 -- 172.21.15.87:0/558362267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 msgr2=0x557f482ac810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.870+0000 7fcf80d89740 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482ac810 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcf7806b1a0 tx=0x7fcf6c009a90 comp rx=0 tx=0).stop 2024-04-22T13:34:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.875+0000 7fcf777fe640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 0x7fcf600a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.877+0000 7fcf80d89740 1 -- 172.21.15.87:0/558362267 shutdown_connections 2024-04-22T13:34:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.877+0000 7fcf80d89740 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482b62d0 0x557f482b86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.877+0000 7fcf80d89740 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.877+0000 7fcf80d89740 1 --2- 172.21.15.87:0/558362267 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482ac810 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.877+0000 7fcf80d89740 1 -- 172.21.15.87:0/558362267 >> 172.21.15.87:0/558362267 conn(0x557f48273c90 msgr2=0x557f482a7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.879+0000 7fcf80d89740 1 -- 172.21.15.87:0/558362267 shutdown_connections 2024-04-22T13:34:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.883+0000 7fcf80d89740 1 -- 172.21.15.87:0/558362267 wait complete. 2024-04-22T13:34:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.884+0000 7fcf80d89740 1 -- start start 2024-04-22T13:34:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.884+0000 7fcf80d89740 1 Processor -- start 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482b62d0 0x557f482b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557f482b9000 con 0x557f482adac0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f48199eb0 con 0x557f482acd50 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557f482b9180 con 0x557f482b62d0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf80d89740 10 client.0 _ll_register_callbacks cb 0x557f481c6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf76ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38912/0 (socket says 172.21.15.87:38912) 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.888+0000 7fcf76ffd640 1 -- 172.21.15.87:0/1841759874 learned_addr learned my addr 172.21.15.87:0/1841759874 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.889+0000 7fcf777fe640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 0x7fcf600a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557f482ae220 tx=0x7fcf68008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.890+0000 7fcf777fe640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.891+0000 7fcf77fff640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482b62d0 0x557f482b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.895+0000 7fcf777fe640 1 -- 172.21.15.87:0/1841759874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482b62d0 msgr2=0x557f482b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.895+0000 7fcf777fe640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482b62d0 0x557f482b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.895+0000 7fcf777fe640 1 -- 172.21.15.87:0/1841759874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 msgr2=0x557f482b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.895+0000 7fcf777fe640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.895+0000 7fcf777fe640 1 -- 172.21.15.87:0/1841759874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f47f24210 con 0x557f482acd50 2024-04-22T13:34:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.896+0000 7fcf777fe640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482b5bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800a3a0 tx=0x7fcf680046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e505f2ac3ee9e6c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.898+0000 7fcf56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:34:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.901+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6801a050 con 0x557f482acd50 2024-04-22T13:34:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.901+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf68014750 con 0x557f482acd50 2024-04-22T13:34:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.902+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf6c011ce0 con 0x557f482acd50 2024-04-22T13:34:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.904+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf68005970 con 0x557f482acd50 2024-04-22T13:34:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.904+0000 7fcf80d89740 10 client.25796 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.904+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557f47ff0620 con 0x557f482acd50 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.906+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf6c011ce0 con 0x557f482acd50 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf680062d0 con 0x557f482acd50 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 10 client.25796 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 10 client.25796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 20 client.25796 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482c6330 0x557f482c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 4 client.25796 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:34:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.908+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f48225880 con 0x557f482c6330 2024-04-22T13:34:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.916+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482c6330 0x557f482c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.922+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482c6330 0x557f482c8760 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92f37ac412362aef server_cookie=1f4fe51fef57fd05 in_seq=0 out_seq=0 2024-04-22T13:34:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.924+0000 7fcf56ffd640 10 client.25796 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.925+0000 7fcf56ffd640 1 -- 172.21.15.87:0/1841759874 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f48225880 con 0x557f482c6330 2024-04-22T13:34:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.925+0000 7fcf56ffd640 10 client.25796 handle_command_reply: tid=0 2024-04-22T13:34:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.926+0000 7fcf80d89740 1 client.25796 shutdown 2024-04-22T13:34:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.926+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 msgr2=0x557f482b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.926+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482b5bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800a3a0 tx=0x7fcf680046e0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 shutdown_connections 2024-04-22T13:34:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482c6330 0x557f482c8760 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482b62d0 0x557f482b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1841759874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482b5bf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 >> 172.21.15.87:0/1841759874 conn(0x557f48273c90 msgr2=0x557f482a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.931+0000 7fcf56ffd640 0 client.25796 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.935+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 shutdown_connections 2024-04-22T13:34:49.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.937+0000 7fcf80d89740 1 -- 172.21.15.87:0/1841759874 wait complete. 2024-04-22T13:34:49.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.937+0000 7fcf80d89740 20 client.25796 trim_cache size 0 max 16384 2024-04-22T13:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.986+0000 7fcf80d89740 1 Processor -- start 2024-04-22T13:34:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.987+0000 7fcf80d89740 1 -- start start 2024-04-22T13:34:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.988+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f48273c90 0x557f482c1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.988+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482ac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.988+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.989+0000 7fcf80d89740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64010e10 con 0x557f482adac0 2024-04-22T13:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.989+0000 7fcf80d89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400e6e0 con 0x557f482acd50 2024-04-22T13:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.989+0000 7fcf80d89740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400c5e0 con 0x557f48273c90 2024-04-22T13:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.989+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.989+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38926/0 (socket says 172.21.15.87:38926) 2024-04-22T13:34:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.990+0000 7fcf77fff640 1 -- 172.21.15.87:0/1477458567 learned_addr learned my addr 172.21.15.87:0/1477458567 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.991+0000 7fcf77fff640 1 -- 172.21.15.87:0/1477458567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f48273c90 msgr2=0x557f482c1f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.991+0000 7fcf77fff640 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f48273c90 0x557f482c1f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.991+0000 7fcf77fff640 1 -- 172.21.15.87:0/1477458567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 msgr2=0x557f482ac270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.991+0000 7fcf77fff640 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482ac270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.991+0000 7fcf77fff640 1 -- 172.21.15.87:0/1477458567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f482a72a0 con 0x557f482adac0 2024-04-22T13:34:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.992+0000 7fcf77fff640 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00b990 tx=0x7fcf6c00ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b7f3e5bcff02b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:49.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:49 smithi087 ceph-mon[27999]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.31k op/s 2024-04-22T13:34:49.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:49 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:34:49.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:49 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:49.989 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:49 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:34:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.994+0000 7fcf5d7fa640 1 -- 172.21.15.87:0/1477458567 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c015070 con 0x557f482adac0 2024-04-22T13:34:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.994+0000 7fcf5d7fa640 1 -- 172.21.15.87:0/1477458567 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf6c004040 con 0x557f482adac0 2024-04-22T13:34:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.997+0000 7fcf80d89740 1 -- 172.21.15.87:0/1477458567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 msgr2=0x557f482b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:49.997+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00b990 tx=0x7fcf6c00ba20 comp rx=0 tx=0).stop 2024-04-22T13:34:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.001+0000 7fcf80d89740 1 -- 172.21.15.87:0/1477458567 shutdown_connections 2024-04-22T13:34:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.001+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f482b56b0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.001+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f482acd50 0x557f482ac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.002+0000 7fcf80d89740 1 --2- 172.21.15.87:0/1477458567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f48273c90 0x557f482c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:49.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.002+0000 7fcf80d89740 1 -- 172.21.15.87:0/1477458567 >> 172.21.15.87:0/1477458567 conn(0x557f482b3620 msgr2=0x557f482b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:50.000 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.005+0000 7fcf80d89740 1 -- 172.21.15.87:0/1477458567 shutdown_connections 2024-04-22T13:34:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.011+0000 7fcf80d89740 1 -- 172.21.15.87:0/1477458567 wait complete. 2024-04-22T13:34:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.012+0000 7fcf80d89740 1 -- start start 2024-04-22T13:34:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.012+0000 7fcf80d89740 1 Processor -- start 2024-04-22T13:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.013+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f48273c90 0x557f482ba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.013+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf80d89740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f4832b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf80d89740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400cba0 con 0x557f482adac0 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf80d89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400e010 con 0x557f48273c90 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf80d89740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400d270 con 0x557f482acd50 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f4832b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf77fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f4832b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38928/0 (socket says 172.21.15.87:38928) 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf77fff640 1 -- 172.21.15.87:0/30526815 learned_addr learned my addr 172.21.15.87:0/30526815 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf80d89740 10 client.0 _ll_register_callbacks cb 0x7fcf6800a680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:34:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.014+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.015+0000 7fcf76ffd640 1 -- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f48273c90 msgr2=0x557f482ba1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.015+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f48273c90 0x557f482ba1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.015+0000 7fcf76ffd640 1 -- 172.21.15.87:0/30526815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 msgr2=0x557f4832b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.015+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f4832b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.015+0000 7fcf76ffd640 1 -- 172.21.15.87:0/30526815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f482bafc0 con 0x557f482acd50 2024-04-22T13:34:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.035+0000 7fcf76ffd640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ba6f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcf700baf10 tx=0x7fcf700df040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a1cdefd52769767 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:34:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.074+0000 7fcf56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:34:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.074+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf700c2e60 con 0x557f482acd50 2024-04-22T13:34:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.074+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcf700bb120 con 0x557f482acd50 2024-04-22T13:34:50.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.074+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf6c004040 con 0x557f482acd50 2024-04-22T13:34:50.074 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.079+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcf700c7690 con 0x557f482acd50 2024-04-22T13:34:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.081+0000 7fcf80d89740 10 client.25825 fetch_fsmap learned FSMap version 21 2024-04-22T13:34:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.081+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557f4832cf70 con 0x557f482acd50 2024-04-22T13:34:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.089+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf6c004040 con 0x557f482acd50 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.089+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcf7003c030 con 0x557f482acd50 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.090+0000 7fcf80d89740 10 client.25825 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.091+0000 7fcf80d89740 10 client.25825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.091+0000 7fcf80d89740 20 client.25825 populate_metadata read hostname 'smithi087' 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.091+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482b62d0 0x557f4832f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.091+0000 7fcf80d89740 4 client.25825 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:34:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.091+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f482aca40 con 0x557f482b62d0 2024-04-22T13:34:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.095+0000 7fcf777fe640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482b62d0 0x557f4832f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:34:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.104+0000 7fcf777fe640 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482b62d0 0x557f4832f6e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b09944037a23988 server_cookie=ca8f68e0702292a9 in_seq=0 out_seq=0 2024-04-22T13:34:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:50.106+0000 7fcf56ffd640 10 client.25825 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:50.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:49 smithi120 ceph-mon[31101]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.31k op/s 2024-04-22T13:34:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:49 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:34:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:49 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:49 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:34:50.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:49 smithi132 ceph-mon[31531]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 964 MiB/s wr, 2.31k op/s 2024-04-22T13:34:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:49 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:34:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:49 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:50.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:49 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:34:51.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:50 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:51.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:50 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:51.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:50 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:34:52.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:51 smithi120 ceph-mon[31101]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 884 MiB/s wr, 1.98k op/s 2024-04-22T13:34:52.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:51 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:34:52.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:51 smithi132 ceph-mon[31531]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 884 MiB/s wr, 1.98k op/s 2024-04-22T13:34:52.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:51 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:34:52.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:51 smithi087 ceph-mon[27999]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 884 MiB/s wr, 1.98k op/s 2024-04-22T13:34:52.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:51 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:34:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.621+0000 7fcf56ffd640 1 -- 172.21.15.87:0/30526815 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f482aca40 con 0x557f482b62d0 2024-04-22T13:34:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.621+0000 7fcf56ffd640 10 client.25825 handle_command_reply: tid=0 2024-04-22T13:34:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.621+0000 7fcf80d89740 1 client.25825 shutdown 2024-04-22T13:34:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 msgr2=0x557f482ba6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ba6f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcf700baf10 tx=0x7fcf700df040 comp rx=0 tx=0).stop 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 shutdown_connections 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557f482b62d0 0x557f4832f6e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557f482adac0 0x557f4832b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557f482acd50 0x557f482ba6f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 --2- 172.21.15.87:0/30526815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f48273c90 0x557f482ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 >> 172.21.15.87:0/30526815 conn(0x557f482b3620 msgr2=0x557f482a6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf56ffd640 0 client.25825 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.622+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 shutdown_connections 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.623+0000 7fcf80d89740 1 -- 172.21.15.87:0/30526815 wait complete. 2024-04-22T13:34:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.623+0000 7fcf80d89740 20 client.25825 trim_cache size 0 max 16384 2024-04-22T13:34:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:34:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:34:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:34:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:34:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.626+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 msgr2=0x7fcf600a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.626+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 0x7fcf600a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557f482ae220 tx=0x7fcf68008040 comp rx=0 tx=0).stop 2024-04-22T13:34:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.626+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 msgr2=0x7fcf7811f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:34:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.626+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf70037040 tx=0x7fcf70035ce0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 shutdown_connections 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcf600a3f20 0x7fcf600a63e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf78114c90 0x7fcf78119f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf78109520 0x7fcf7811f3b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 --2- 172.21.15.87:0/623402308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf7806e240 0x7fcf7811ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:34:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.630+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 >> 172.21.15.87:0/623402308 conn(0x7fcf7806ce90 msgr2=0x7fcf78070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:34:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.632+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 shutdown_connections 2024-04-22T13:34:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:34:52.632+0000 7fcf7cf34640 1 -- 172.21.15.87:0/623402308 wait complete. 2024-04-22T13:34:54.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:53 smithi120 ceph-mon[31101]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.63k op/s 2024-04-22T13:34:54.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:53 smithi132 ceph-mon[31531]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.63k op/s 2024-04-22T13:34:54.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:53 smithi087 ceph-mon[27999]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 742 MiB/s wr, 1.63k op/s 2024-04-22T13:34:56.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:55 smithi120 ceph-mon[31101]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.45k op/s 2024-04-22T13:34:56.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:55 smithi132 ceph-mon[31531]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.45k op/s 2024-04-22T13:34:56.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:55 smithi087 ceph-mon[27999]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.1 GiB/s wr, 2.45k op/s 2024-04-22T13:34:58.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:57 smithi132 ceph-mon[31531]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 739 MiB/s wr, 1.61k op/s 2024-04-22T13:34:58.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:57 smithi087 ceph-mon[27999]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 739 MiB/s wr, 1.61k op/s 2024-04-22T13:34:58.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:57 smithi120 ceph-mon[31101]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 739 MiB/s wr, 1.61k op/s 2024-04-22T13:35:00.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:34:59 smithi132 ceph-mon[31531]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 964 MiB/s wr, 2.35k op/s 2024-04-22T13:35:00.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:34:59 smithi087 ceph-mon[27999]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 964 MiB/s wr, 2.35k op/s 2024-04-22T13:35:00.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:34:59 smithi120 ceph-mon[31101]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 964 MiB/s wr, 2.35k op/s 2024-04-22T13:35:01.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:01 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:35:01.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:01 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:35:01.491 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:01 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:35:02.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:02 smithi120 ceph-mon[31101]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 879 MiB/s wr, 2.30k op/s 2024-04-22T13:35:02.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:02 smithi132 ceph-mon[31531]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 879 MiB/s wr, 2.30k op/s 2024-04-22T13:35:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:02 smithi087 ceph-mon[27999]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 879 MiB/s wr, 2.30k op/s 2024-04-22T13:35:03.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:03 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:35:03.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:03 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:35:03.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:03 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:35:03.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:03 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:35:03.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:03 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:35:03.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:03 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:35:03.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:03 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:35:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:03 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:35:03.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:03 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:35:04.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:04 smithi120 ceph-mon[31101]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 741 MiB/s wr, 2.00k op/s 2024-04-22T13:35:04.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:04 smithi132 ceph-mon[31531]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 741 MiB/s wr, 2.00k op/s 2024-04-22T13:35:04.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:04 smithi087 ceph-mon[27999]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 741 MiB/s wr, 2.00k op/s 2024-04-22T13:35:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 -- 172.21.15.87:0/362905065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00b7aa0 msgr2=0x7f34f00b9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 --2- 172.21.15.87:0/362905065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00b7aa0 0x7f34f00b9e90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009900 tx=0x7f34ec02bc50 comp rx=0 tx=0).stop 2024-04-22T13:35:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 -- 172.21.15.87:0/362905065 shutdown_connections 2024-04-22T13:35:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 --2- 172.21.15.87:0/362905065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00b7aa0 0x7f34f00b9e90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 --2- 172.21.15.87:0/362905065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 0x7f34f00b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 --2- 172.21.15.87:0/362905065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00a4e10 0x7f34f00a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.902+0000 7f34fd69a640 1 -- 172.21.15.87:0/362905065 >> 172.21.15.87:0/362905065 conn(0x7f34f001aaa0 msgr2=0x7f34f001aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.904+0000 7f34fd69a640 1 -- 172.21.15.87:0/362905065 shutdown_connections 2024-04-22T13:35:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.904+0000 7f34fd69a640 1 -- 172.21.15.87:0/362905065 wait complete. 2024-04-22T13:35:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 Processor -- start 2024-04-22T13:35:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 -- start start 2024-04-22T13:35:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 0x7f34f00b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00b3c00 0x7f34f00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34f00ad210 con 0x7f34f00a4e10 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34f00ad090 con 0x7f34f00b3c00 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.907+0000 7f34fd69a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34f00ad390 con 0x7f34f00a5730 2024-04-22T13:35:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.908+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.908+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46256/0 (socket says 172.21.15.87:46256) 2024-04-22T13:35:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.908+0000 7f34f6ffd640 1 -- 172.21.15.87:0/1594861562 learned_addr learned my addr 172.21.15.87:0/1594861562 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f77fe640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00b3c00 0x7f34f00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f67fc640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 0x7f34f00b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f6ffd640 1 -- 172.21.15.87:0/1594861562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 msgr2=0x7f34f00b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 0x7f34f00b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f6ffd640 1 -- 172.21.15.87:0/1594861562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00b3c00 msgr2=0x7f34f00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00b3c00 0x7f34f00b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.910+0000 7f34f6ffd640 1 -- 172.21.15.87:0/1594861562 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f00b8350 con 0x7f34f00a4e10 2024-04-22T13:35:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.911+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f34e800b450 tx=0x7f34e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.918+0000 7f34dffff640 1 -- 172.21.15.87:0/1594861562 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8015020 con 0x7f34f00a4e10 2024-04-22T13:35:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.918+0000 7f34dffff640 1 -- 172.21.15.87:0/1594861562 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e8011d40 con 0x7f34f00a4e10 2024-04-22T13:35:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.918+0000 7f34dffff640 1 -- 172.21.15.87:0/1594861562 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e800f340 con 0x7f34f00a4e10 2024-04-22T13:35:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.919+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f00b85e0 con 0x7f34f00a4e10 2024-04-22T13:35:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.919+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f0158f80 con 0x7f34f00a4e10 2024-04-22T13:35:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.923+0000 7f34dffff640 1 -- 172.21.15.87:0/1594861562 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f34e8018020 con 0x7f34f00a4e10 2024-04-22T13:35:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.924+0000 7f34dffff640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 0x7f34e00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.924+0000 7f34dffff640 1 -- 172.21.15.87:0/1594861562 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34e80d0050 con 0x7f34f00a4e10 2024-04-22T13:35:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.925+0000 7f34f67fc640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 0x7f34e00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.929+0000 7f350036c740 1 Processor -- start 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.929+0000 7f350036c740 1 -- start start 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.929+0000 7f350036c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.929+0000 7f350036c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f350036c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f350036c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c7551558d0 con 0x55c7551f0d50 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f350036c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c7550f2e80 con 0x55c7551fa2d0 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f350036c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c755151890 con 0x55c7551f1ac0 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:44762/0 (socket says 172.21.15.87:44762) 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f34f6ffd640 1 -- 172.21.15.87:0/858979384 learned_addr learned my addr 172.21.15.87:0/858979384 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.930+0000 7f34f77fe640 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.935+0000 7f34f67fc640 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.935+0000 7f34f6ffd640 1 -- 172.21.15.87:0/858979384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 msgr2=0x55c7551fc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.935+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551fc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.935+0000 7f34f6ffd640 1 -- 172.21.15.87:0/858979384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 msgr2=0x55c7551f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.935+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.936+0000 7f34f6ffd640 1 -- 172.21.15.87:0/858979384 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c755114320 con 0x55c7551f1ac0 2024-04-22T13:35:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.937+0000 7f34f67fc640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 0x7f34e00a62a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34f80657a0 tx=0x7f34f8067920 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.942+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f34e800be10 tx=0x7f34e80a0680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c9624a0f83a4f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.943+0000 7f34dd7fa640 1 -- 172.21.15.87:0/858979384 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8098880 con 0x55c7551f1ac0 2024-04-22T13:35:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.944+0000 7f34dd7fa640 1 -- 172.21.15.87:0/858979384 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e80a0170 con 0x55c7551f1ac0 2024-04-22T13:35:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.944+0000 7f34dd7fa640 1 -- 172.21.15.87:0/858979384 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e809b8c0 con 0x55c7551f1ac0 2024-04-22T13:35:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.945+0000 7f350036c740 1 -- 172.21.15.87:0/858979384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 msgr2=0x55c7551f0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.945+0000 7f350036c740 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f34e800be10 tx=0x7f34e80a0680 comp rx=0 tx=0).stop 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 -- 172.21.15.87:0/858979384 shutdown_connections 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 --2- 172.21.15.87:0/858979384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f0810 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 -- 172.21.15.87:0/858979384 >> 172.21.15.87:0/858979384 conn(0x55c7551b7c90 msgr2=0x55c7551eba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 -- 172.21.15.87:0/858979384 shutdown_connections 2024-04-22T13:35:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.947+0000 7f350036c740 1 -- 172.21.15.87:0/858979384 wait complete. 2024-04-22T13:35:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.948+0000 7f350036c740 1 -- start start 2024-04-22T13:35:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.948+0000 7f350036c740 1 Processor -- start 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.948+0000 7f350036c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.948+0000 7f350036c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.948+0000 7f350036c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f350036c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c7551fd000 con 0x55c7551f0d50 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f350036c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c7550ddeb0 con 0x55c7551fa2d0 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f350036c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c7551fd180 con 0x55c7551f1ac0 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f350036c740 10 client.0 _ll_register_callbacks cb 0x55c75510ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f34f77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f34f77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:40774/0 (socket says 172.21.15.87:40774) 2024-04-22T13:35:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.949+0000 7f34f77fe640 1 -- 172.21.15.87:0/1444352925 learned_addr learned my addr 172.21.15.87:0/1444352925 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.950+0000 7f34f67fc640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.946 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.950+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 -- 172.21.15.87:0/1444352925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 msgr2=0x55c7551f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 -- 172.21.15.87:0/1444352925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 msgr2=0x55c7551f9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 -- 172.21.15.87:0/1444352925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c754e68210 con 0x55c7551fa2d0 2024-04-22T13:35:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34f77fe640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34f0018b70 tx=0x7f34ec02c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c0676a7489715da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.951+0000 7f34daff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.952+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec045070 con 0x55c7551fa2d0 2024-04-22T13:35:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.952+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34ec02cba0 con 0x55c7551fa2d0 2024-04-22T13:35:05.949 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.953+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e80a0170 con 0x55c7551fa2d0 2024-04-22T13:35:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.954+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f34ec04f050 con 0x55c7551fa2d0 2024-04-22T13:35:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.954+0000 7f350036c740 10 client.25802 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:05.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.954+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c754f34620 con 0x55c7551fa2d0 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e80a0170 con 0x55c7551fa2d0 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f34ec03f820 con 0x55c7551fa2d0 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 10 client.25802 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 20 client.25802 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c75520a1d0 0x55c75520c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:05.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 4 client.25802 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:05.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.955+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c755169880 con 0x55c75520a1d0 2024-04-22T13:35:05.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.959+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c75520a1d0 0x55c75520c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.974+0000 7f34daff5640 10 client.25802 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.974+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c75520a1d0 0x55c75520c5c0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb621b35be4c018b server_cookie=7326d5c3a89e935f in_seq=0 out_seq=0 2024-04-22T13:35:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.981+0000 7f34daff5640 1 -- 172.21.15.87:0/1444352925 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c755169880 con 0x55c75520a1d0 2024-04-22T13:35:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.981+0000 7f34daff5640 10 client.25802 handle_command_reply: tid=0 2024-04-22T13:35:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.981+0000 7f350036c740 1 client.25802 shutdown 2024-04-22T13:35:05.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.989+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 msgr2=0x55c7551f7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:05.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.989+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34f0018b70 tx=0x7f34ec02c820 comp rx=0 tx=0).stop 2024-04-22T13:35:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 shutdown_connections 2024-04-22T13:35:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c75520a1d0 0x55c75520c5c0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fa2d0 0x55c7551f7650 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7551f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 --2- 172.21.15.87:0/1444352925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c7551f9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f34daff5640 0 client.25802 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.993+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 >> 172.21.15.87:0/1444352925 conn(0x55c7551b7c90 msgr2=0x55c7551ed790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.994+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 shutdown_connections 2024-04-22T13:35:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.994+0000 7f350036c740 1 -- 172.21.15.87:0/1444352925 wait complete. 2024-04-22T13:35:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:05.995+0000 7f350036c740 20 client.25802 trim_cache size 0 max 16384 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 Processor -- start 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 -- start start 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551f1ac0 0x55c7551f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551fa2d0 0x55c7551ed5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.032+0000 7f350036c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34e4010ee0 con 0x55c7551f0d50 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f350036c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34e400e7b0 con 0x55c7551f1ac0 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f350036c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34e400c600 con 0x55c7551fa2d0 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46276/0 (socket says 172.21.15.87:46276) 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 -- 172.21.15.87:0/808035477 learned_addr learned my addr 172.21.15.87:0/808035477 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f67fc640 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551f1ac0 0x55c7551f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 -- 172.21.15.87:0/808035477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551fa2d0 msgr2=0x55c7551ed5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551fa2d0 0x55c7551ed5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 -- 172.21.15.87:0/808035477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551f1ac0 msgr2=0x55c7551f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551f1ac0 0x55c7551f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.033+0000 7f34f6ffd640 1 -- 172.21.15.87:0/808035477 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c75520c930 con 0x55c7551f0d50 2024-04-22T13:35:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.034+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e809a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac194f80b30b741 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.034+0000 7f34dcff9640 1 -- 172.21.15.87:0/808035477 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e80b9070 con 0x55c7551f0d50 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.034+0000 7f34dcff9640 1 -- 172.21.15.87:0/808035477 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e8085070 con 0x55c7551f0d50 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.034+0000 7f34dcff9640 1 -- 172.21.15.87:0/808035477 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e80b4040 con 0x55c7551f0d50 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- 172.21.15.87:0/808035477 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 msgr2=0x55c755205f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e809a370 comp rx=0 tx=0).stop 2024-04-22T13:35:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- 172.21.15.87:0/808035477 shutdown_connections 2024-04-22T13:35:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551fa2d0 0x55c7551ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551f1ac0 0x55c7551f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 --2- 172.21.15.87:0/808035477 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551f0d50 0x55c755205f00 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- 172.21.15.87:0/808035477 >> 172.21.15.87:0/808035477 conn(0x55c7551b7c90 msgr2=0x55c7551ece90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- 172.21.15.87:0/808035477 shutdown_connections 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- 172.21.15.87:0/808035477 wait complete. 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 -- start start 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.036+0000 7f350036c740 1 Processor -- start 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7552026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 0x55c7551fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f34f67fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f34f67fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46288/0 (socket says 172.21.15.87:46288) 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f34f67fc640 1 -- 172.21.15.87:0/37386122 learned_addr learned my addr 172.21.15.87:0/37386122 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f34e400cc70 con 0x55c7551fa2d0 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34e400e0e0 con 0x55c7551fdf30 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34e400d340 con 0x55c7551f1ac0 2024-04-22T13:35:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.037+0000 7f350036c740 10 client.0 _ll_register_callbacks cb 0x7f34ec004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f77fe640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 0x55c7551fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 -- 172.21.15.87:0/37386122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 msgr2=0x55c7552026d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7552026d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 -- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 msgr2=0x55c7551fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 0x55c7551fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c75526f740 con 0x55c7551fa2d0 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f77fe640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 0x55c7551fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34f67fc640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f34f8063510 tx=0x7f34f8067730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7506b1cbc0ec8941 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.038+0000 7f34daff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.046+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8074040 con 0x55c7551fa2d0 2024-04-22T13:35:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.046+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f34f807d680 con 0x55c7551fa2d0 2024-04-22T13:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.046+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e8085070 con 0x55c7551fa2d0 2024-04-22T13:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.047+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f34f807d880 con 0x55c7551fa2d0 2024-04-22T13:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.047+0000 7f350036c740 10 client.16593 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.047+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c755270ac0 con 0x55c7551fa2d0 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.048+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e8085070 con 0x55c7551fa2d0 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.048+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f34f807a030 con 0x55c7551fa2d0 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.048+0000 7f350036c740 10 client.16593 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.048+0000 7f350036c740 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.049+0000 7f350036c740 20 client.16593 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.049+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c755271220 0x55c755273640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.049+0000 7f350036c740 4 client.16593 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.049+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7550f53d0 con 0x55c755271220 2024-04-22T13:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.050+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c755271220 0x55c755273640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.056+0000 7f34f6ffd640 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c755271220 0x55c755273640 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=289312e57d2f93c2 server_cookie=5b189a2d6c033db1 in_seq=0 out_seq=0 2024-04-22T13:35:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:06.056+0000 7f34daff5640 10 client.16593 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:06.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:06 smithi120 ceph-mon[31101]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.81k op/s 2024-04-22T13:35:06.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:06 smithi132 ceph-mon[31531]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.81k op/s 2024-04-22T13:35:06.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:06 smithi087 ceph-mon[27999]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.1 GiB/s wr, 2.81k op/s 2024-04-22T13:35:08.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:08 smithi120 ceph-mon[31101]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 742 MiB/s wr, 2.00k op/s 2024-04-22T13:35:08.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:08 smithi132 ceph-mon[31531]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 742 MiB/s wr, 2.00k op/s 2024-04-22T13:35:08.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:08 smithi087 ceph-mon[27999]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 742 MiB/s wr, 2.00k op/s 2024-04-22T13:35:10.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:10 smithi120 ceph-mon[31101]: pgmap v101: 97 pgs: 97 active+clean; 21 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 963 MiB/s wr, 3.12k op/s 2024-04-22T13:35:10.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:10 smithi132 ceph-mon[31531]: pgmap v101: 97 pgs: 97 active+clean; 21 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 963 MiB/s wr, 3.12k op/s 2024-04-22T13:35:10.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:10 smithi087 ceph-mon[27999]: pgmap v101: 97 pgs: 97 active+clean; 21 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 963 MiB/s wr, 3.12k op/s 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.792+0000 7f34daff5640 1 -- 172.21.15.87:0/37386122 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7550f53d0 con 0x55c755271220 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.792+0000 7f34daff5640 10 client.16593 handle_command_reply: tid=0 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.792+0000 7f350036c740 1 client.16593 shutdown 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 msgr2=0x55c7551fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f34f8063510 tx=0x7f34f8067730 comp rx=0 tx=0).stop 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 shutdown_connections 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c755271220 0x55c755273640 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c7551fdf30 0x55c7551fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c7551fa2d0 0x55c7551fd9f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 --2- 172.21.15.87:0/37386122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c7551f1ac0 0x55c7552026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 >> 172.21.15.87:0/37386122 conn(0x55c7551b7c90 msgr2=0x55c7551ea250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f34daff5640 0 client.16593 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 shutdown_connections 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 1 -- 172.21.15.87:0/37386122 wait complete. 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.793+0000 7f350036c740 20 client.16593 trim_cache size 0 max 16384 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:35:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 msgr2=0x7f34e00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 0x7f34e00a62a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34f80657a0 tx=0x7f34f8067920 comp rx=0 tx=0).stop 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 msgr2=0x7f34f00b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f34e800b450 tx=0x7f34e800b910 comp rx=0 tx=0).stop 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 shutdown_connections 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f34e00a3de0 0x7f34e00a62a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f00b3c00 0x7f34f00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34f00a5730 0x7f34f00b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.795+0000 7f34fd69a640 1 --2- 172.21.15.87:0/1594861562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34f00a4e10 0x7f34f00b3180 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.796+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 >> 172.21.15.87:0/1594861562 conn(0x7f34f001aaa0 msgr2=0x7f34f00a4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.796+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 shutdown_connections 2024-04-22T13:35:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:10.796+0000 7f34fd69a640 1 -- 172.21.15.87:0/1594861562 wait complete. 2024-04-22T13:35:12.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:12 smithi120 ceph-mon[31101]: pgmap v102: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 872 MiB/s wr, 3.05k op/s 2024-04-22T13:35:12.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:12 smithi132 ceph-mon[31531]: pgmap v102: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 872 MiB/s wr, 3.05k op/s 2024-04-22T13:35:12.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:12 smithi087 ceph-mon[27999]: pgmap v102: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 872 MiB/s wr, 3.05k op/s 2024-04-22T13:35:14.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:14 smithi120 ceph-mon[31101]: pgmap v103: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 732 MiB/s wr, 2.62k op/s 2024-04-22T13:35:14.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:14 smithi132 ceph-mon[31531]: pgmap v103: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 732 MiB/s wr, 2.62k op/s 2024-04-22T13:35:14.490 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:14 smithi087 ceph-mon[27999]: pgmap v103: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 732 MiB/s wr, 2.62k op/s 2024-04-22T13:35:16.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:16 smithi120 ceph-mon[31101]: pgmap v104: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1018 MiB/s wr, 5.17k op/s 2024-04-22T13:35:16.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:16 smithi132 ceph-mon[31531]: pgmap v104: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1018 MiB/s wr, 5.17k op/s 2024-04-22T13:35:16.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:16 smithi087 ceph-mon[27999]: pgmap v104: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1018 MiB/s wr, 5.17k op/s 2024-04-22T13:35:18.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:18 smithi120 ceph-mon[31101]: pgmap v105: 97 pgs: 97 active+clean; 23 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 682 MiB/s wr, 4.73k op/s 2024-04-22T13:35:18.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:18 smithi132 ceph-mon[31531]: pgmap v105: 97 pgs: 97 active+clean; 23 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 682 MiB/s wr, 4.73k op/s 2024-04-22T13:35:18.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:18 smithi087 ceph-mon[27999]: pgmap v105: 97 pgs: 97 active+clean; 23 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 682 MiB/s wr, 4.73k op/s 2024-04-22T13:35:19.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:19 smithi120 ceph-mon[31101]: pgmap v106: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 MiB/s wr, 5.46k op/s 2024-04-22T13:35:19.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:19 smithi132 ceph-mon[31531]: pgmap v106: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 MiB/s wr, 5.46k op/s 2024-04-22T13:35:19.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:19 smithi087 ceph-mon[27999]: pgmap v106: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 883 MiB/s wr, 5.46k op/s 2024-04-22T13:35:19.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:35:19.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:35:20.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.009+0000 7f48e262a640 1 -- 172.21.15.87:0/1779713571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 msgr2=0x7f48dc115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.009+0000 7f48e262a640 1 --2- 172.21.15.87:0/1779713571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc115200 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009be0 tx=0x7f48d402bfa0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.014+0000 7f48e262a640 1 -- 172.21.15.87:0/1779713571 shutdown_connections 2024-04-22T13:35:20.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.014+0000 7f48e262a640 1 --2- 172.21.15.87:0/1779713571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc115200 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.014+0000 7f48e262a640 1 --2- 172.21.15.87:0/1779713571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc074c10 0x7f48dc1128d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.014+0000 7f48e262a640 1 --2- 172.21.15.87:0/1779713571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48dc0742f0 0x7f48dc0746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.014+0000 7f48e262a640 1 -- 172.21.15.87:0/1779713571 >> 172.21.15.87:0/1779713571 conn(0x7f48dc0fd8f0 msgr2=0x7f48dc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.016+0000 7f48e262a640 1 -- 172.21.15.87:0/1779713571 shutdown_connections 2024-04-22T13:35:20.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.016+0000 7f48e262a640 1 -- 172.21.15.87:0/1779713571 wait complete. 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.018+0000 7f48e262a640 1 Processor -- start 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.018+0000 7f48e262a640 1 -- start start 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0742f0 0x7f48dc0737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48dc074c10 0x7f48dc06c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc118700 con 0x7f48dc112e10 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc118580 con 0x7f48dc0742f0 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.019+0000 7f48e262a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc118880 con 0x7f48dc074c10 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53868/0 (socket says 172.21.15.87:53868) 2024-04-22T13:35:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 -- 172.21.15.87:0/2069126658 learned_addr learned my addr 172.21.15.87:0/2069126658 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 -- 172.21.15.87:0/2069126658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48dc074c10 msgr2=0x7f48dc06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48dc074c10 0x7f48dc06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 -- 172.21.15.87:0/2069126658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0742f0 msgr2=0x7f48dc0737b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0742f0 0x7f48dc0737b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.020+0000 7f48e1e29640 1 -- 172.21.15.87:0/2069126658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc06d3d0 con 0x7f48dc112e10 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.021+0000 7f48e1e29640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f48d4033420 tx=0x7f48d40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.021+0000 7f48d27fc640 1 -- 172.21.15.87:0/2069126658 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4043020 con 0x7f48dc112e10 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.021+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc06d660 con 0x7f48dc112e10 2024-04-22T13:35:20.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.021+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc071640 con 0x7f48dc112e10 2024-04-22T13:35:20.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.022+0000 7f48d27fc640 1 -- 172.21.15.87:0/2069126658 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d40046f0 con 0x7f48dc112e10 2024-04-22T13:35:20.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.023+0000 7f48d27fc640 1 -- 172.21.15.87:0/2069126658 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4043020 con 0x7f48dc112e10 2024-04-22T13:35:20.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.027+0000 7f48e6481740 1 Processor -- start 2024-04-22T13:35:20.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.029+0000 7f48d27fc640 1 -- 172.21.15.87:0/2069126658 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48d4046020 con 0x7f48dc112e10 2024-04-22T13:35:20.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 -- start start 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441e1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x5642441ecff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5642441461e0 con 0x5642441e2020 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642440e3390 con 0x5642441e1700 2024-04-22T13:35:20.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.030+0000 7f48e6481740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5642441421a0 con 0x5642441eac00 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.031+0000 7f48d27fc640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 0x7f48c80a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.031+0000 7f48d27fc640 1 -- 172.21.15.87:0/2069126658 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f48d403a040 con 0x7f48dc112e10 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.031+0000 7f48e1628640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.031+0000 7f48e1628640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53876/0 (socket says 172.21.15.87:53876) 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.031+0000 7f48e1628640 1 -- 172.21.15.87:0/685759479 learned_addr learned my addr 172.21.15.87:0/685759479 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.032+0000 7f48e0e27640 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441e1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 -- 172.21.15.87:0/685759479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 msgr2=0x5642441ecff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x5642441ecff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 -- 172.21.15.87:0/685759479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 msgr2=0x5642441e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441e1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 -- 172.21.15.87:0/685759479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564243f227d0 con 0x5642441e2020 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1e29640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 0x7f48c80a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.033+0000 7f48e1628640 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ba80 tx=0x7f48cc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca56584cac58834 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.034+0000 7f48c77fe640 1 -- 172.21.15.87:0/685759479 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc016070 con 0x5642441e2020 2024-04-22T13:35:20.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.034+0000 7f48c77fe640 1 -- 172.21.15.87:0/685759479 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48cc0054d0 con 0x5642441e2020 2024-04-22T13:35:20.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.035+0000 7f48e6481740 1 -- 172.21.15.87:0/685759479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 msgr2=0x5642441ea6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.035+0000 7f48e6481740 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ba80 tx=0x7f48cc00bf40 comp rx=0 tx=0).stop 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.036+0000 7f48e1e29640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 0x7f48c80a6620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005bc0 tx=0x7f48d40048a0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.037+0000 7f48e6481740 1 -- 172.21.15.87:0/685759479 shutdown_connections 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.037+0000 7f48e6481740 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x5642441ecff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.037+0000 7f48e6481740 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441ea6c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.037+0000 7f48e6481740 1 --2- 172.21.15.87:0/685759479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441e1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.037+0000 7f48e6481740 1 -- 172.21.15.87:0/685759479 >> 172.21.15.87:0/685759479 conn(0x5642441dbe30 msgr2=0x5642441dc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.040+0000 7f48e6481740 1 -- 172.21.15.87:0/685759479 shutdown_connections 2024-04-22T13:35:20.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.040+0000 7f48e6481740 1 -- 172.21.15.87:0/685759479 wait complete. 2024-04-22T13:35:20.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.040+0000 7f48e6481740 1 -- start start 2024-04-22T13:35:20.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.040+0000 7f48e6481740 1 Processor -- start 2024-04-22T13:35:20.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441eac00 0x5642441e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5642441eda90 con 0x5642441e2020 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642440ff2f0 con 0x5642441eac00 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5642441edc10 con 0x5642441e1700 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e6481740 10 client.0 _ll_register_callbacks cb 0x56424410c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.041+0000 7f48e1628640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53882/0 (socket says 172.21.15.87:53882) 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 -- 172.21.15.87:0/3889688076 learned_addr learned my addr 172.21.15.87:0/3889688076 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 -- 172.21.15.87:0/3889688076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 msgr2=0x5642441e7d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441e7d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 -- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441eac00 msgr2=0x5642441e87b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441eac00 0x5642441e87b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.042+0000 7f48e1628640 1 -- 172.21.15.87:0/3889688076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564243e583f0 con 0x5642441e2020 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.043+0000 7f48e1628640 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b580 tx=0x7f48cc00dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9964006286ee79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.043+0000 7f48c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:20.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.043+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc016030 con 0x5642441e2020 2024-04-22T13:35:20.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.043+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564244153c00 con 0x5642441e2020 2024-04-22T13:35:20.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.043+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f48cc005430 con 0x5642441e2020 2024-04-22T13:35:20.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.044+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f48cc005630 con 0x5642441e2020 2024-04-22T13:35:20.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.044+0000 7f48e6481740 10 client.16605 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:20.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.044+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564243e180f0 con 0x5642441e2020 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564244153c00 con 0x5642441e2020 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f48cc022070 con 0x5642441e2020 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 10 client.16605 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 10 client.16605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 20 client.16605 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642441fb330 0x5642441fd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 4 client.16605 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.045+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56424414da70 con 0x5642441fb330 2024-04-22T13:35:20.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.046+0000 7f48e0e27640 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642441fb330 0x5642441fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.049+0000 7f48e0e27640 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642441fb330 0x5642441fd770 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af2fdae520db3615 server_cookie=2a56348bd9cf8332 in_seq=0 out_seq=0 2024-04-22T13:35:20.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.049+0000 7f48c4ff9640 10 client.16605 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:20.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.054+0000 7f48c4ff9640 1 -- 172.21.15.87:0/3889688076 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56424414da70 con 0x5642441fb330 2024-04-22T13:35:20.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.054+0000 7f48c4ff9640 10 client.16605 handle_command_reply: tid=0 2024-04-22T13:35:20.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 client.16605 shutdown 2024-04-22T13:35:20.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 msgr2=0x5642441e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b580 tx=0x7f48cc00dd60 comp rx=0 tx=0).stop 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 shutdown_connections 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642441fb330 0x5642441fd770 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441eac00 0x5642441e87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e8270 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 --2- 172.21.15.87:0/3889688076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.055+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 >> 172.21.15.87:0/3889688076 conn(0x5642441dbe30 msgr2=0x5642441ddb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.056+0000 7f48c4ff9640 0 client.16605 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.056+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 shutdown_connections 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.056+0000 7f48e6481740 1 -- 172.21.15.87:0/3889688076 wait complete. 2024-04-22T13:35:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.056+0000 7f48e6481740 20 client.16605 trim_cache size 0 max 16384 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.090+0000 7f48e6481740 1 Processor -- start 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 -- start start 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f48c0010fb0 con 0x5642441e2020 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48c000e880 con 0x5642441e1700 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.091+0000 7f48e6481740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f48c000c6d0 con 0x5642441eac00 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.092+0000 7f48e1628640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.093+0000 7f48e0e27640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.093+0000 7f48e0e27640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:46592/0 (socket says 172.21.15.87:46592) 2024-04-22T13:35:20.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.093+0000 7f48e0e27640 1 -- 172.21.15.87:0/1893517093 learned_addr learned my addr 172.21.15.87:0/1893517093 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.094+0000 7f48e0e27640 1 -- 172.21.15.87:0/1893517093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 msgr2=0x5642441f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.094+0000 7f48e0e27640 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.094+0000 7f48e0e27640 1 -- 172.21.15.87:0/1893517093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 msgr2=0x5642441e85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.094+0000 7f48e0e27640 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.094+0000 7f48e0e27640 1 -- 172.21.15.87:0/1893517093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642441e1e50 con 0x5642441eac00 2024-04-22T13:35:20.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.096+0000 7f48e0e27640 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f48d80014f0 tx=0x7f48d8001d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619f4995bab47218 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.103+0000 7f48c6ffd640 1 -- 172.21.15.87:0/1893517093 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8015670 con 0x5642441eac00 2024-04-22T13:35:20.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.103+0000 7f48c6ffd640 1 -- 172.21.15.87:0/1893517093 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d8006d90 con 0x5642441eac00 2024-04-22T13:35:20.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.103+0000 7f48c6ffd640 1 -- 172.21.15.87:0/1893517093 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8007030 con 0x5642441eac00 2024-04-22T13:35:20.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.105+0000 7f48e6481740 1 -- 172.21.15.87:0/1893517093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 msgr2=0x564244203080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.105+0000 7f48e6481740 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f48d80014f0 tx=0x7f48d8001d50 comp rx=0 tx=0).stop 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.107+0000 7f48e6481740 1 -- 172.21.15.87:0/1893517093 shutdown_connections 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.107+0000 7f48e6481740 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441eac00 0x564244203080 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.107+0000 7f48e6481740 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441e2020 0x5642441e85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.107+0000 7f48e6481740 1 --2- 172.21.15.87:0/1893517093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e1700 0x5642441f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.107+0000 7f48e6481740 1 -- 172.21.15.87:0/1893517093 >> 172.21.15.87:0/1893517093 conn(0x5642441dbe30 msgr2=0x5642441da640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.108+0000 7f48e6481740 1 -- 172.21.15.87:0/1893517093 shutdown_connections 2024-04-22T13:35:20.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.108+0000 7f48e6481740 1 -- 172.21.15.87:0/1893517093 wait complete. 2024-04-22T13:35:20.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.109+0000 7f48e6481740 1 -- start start 2024-04-22T13:35:20.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.109+0000 7f48e6481740 1 Processor -- start 2024-04-22T13:35:20.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441edf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441eac00 0x5642441ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f48c000cd40 con 0x5642441eac00 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48c000e1b0 con 0x5642441e2020 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f48c000d410 con 0x5642441e1700 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.110+0000 7f48e6481740 10 client.0 _ll_register_callbacks cb 0x7f48cc00b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:20.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.111+0000 7f48e1628640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.111+0000 7f48e1628640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48106/0 (socket says 172.21.15.87:48106) 2024-04-22T13:35:20.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.111+0000 7f48e1628640 1 -- 172.21.15.87:0/2824367290 learned_addr learned my addr 172.21.15.87:0/2824367290 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.112+0000 7f48e1628640 1 -- 172.21.15.87:0/2824367290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 msgr2=0x5642441edf00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:20.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.112+0000 7f48e1628640 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441edf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.112+0000 7f48e1628640 1 -- 172.21.15.87:0/2824367290 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441eac00 msgr2=0x5642441ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:20.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.112+0000 7f48e1628640 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441eac00 0x5642441ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.112+0000 7f48e1628640 1 -- 172.21.15.87:0/2824367290 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56424425fbb0 con 0x5642441e2020 2024-04-22T13:35:20.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.118+0000 7f48e1628640 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ddd0 tx=0x7f48cc00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af57a28087876ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.119+0000 7f48c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:20.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.119+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc031070 con 0x5642441e2020 2024-04-22T13:35:20.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.119+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48d8006d90 con 0x5642441e2020 2024-04-22T13:35:20.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.120+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f48cc004970 con 0x5642441e2020 2024-04-22T13:35:20.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.121+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f48cc03d050 con 0x5642441e2020 2024-04-22T13:35:20.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.121+0000 7f48e6481740 10 client.25808 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:20.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.121+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564244260ea0 con 0x5642441e2020 2024-04-22T13:35:20.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d8006d90 con 0x5642441e2020 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f48cc022040 con 0x5642441e2020 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 10 client.25808 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 10 client.25808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 20 client.25808 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642442615c0 0x564244263a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 4 client.25808 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:35:20.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.123+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642440c9590 con 0x5642442615c0 2024-04-22T13:35:20.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.130+0000 7f48e0e27640 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642442615c0 0x564244263a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.139+0000 7f48e0e27640 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642442615c0 0x564244263a20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae30e96161aaf3e server_cookie=df29af52d6c79c64 in_seq=0 out_seq=0 2024-04-22T13:35:20.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.140+0000 7f48c4ff9640 10 client.25808 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:20.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.149+0000 7f48c4ff9640 1 -- 172.21.15.87:0/2824367290 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5642440c9590 con 0x5642442615c0 2024-04-22T13:35:20.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.149+0000 7f48c4ff9640 10 client.25808 handle_command_reply: tid=0 2024-04-22T13:35:20.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.149+0000 7f48e6481740 1 client.25808 shutdown 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 msgr2=0x5642441ee440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00ddd0 tx=0x7f48cc00b260 comp rx=0 tx=0).stop 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 shutdown_connections 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5642442615c0 0x564244263a20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5642441eac00 0x5642441ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642441e2020 0x5642441ee440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 --2- 172.21.15.87:0/2824367290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5642441e1700 0x5642441edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 >> 172.21.15.87:0/2824367290 conn(0x5642441dbe30 msgr2=0x5642441daed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48c4ff9640 0 client.25808 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:20.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 shutdown_connections 2024-04-22T13:35:20.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.164+0000 7f48e6481740 1 -- 172.21.15.87:0/2824367290 wait complete. 2024-04-22T13:35:20.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.165+0000 7f48e6481740 20 client.25808 trim_cache size 0 max 16384 2024-04-22T13:35:20.170 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:20.170 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:35:20.170 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:35:20.170 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:20.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.177+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 msgr2=0x7f48c80a6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.177+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 0x7f48c80a6620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005bc0 tx=0x7f48d40048a0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.177+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 msgr2=0x7f48dc06cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.177+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f48d4033420 tx=0x7f48d40043b0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.180+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 shutdown_connections 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.181+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48c80a4160 0x7f48c80a6620 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.181+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48dc112e10 0x7f48dc06cdc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.181+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48dc074c10 0x7f48dc06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.181+0000 7f48e262a640 1 --2- 172.21.15.87:0/2069126658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48dc0742f0 0x7f48dc0737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.181+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 >> 172.21.15.87:0/2069126658 conn(0x7f48dc0fd8f0 msgr2=0x7f48dc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.185+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 shutdown_connections 2024-04-22T13:35:20.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.185+0000 7f48e262a640 1 -- 172.21.15.87:0/2069126658 wait complete. 2024-04-22T13:35:20.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:35:20.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:35:20.575 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.2 2024-04-22T13:35:20.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:35:20.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.593+0000 7f225b9ce640 1 -- 172.21.15.87:0/1466112453 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 msgr2=0x7f2254110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.593+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1466112453 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254110930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2244009860 tx=0x7f224402bc10 comp rx=0 tx=0).stop 2024-04-22T13:35:20.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.601+0000 7f225b9ce640 1 -- 172.21.15.87:0/1466112453 shutdown_connections 2024-04-22T13:35:20.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.601+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1466112453 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 0x7f225411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.601+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1466112453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.601+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1466112453 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254110930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.601+0000 7f225b9ce640 1 -- 172.21.15.87:0/1466112453 >> 172.21.15.87:0/1466112453 conn(0x7f225406ce80 msgr2=0x7f225406d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.609+0000 7f225b9ce640 1 -- 172.21.15.87:0/1466112453 shutdown_connections 2024-04-22T13:35:20.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.609+0000 7f225b9ce640 1 -- 172.21.15.87:0/1466112453 wait complete. 2024-04-22T13:35:20.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 Processor -- start 2024-04-22T13:35:20.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 -- start start 2024-04-22T13:35:20.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 0x7f2254119350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254119890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f22541137a0 con 0x7f225410eb40 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2254113620 con 0x7f225410e0c0 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.611+0000 7f225b9ce640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2254113920 con 0x7f2254110550 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.612+0000 7f2259f6b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254119890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.612+0000 7f225976a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.612+0000 7f225976a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48124/0 (socket says 172.21.15.87:48124) 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.612+0000 7f225976a640 1 -- 172.21.15.87:0/1814067495 learned_addr learned my addr 172.21.15.87:0/1814067495 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.612+0000 7f2258f69640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 0x7f2254119350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.613+0000 7f225976a640 1 -- 172.21.15.87:0/1814067495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 msgr2=0x7f2254119890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.613+0000 7f225976a640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254119890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.613+0000 7f225976a640 1 -- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 msgr2=0x7f2254119350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.613+0000 7f225976a640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 0x7f2254119350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.613+0000 7f225976a640 1 -- 172.21.15.87:0/1814067495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2254119fc0 con 0x7f225410e0c0 2024-04-22T13:35:20.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.618+0000 7f225976a640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2244034e00 tx=0x7f2244034a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.624+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2244041020 con 0x7f225410e0c0 2024-04-22T13:35:20.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.624+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225407a1b0 con 0x7f225410e0c0 2024-04-22T13:35:20.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.624+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225407a6a0 con 0x7f225410e0c0 2024-04-22T13:35:20.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.624+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2244003de0 con 0x7f225410e0c0 2024-04-22T13:35:20.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.624+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2244039040 con 0x7f225410e0c0 2024-04-22T13:35:20.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.628+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2238005290 con 0x7f225410e0c0 2024-04-22T13:35:20.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.633+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2244044020 con 0x7f225410e0c0 2024-04-22T13:35:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.634+0000 7f224a7fc640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 0x7f22400a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.635+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f22440f20d0 con 0x7f225410e0c0 2024-04-22T13:35:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.638+0000 7f2258f69640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 0x7f22400a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.640+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2244071020 con 0x7f225410e0c0 2024-04-22T13:35:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.644+0000 7f2258f69640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 0x7f22400a63f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f224c005db0 tx=0x7f224c005d40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:20.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.992+0000 7f99c4b50640 1 -- 172.21.15.87:0/336739667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01104d0 msgr2=0x7f99c01108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:20.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.992+0000 7f99c4b50640 1 --2- 172.21.15.87:0/336739667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01104d0 0x7f99c01108b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009900 tx=0x7f99b002bc80 comp rx=0 tx=0).stop 2024-04-22T13:35:20.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.993+0000 7f99c4b50640 1 -- 172.21.15.87:0/336739667 shutdown_connections 2024-04-22T13:35:20.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.993+0000 7f99c4b50640 1 --2- 172.21.15.87:0/336739667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010eb50 0x7f99c011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.993+0000 7f99c4b50640 1 --2- 172.21.15.87:0/336739667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010e030 0x7f99c010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:20.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.993+0000 7f99c4b50640 1 --2- 172.21.15.87:0/336739667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01104d0 0x7f99c01108b0 secure :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009900 tx=0x7f99b002bc80 comp rx=0 tx=0).stop 2024-04-22T13:35:20.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.993+0000 7f99c4b50640 1 -- 172.21.15.87:0/336739667 >> 172.21.15.87:0/336739667 conn(0x7f99c006ce20 msgr2=0x7f99c006d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:20.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.996+0000 7f99c4b50640 1 -- 172.21.15.87:0/336739667 shutdown_connections 2024-04-22T13:35:20.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.996+0000 7f99c4b50640 1 -- 172.21.15.87:0/336739667 wait complete. 2024-04-22T13:35:20.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.997+0000 7f99c4b50640 1 Processor -- start 2024-04-22T13:35:20.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.998+0000 7f99c4b50640 1 -- start start 2024-04-22T13:35:20.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.998+0000 7f99c4b50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010e030 0x7f99c01ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.998+0000 7f99c4b50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 0x7f99c01ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.999+0000 7f99c4b50640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 0x7f99c01a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:20.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.999+0000 7f99c4b50640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f99c0120f20 con 0x7f99c01a7590 2024-04-22T13:35:20.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.999+0000 7f99c4b50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99c0120da0 con 0x7f99c010eb50 2024-04-22T13:35:20.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:20.999+0000 7f99c4b50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01210a0 con 0x7f99c010e030 2024-04-22T13:35:21.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.003+0000 7f99beffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 0x7f99c01ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:21.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.004+0000 7f99bffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 0x7f99c01a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:21.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.004+0000 7f99bffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 0x7f99c01a7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53924/0 (socket says 172.21.15.87:53924) 2024-04-22T13:35:21.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.004+0000 7f99bffff640 1 -- 172.21.15.87:0/1971562606 learned_addr learned my addr 172.21.15.87:0/1971562606 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:21.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.004+0000 7f99bf7fe640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010e030 0x7f99c01ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:21.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.009+0000 7f99beffd640 1 -- 172.21.15.87:0/1971562606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010e030 msgr2=0x7f99c01ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.009+0000 7f99beffd640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010e030 0x7f99c01ad3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.009+0000 7f99beffd640 1 -- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 msgr2=0x7f99c01a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.009+0000 7f99beffd640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 0x7f99c01a7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.009+0000 7f99beffd640 1 -- 172.21.15.87:0/1971562606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c01a80b0 con 0x7f99c010eb50 2024-04-22T13:35:21.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.012+0000 7f99beffd640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 0x7f99c01ad920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99b800be20 tx=0x7f99b8009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.023+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8019070 con 0x7f99c010eb50 2024-04-22T13:35:21.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.023+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99b80148f0 con 0x7f99c010eb50 2024-04-22T13:35:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.024+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8012920 con 0x7f99c010eb50 2024-04-22T13:35:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.027+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c01a83a0 con 0x7f99c010eb50 2024-04-22T13:35:21.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.027+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c01b45a0 con 0x7f99c010eb50 2024-04-22T13:35:21.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.030+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c0068c90 con 0x7f99c010eb50 2024-04-22T13:35:21.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.040+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f99b8014440 con 0x7f99c010eb50 2024-04-22T13:35:21.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.052+0000 7f99bcff9640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 0x7f99a80a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:21.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.052+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f99b80d0bd0 con 0x7f99c010eb50 2024-04-22T13:35:21.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.053+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99b8022070 con 0x7f99c010eb50 2024-04-22T13:35:21.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.054+0000 7f99bf7fe640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 0x7f99a80a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:21.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.054+0000 7f99bf7fe640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 0x7f99a80a6360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99b0036040 tx=0x7f99b0000bc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:21.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:21 smithi120 ceph-mon[31101]: pgmap v107: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 793 MiB/s wr, 4.79k op/s 2024-04-22T13:35:21.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:21 smithi132 ceph-mon[31531]: pgmap v107: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 793 MiB/s wr, 4.79k op/s 2024-04-22T13:35:21.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:21 smithi087 ceph-mon[27999]: pgmap v107: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 793 MiB/s wr, 4.79k op/s 2024-04-22T13:35:21.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.512+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2238005580 con 0x7f225410e0c0 2024-04-22T13:35:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:35:21.515 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:35:21.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.516+0000 7f224a7fc640 1 -- 172.21.15.87:0/1814067495 <== mon.2 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f22440be920 con 0x7f225410e0c0 2024-04-22T13:35:21.515 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:35:21.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.527+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 msgr2=0x7f22400a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.527+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 0x7f22400a63f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f224c005db0 tx=0x7f224c005d40 comp rx=0 tx=0).stop 2024-04-22T13:35:21.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.527+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 msgr2=0x7f225411d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.527+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2244034e00 tx=0x7f2244034a50 comp rx=0 tx=0).stop 2024-04-22T13:35:21.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 shutdown_connections 2024-04-22T13:35:21.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f22400a3f30 0x7f22400a63f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2254110550 0x7f2254119890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f225410eb40 0x7f2254119350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 --2- 172.21.15.87:0/1814067495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f225410e0c0 0x7f225411d2c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.531+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 >> 172.21.15.87:0/1814067495 conn(0x7f225406ce80 msgr2=0x7f2254071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:21.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.537+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 shutdown_connections 2024-04-22T13:35:21.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.537+0000 7f225b9ce640 1 -- 172.21.15.87:0/1814067495 wait complete. 2024-04-22T13:35:21.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:35:21.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.753+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99c01a8ac0 con 0x7f99c010eb50 2024-04-22T13:35:21.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.756+0000 7f99bcff9640 1 -- 172.21.15.87:0/1971562606 <== mon.2 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11872 (secure 0 0 0) 0x7f99b809d470 con 0x7f99c010eb50 2024-04-22T13:35:21.763 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:35:21.764 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:31:50.833203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1119314155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1119314155},{"type":"v1","addr":"172.21.15.132:6835","nonce":1119314155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:35:21.764 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 21 2024-04-22T13:35:21.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 msgr2=0x7f99a80a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 0x7f99a80a6360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f99b0036040 tx=0x7f99b0000bc0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 msgr2=0x7f99c01ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:21.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 0x7f99c01ad920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99b800be20 tx=0x7f99b8009f70 comp rx=0 tx=0).stop 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 shutdown_connections 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99a80a3ea0 0x7f99a80a6360 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99c01a7590 0x7f99c01a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99c010eb50 0x7f99c01ad920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 --2- 172.21.15.87:0/1971562606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99c010e030 0x7f99c01ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 >> 172.21.15.87:0/1971562606 conn(0x7f99c006ce20 msgr2=0x7f99c0071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.779+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 shutdown_connections 2024-04-22T13:35:21.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:21.780+0000 7f99c4b50640 1 -- 172.21.15.87:0/1971562606 wait complete. 2024-04-22T13:35:21.813 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 71.2 --await' 2024-04-22T13:35:21.813 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 71.2 --await 2024-04-22T13:35:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.126+0000 7f23fdac1640 1 -- 172.21.15.87:0/353400732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f81105f0 msgr2=0x7f23f81109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.126+0000 7f23fdac1640 1 --2- 172.21.15.87:0/353400732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f81105f0 0x7f23f81109d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009900 tx=0x7f23e802bcf0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.129+0000 7f23fdac1640 1 -- 172.21.15.87:0/353400732 shutdown_connections 2024-04-22T13:35:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.129+0000 7f23fdac1640 1 --2- 172.21.15.87:0/353400732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81099d0 0x7f23f811d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.129+0000 7f23fdac1640 1 --2- 172.21.15.87:0/353400732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f8108f50 0x7f23f81093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.129+0000 7f23fdac1640 1 --2- 172.21.15.87:0/353400732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f81105f0 0x7f23f81109d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.129+0000 7f23fdac1640 1 -- 172.21.15.87:0/353400732 >> 172.21.15.87:0/353400732 conn(0x7f23f806ced0 msgr2=0x7f23f806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.134+0000 7f23fdac1640 1 -- 172.21.15.87:0/353400732 shutdown_connections 2024-04-22T13:35:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.134+0000 7f23fdac1640 1 -- 172.21.15.87:0/353400732 wait complete. 2024-04-22T13:35:22.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.135+0000 7f23fdac1640 1 Processor -- start 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 -- start start 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81099d0 0x7f23f81163b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81105f0 0x7f23f81168f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f23f8120f20 con 0x7f23f8108f50 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23f8120da0 con 0x7f23f81099d0 2024-04-22T13:35:22.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.136+0000 7f23fdac1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81210a0 con 0x7f23f81105f0 2024-04-22T13:35:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.137+0000 7f23fcabf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.137+0000 7f23fcabf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53954/0 (socket says 172.21.15.87:53954) 2024-04-22T13:35:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.137+0000 7f23fcabf640 1 -- 172.21.15.87:0/2129980327 learned_addr learned my addr 172.21.15.87:0/2129980327 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.137+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81105f0 0x7f23f81168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.138+0000 7f23fcabf640 1 -- 172.21.15.87:0/2129980327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81105f0 msgr2=0x7f23f81168f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.138+0000 7f23fcabf640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81105f0 0x7f23f81168f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.138+0000 7f23fcabf640 1 -- 172.21.15.87:0/2129980327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81099d0 msgr2=0x7f23f81163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.138+0000 7f23fcabf640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81099d0 0x7f23f81163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.138+0000 7f23fcabf640 1 -- 172.21.15.87:0/2129980327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f8116e60 con 0x7f23f8108f50 2024-04-22T13:35:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.143+0000 7f23fcabf640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f23e802c300 tx=0x7f23e80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.153+0000 7f23f5ffb640 1 -- 172.21.15.87:0/2129980327 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8042020 con 0x7f23f8108f50 2024-04-22T13:35:22.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.153+0000 7f23f5ffb640 1 -- 172.21.15.87:0/2129980327 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23e802c690 con 0x7f23f8108f50 2024-04-22T13:35:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.154+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f81170f0 con 0x7f23f8108f50 2024-04-22T13:35:22.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.159+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f811b030 con 0x7f23f8108f50 2024-04-22T13:35:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.154+0000 7f23f5ffb640 1 -- 172.21.15.87:0/2129980327 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e803e6c0 con 0x7f23f8108f50 2024-04-22T13:35:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.168+0000 7f24019eb740 1 Processor -- start 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f23f5ffb640 1 -- 172.21.15.87:0/2129980327 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f23e8045020 con 0x7f23f8108f50 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 -- start start 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028aac00 0x557c028ad050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557c028061f0 con 0x557c028aac00 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c027a34b0 con 0x557c028a2080 2024-04-22T13:35:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.173+0000 7f24019eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557c028021b0 con 0x557c028a16d0 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.174+0000 7f23fcabf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.174+0000 7f23fcabf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48160/0 (socket says 172.21.15.87:48160) 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.174+0000 7f23fcabf640 1 -- 172.21.15.87:0/3549715778 learned_addr learned my addr 172.21.15.87:0/3549715778 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.175+0000 7f23f7fff640 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028aac00 0x557c028ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.175+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.176+0000 7f23fcabf640 1 -- 172.21.15.87:0/3549715778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 msgr2=0x557c028a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.176+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.176+0000 7f23fcabf640 1 -- 172.21.15.87:0/3549715778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028aac00 msgr2=0x557c028ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.176+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028aac00 0x557c028ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.176+0000 7f23fcabf640 1 -- 172.21.15.87:0/3549715778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c027951b0 con 0x557c028a2080 2024-04-22T13:35:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.177+0000 7f23f5ffb640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 0x7f23e00a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.177+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f23e80ef7d0 tx=0x7f23e80fa0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51d37f10612986cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.179+0000 7f23f5ffb640 1 -- 172.21.15.87:0/2129980327 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f23e80f2020 con 0x7f23f8108f50 2024-04-22T13:35:22.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.179+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 0x7f23e00a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.180+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 0x7f23e00a62c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23f0005fd0 tx=0x7f23f0009500 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.184+0000 7f23deffd640 1 -- 172.21.15.87:0/3549715778 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e80ba2f0 con 0x557c028a2080 2024-04-22T13:35:22.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.184+0000 7f23deffd640 1 -- 172.21.15.87:0/3549715778 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23e80ff070 con 0x557c028a2080 2024-04-22T13:35:22.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.184+0000 7f23deffd640 1 -- 172.21.15.87:0/3549715778 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e802cca0 con 0x557c028a2080 2024-04-22T13:35:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.185+0000 7f24019eb740 1 -- 172.21.15.87:0/3549715778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 msgr2=0x557c028aa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.185+0000 7f24019eb740 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f23e80ef7d0 tx=0x7f23e80fa0e0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.186+0000 7f24019eb740 1 -- 172.21.15.87:0/3549715778 shutdown_connections 2024-04-22T13:35:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.186+0000 7f24019eb740 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028aac00 0x557c028ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.186+0000 7f24019eb740 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028aa630 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.186+0000 7f24019eb740 1 --2- 172.21.15.87:0/3549715778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.186+0000 7f24019eb740 1 -- 172.21.15.87:0/3549715778 >> 172.21.15.87:0/3549715778 conn(0x557c0289bda0 msgr2=0x557c0289c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.190+0000 7f24019eb740 1 -- 172.21.15.87:0/3549715778 shutdown_connections 2024-04-22T13:35:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.190+0000 7f24019eb740 1 -- 172.21.15.87:0/3549715778 wait complete. 2024-04-22T13:35:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.191+0000 7f24019eb740 1 -- start start 2024-04-22T13:35:22.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.191+0000 7f24019eb740 1 Processor -- start 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028a2080 0x557c028a7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557c028ad930 con 0x557c028a2080 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c027bf500 con 0x557c028aac00 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557c028adab0 con 0x557c028a16d0 2024-04-22T13:35:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.192+0000 7f24019eb740 10 client.0 _ll_register_callbacks cb 0x557c0278db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.193+0000 7f23fd2c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.193+0000 7f23fd2c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48164/0 (socket says 172.21.15.87:48164) 2024-04-22T13:35:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.193+0000 7f23fd2c0640 1 -- 172.21.15.87:0/3442096543 learned_addr learned my addr 172.21.15.87:0/3442096543 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.193+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 -- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 msgr2=0x557c028aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 -- 172.21.15.87:0/3442096543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028a2080 msgr2=0x557c028a7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028a2080 0x557c028a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 -- 172.21.15.87:0/3442096543 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c025183f0 con 0x557c028aac00 2024-04-22T13:35:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.194+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f23f0003fb0 tx=0x7f23f0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963dfda326b81fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.202+0000 7f23d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.202+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f000f8f0 con 0x557c028aac00 2024-04-22T13:35:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.202+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23e80ff070 con 0x557c028aac00 2024-04-22T13:35:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.202+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f23f000fa90 con 0x557c028aac00 2024-04-22T13:35:22.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.205+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f23f000fc90 con 0x557c028aac00 2024-04-22T13:35:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.205+0000 7f24019eb740 10 client.25820 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.205+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557c025e27d0 con 0x557c028aac00 2024-04-22T13:35:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.208+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23e80ff070 con 0x557c028aac00 2024-04-22T13:35:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f23f0026070 con 0x557c028aac00 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 10 client.25820 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 10 client.25820 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 20 client.25820 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028bab40 0x557c028bcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 4 client.25820 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.209+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c027a5930 con 0x557c028bab40 2024-04-22T13:35:22.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.215+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028bab40 0x557c028bcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.219+0000 7f23fcabf640 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028bab40 0x557c028bcfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9b4aed5a3b4d78a server_cookie=dda1b9eb52e6b467 in_seq=0 out_seq=0 2024-04-22T13:35:22.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.219+0000 7f23d7fff640 10 client.25820 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:22.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.221+0000 7f23d7fff640 1 -- 172.21.15.87:0/3442096543 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c027a5930 con 0x557c028bab40 2024-04-22T13:35:22.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.221+0000 7f23d7fff640 10 client.25820 handle_command_reply: tid=0 2024-04-22T13:35:22.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.221+0000 7f24019eb740 1 client.25820 shutdown 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 msgr2=0x557c028a7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f23f0003fb0 tx=0x7f23f0004ab0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 shutdown_connections 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028bab40 0x557c028bcfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028aac00 0x557c028a7d80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c028a2080 0x557c028a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 --2- 172.21.15.87:0/3442096543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 >> 172.21.15.87:0/3442096543 conn(0x557c0289bda0 msgr2=0x557c0289daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.233+0000 7f23d7fff640 0 client.25820 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:22.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.234+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 shutdown_connections 2024-04-22T13:35:22.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.234+0000 7f24019eb740 1 -- 172.21.15.87:0/3442096543 wait complete. 2024-04-22T13:35:22.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.235+0000 7f24019eb740 20 client.25820 trim_cache size 0 max 16384 2024-04-22T13:35:22.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.305+0000 7f24019eb740 1 Processor -- start 2024-04-22T13:35:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 -- start start 2024-04-22T13:35:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028b6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 0x557c0289aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 0x557c028a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c73d0 con 0x557c0289bda0 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c4ca0 con 0x557c028a16d0 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.311+0000 7f24019eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c2bc0 con 0x557c028a2080 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.312+0000 7f23f7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 0x557c0289aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.312+0000 7f23f7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 0x557c0289aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48178/0 (socket says 172.21.15.87:48178) 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.312+0000 7f23f7fff640 1 -- 172.21.15.87:0/285849578 learned_addr learned my addr 172.21.15.87:0/285849578 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.313+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 0x557c028a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 -- 172.21.15.87:0/285849578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 msgr2=0x557c0289aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 0x557c0289aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 -- 172.21.15.87:0/285849578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 msgr2=0x557c028b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 -- 172.21.15.87:0/285849578 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c028bd320 con 0x557c028a2080 2024-04-22T13:35:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.314+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 0x557c028a9f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23f00285e0 tx=0x7f23f00049c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f33a8e3d45ba61e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.315+0000 7f23de7fc640 1 -- 172.21.15.87:0/285849578 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f0035020 con 0x557c028a2080 2024-04-22T13:35:22.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.315+0000 7f23de7fc640 1 -- 172.21.15.87:0/285849578 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23f0028c80 con 0x557c028a2080 2024-04-22T13:35:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.316+0000 7f24019eb740 1 -- 172.21.15.87:0/285849578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 msgr2=0x557c028a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.316+0000 7f24019eb740 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 0x557c028a9f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23f00285e0 tx=0x7f23f00049c0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 -- 172.21.15.87:0/285849578 shutdown_connections 2024-04-22T13:35:22.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a2080 0x557c028a9f70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a16d0 0x557c0289aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 --2- 172.21.15.87:0/285849578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 -- 172.21.15.87:0/285849578 >> 172.21.15.87:0/285849578 conn(0x557c028a7ee0 msgr2=0x557c0289a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 -- 172.21.15.87:0/285849578 shutdown_connections 2024-04-22T13:35:22.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.323+0000 7f24019eb740 1 -- 172.21.15.87:0/285849578 wait complete. 2024-04-22T13:35:22.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.324+0000 7f24019eb740 1 -- start start 2024-04-22T13:35:22.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.324+0000 7f24019eb740 1 Processor -- start 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028a4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c3180 con 0x557c0289bda0 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c45d0 con 0x557c028a2080 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f23f81c3830 con 0x557c028a16d0 2024-04-22T13:35:22.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.325+0000 7f24019eb740 10 client.0 _ll_register_callbacks cb 0x7f23f0003e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.326+0000 7f23f7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.326+0000 7f23f7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:46694/0 (socket says 172.21.15.87:46694) 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.326+0000 7f23f7fff640 1 -- 172.21.15.87:0/1419776812 learned_addr learned my addr 172.21.15.87:0/1419776812 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.326+0000 7f23fd2c0640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 -- 172.21.15.87:0/1419776812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 msgr2=0x557c028a51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028a51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 -- 172.21.15.87:0/1419776812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 msgr2=0x557c028a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 -- 172.21.15.87:0/1419776812 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c02920370 con 0x557c028a16d0 2024-04-22T13:35:22.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23f7fff640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00bfb0 tx=0x7f23ec0106e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29cf216e13fba1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.327+0000 7f23d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.328+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec01a020 con 0x557c028a16d0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.328+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23f0028c80 con 0x557c028a16d0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.328+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f23ec016690 con 0x557c028a16d0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.328+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f23ec025050 con 0x557c028a16d0 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.329+0000 7f24019eb740 10 client.25861 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.329+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557c02921690 con 0x557c028a16d0 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23f0028c80 con 0x557c028a16d0 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f23ec014b00 con 0x557c028a16d0 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 10 client.25861 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 10 client.25861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 20 client.25861 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028aac00 0x557c02923fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 4 client.25861 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:22.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.333+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c02813c10 con 0x557c028aac00 2024-04-22T13:35:22.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.336+0000 7f23fcabf640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028aac00 0x557c02923fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.344+0000 7f23fcabf640 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028aac00 0x557c02923fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99caad012b20303b server_cookie=896706ce7045fbad in_seq=0 out_seq=0 2024-04-22T13:35:22.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.344+0000 7f23d7fff640 10 client.25861 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:22.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.345+0000 7f23d7fff640 1 -- 172.21.15.87:0/1419776812 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c02813c10 con 0x557c028aac00 2024-04-22T13:35:22.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.345+0000 7f23d7fff640 10 client.25861 handle_command_reply: tid=0 2024-04-22T13:35:22.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.345+0000 7f24019eb740 1 client.25861 shutdown 2024-04-22T13:35:22.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.357+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 msgr2=0x557c028a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.357+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00bfb0 tx=0x7f23ec0106e0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 shutdown_connections 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x557c028aac00 0x557c02923fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c028a2080 0x557c028a51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c028a16d0 0x557c028a4ca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 --2- 172.21.15.87:0/1419776812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c0289bda0 0x557c028a4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.370+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 >> 172.21.15.87:0/1419776812 conn(0x557c028a7ee0 msgr2=0x557c0289ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.372+0000 7f23d7fff640 0 client.25861 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:22.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.377+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 shutdown_connections 2024-04-22T13:35:22.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.377+0000 7f24019eb740 1 -- 172.21.15.87:0/1419776812 wait complete. 2024-04-22T13:35:22.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.378+0000 7f24019eb740 20 client.25861 trim_cache size 0 max 16384 2024-04-22T13:35:22.378 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:22.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.403+0000 7f8ec47c7640 1 -- 172.21.15.87:0/1462084961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc1099f0 msgr2=0x7f8ebc11d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.403+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/1462084961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8eb801a910 tx=0x7f8eb803cd40 comp rx=0 tx=0).stop 2024-04-22T13:35:22.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.405+0000 7f8ec47c7640 1 -- 172.21.15.87:0/1462084961 shutdown_connections 2024-04-22T13:35:22.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.405+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/1462084961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.405+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/1462084961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 0x7f8ebc1093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.405+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/1462084961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc110550 0x7f8ebc110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.405+0000 7f8ec47c7640 1 -- 172.21.15.87:0/1462084961 >> 172.21.15.87:0/1462084961 conn(0x7f8ebc06ce90 msgr2=0x7f8ebc06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.412+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 msgr2=0x7f23e00a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.412+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 0x7f23e00a62c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23f0005fd0 tx=0x7f23f0009500 comp rx=0 tx=0).stop 2024-04-22T13:35:22.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.413+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 msgr2=0x7f23f811d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.413+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f23e802c300 tx=0x7f23e80045e0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 shutdown_connections 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f23e00a3e00 0x7f23e00a62c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23f81105f0 0x7f23f81168f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81099d0 0x7f23f81163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 --2- 172.21.15.87:0/2129980327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23f8108f50 0x7f23f811d2e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.416+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 >> 172.21.15.87:0/2129980327 conn(0x7f23f806ced0 msgr2=0x7f23f8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.417+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 shutdown_connections 2024-04-22T13:35:22.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.417+0000 7f23fdac1640 1 -- 172.21.15.87:0/2129980327 wait complete. 2024-04-22T13:35:22.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.414+0000 7f8ec47c7640 1 -- 172.21.15.87:0/1462084961 shutdown_connections 2024-04-22T13:35:22.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.418+0000 7f8ec47c7640 1 -- 172.21.15.87:0/1462084961 wait complete. 2024-04-22T13:35:22.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.419+0000 7f8ec47c7640 1 Processor -- start 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 -- start start 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 0x7f8ebc11d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc1137e0 con 0x7f8ebc108f50 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc113660 con 0x7f8ebc1099f0 2024-04-22T13:35:22.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.422+0000 7f8ec47c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8ebc113960 con 0x7f8ebc110550 2024-04-22T13:35:22.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.424+0000 7f8ec2d64640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.425+0000 7f8ec2d64640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:46720/0 (socket says 172.21.15.87:46720) 2024-04-22T13:35:22.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.425+0000 7f8ec2d64640 1 -- 172.21.15.87:0/3312151264 learned_addr learned my addr 172.21.15.87:0/3312151264 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.427+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.428+0000 7f8ec2563640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 0x7f8ebc11d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.429+0000 7f8ec2d64640 1 -- 172.21.15.87:0/3312151264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc1099f0 msgr2=0x7f8ebc11d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.429+0000 7f8ec2d64640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.429+0000 7f8ec2d64640 1 -- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 msgr2=0x7f8ebc11d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.429+0000 7f8ec2d64640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 0x7f8ebc11d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.429+0000 7f8ec2d64640 1 -- 172.21.15.87:0/3312151264 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ebc119de0 con 0x7f8ebc110550 2024-04-22T13:35:22.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.439+0000 7f8ec2d64640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005ab0 tx=0x7f8eb8004a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.447+0000 7f8eb37fe640 1 -- 172.21.15.87:0/3312151264 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb803d8f0 con 0x7f8ebc110550 2024-04-22T13:35:22.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.447+0000 7f8eb37fe640 1 -- 172.21.15.87:0/3312151264 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eb8053070 con 0x7f8ebc110550 2024-04-22T13:35:22.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.447+0000 7f8eb37fe640 1 -- 172.21.15.87:0/3312151264 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb804e480 con 0x7f8ebc110550 2024-04-22T13:35:22.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.447+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ebc11a070 con 0x7f8ebc110550 2024-04-22T13:35:22.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.447+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ebc1ab530 con 0x7f8ebc110550 2024-04-22T13:35:22.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.454+0000 7f8ec73b8740 1 Processor -- start 2024-04-22T13:35:22.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.455+0000 7f8eb37fe640 1 -- 172.21.15.87:0/3312151264 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8eb804e620 con 0x7f8ebc110550 2024-04-22T13:35:22.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.456+0000 7f8eb37fe640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 0x7f8ea80a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.458+0000 7f8eb37fe640 1 -- 172.21.15.87:0/3312151264 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8eb8004a80 con 0x7f8ebc110550 2024-04-22T13:35:22.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.460+0000 7f8ec73b8740 1 -- start start 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b31463cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314645b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314648480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b31459d0b0 con 0x55b314646090 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b31454e680 con 0x55b31463cb90 2024-04-22T13:35:22.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.463+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b31455ac60 con 0x55b31463d4b0 2024-04-22T13:35:22.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.465+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 0x7f8ea80a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1814067495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1971562606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.466+0000 7f8ec2563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b31463cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.466+0000 7f8ec2563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b31463cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48202/0 (socket says 172.21.15.87:48202) 2024-04-22T13:35:22.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.466+0000 7f8ec2563640 1 -- 172.21.15.87:0/2221593946 learned_addr learned my addr 172.21.15.87:0/2221593946 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.471+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314645b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec2d64640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314648480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 -- 172.21.15.87:0/2221593946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 msgr2=0x55b31463cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b31463cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 -- 172.21.15.87:0/2221593946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 msgr2=0x55b314648480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314648480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 -- 172.21.15.87:0/2221593946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b31437fc90 con 0x55b31463d4b0 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec2d64640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314648480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.472+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314645b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc11ab60 tx=0x7f8eac013e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e44ee6125f2bad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.473+0000 7f8eb0ff9640 1 -- 172.21.15.87:0/2221593946 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac025070 con 0x55b31463d4b0 2024-04-22T13:35:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.474+0000 7f8eb0ff9640 1 -- 172.21.15.87:0/2221593946 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eac0130e0 con 0x55b31463d4b0 2024-04-22T13:35:22.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.477+0000 7f8ec73b8740 1 -- 172.21.15.87:0/2221593946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 msgr2=0x55b314645b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.477+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314645b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc11ab60 tx=0x7f8eac013e80 comp rx=0 tx=0).stop 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.480+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 0x7f8ea80a6370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00ffd0 tx=0x7f8eac00e780 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.480+0000 7f8ec73b8740 1 -- 172.21.15.87:0/2221593946 shutdown_connections 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.480+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314648480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.480+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314645b50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.481+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/2221593946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b31463cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.481+0000 7f8ec73b8740 1 -- 172.21.15.87:0/2221593946 >> 172.21.15.87:0/2221593946 conn(0x55b3146372c0 msgr2=0x55b3146376a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.482+0000 7f8ec73b8740 1 -- 172.21.15.87:0/2221593946 shutdown_connections 2024-04-22T13:35:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.483+0000 7f8ec73b8740 1 -- 172.21.15.87:0/2221593946 wait complete. 2024-04-22T13:35:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 -- start start 2024-04-22T13:35:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 Processor -- start 2024-04-22T13:35:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314643520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b3145a0400 con 0x55b314646090 2024-04-22T13:35:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b31455c840 con 0x55b31463cb90 2024-04-22T13:35:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b3145a5720 con 0x55b31463d4b0 2024-04-22T13:35:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.484+0000 7f8ec73b8740 10 client.0 _ll_register_callbacks cb 0x55b3145adea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.487+0000 7f8ec2563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.487+0000 7f8ec2563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48218/0 (socket says 172.21.15.87:48218) 2024-04-22T13:35:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.487+0000 7f8ec2563640 1 -- 172.21.15.87:0/1761437639 learned_addr learned my addr 172.21.15.87:0/1761437639 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.488+0000 7f8ec2563640 1 -- 172.21.15.87:0/1761437639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 msgr2=0x55b314643520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.488+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314643520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.488+0000 7f8ec2563640 1 -- 172.21.15.87:0/1761437639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 msgr2=0x55b314643a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.488+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.488+0000 7f8ec2563640 1 -- 172.21.15.87:0/1761437639 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3143e0420 con 0x55b31463cb90 2024-04-22T13:35:22.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.490+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400ac30 tx=0x7f8eb4007680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10cd9a8811829633 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1814067495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1971562606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.495+0000 7f8ea5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:22.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.495+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb401b060 con 0x55b31463cb90 2024-04-22T13:35:22.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.495+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eac0130e0 con 0x55b31463cb90 2024-04-22T13:35:22.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.495+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8eb4016640 con 0x55b31463cb90 2024-04-22T13:35:22.496 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.499+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8eb4016840 con 0x55b31463cb90 2024-04-22T13:35:22.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.501+0000 7f8ec73b8740 10 client.25835 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.501+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b314273580 con 0x55b31463cb90 2024-04-22T13:35:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.502+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eac0130e0 con 0x55b31463cb90 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8eb401e020 con 0x55b31463cb90 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 10 client.25835 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 10 client.25835 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 20 client.25835 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b314655e40 0x55b3146582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 4 client.25835 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.504+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3145aeb10 con 0x55b314655e40 2024-04-22T13:35:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.508+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b314655e40 0x55b3146582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.515+0000 7f8ea5ffb640 10 client.25835 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:22.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.515+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b314655e40 0x55b3146582b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10e1fe98869f8bb server_cookie=863fc5b8709ee615 in_seq=0 out_seq=0 2024-04-22T13:35:22.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.519+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1761437639 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3145aeb10 con 0x55b314655e40 2024-04-22T13:35:22.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.519+0000 7f8ea5ffb640 10 client.25835 handle_command_reply: tid=0 2024-04-22T13:35:22.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.520+0000 7f8ec73b8740 1 client.25835 shutdown 2024-04-22T13:35:22.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.523+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 msgr2=0x55b314642fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.523+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400ac30 tx=0x7f8eb4007680 comp rx=0 tx=0).stop 2024-04-22T13:35:22.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 shutdown_connections 2024-04-22T13:35:22.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b314655e40 0x55b3146582b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463d4b0 0x55b314643520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1761437639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463cb90 0x55b314642fe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.529+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 >> 172.21.15.87:0/1761437639 conn(0x55b3146372c0 msgr2=0x55b314639010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.533+0000 7f8ea5ffb640 0 client.25835 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:22.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.535+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 shutdown_connections 2024-04-22T13:35:22.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.537+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1761437639 wait complete. 2024-04-22T13:35:22.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.537+0000 7f8ec73b8740 20 client.25835 trim_cache size 0 max 16384 2024-04-22T13:35:22.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:35:22.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.593+0000 7f8ec73b8740 1 Processor -- start 2024-04-22T13:35:22.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.593+0000 7f8ec73b8740 1 -- start start 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.593+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b314651b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.593+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.594+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.594+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000e6f0 con 0x55b314646090 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.594+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000c5f0 con 0x55b31463d4b0 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.594+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000e020 con 0x55b31463cb90 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.595+0000 7f8ec1d62640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.595+0000 7f8ec1d62640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48230/0 (socket says 172.21.15.87:48230) 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.595+0000 7f8ec1d62640 1 -- 172.21.15.87:0/974563235 learned_addr learned my addr 172.21.15.87:0/974563235 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.595+0000 7f8ec1d62640 1 -- 172.21.15.87:0/974563235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 msgr2=0x55b314651b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.596+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b314651b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.596+0000 7f8ec1d62640 1 -- 172.21.15.87:0/974563235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 msgr2=0x55b314643880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.596+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.596+0000 7f8ec1d62640 1 -- 172.21.15.87:0/974563235 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b314658620 con 0x55b31463d4b0 2024-04-22T13:35:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.596+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8eac023040 tx=0x7f8eac027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=742c44f3bc1689f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.597+0000 7f8ea7fff640 1 -- 172.21.15.87:0/974563235 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac038070 con 0x55b31463d4b0 2024-04-22T13:35:22.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.597+0000 7f8ea7fff640 1 -- 172.21.15.87:0/974563235 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eac02f070 con 0x55b31463d4b0 2024-04-22T13:35:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.598+0000 7f8ec73b8740 1 -- 172.21.15.87:0/974563235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 msgr2=0x55b314641f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.598+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8eac023040 tx=0x7f8eac027000 comp rx=0 tx=0).stop 2024-04-22T13:35:22.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.603+0000 7f8ec73b8740 1 -- 172.21.15.87:0/974563235 shutdown_connections 2024-04-22T13:35:22.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.603+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b314646090 0x55b314643880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.603+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b31463d4b0 0x55b314641f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.603+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/974563235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b314651b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.603+0000 7f8ec73b8740 1 -- 172.21.15.87:0/974563235 >> 172.21.15.87:0/974563235 conn(0x55b3146372c0 msgr2=0x55b314638f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.604+0000 7f8ec73b8740 1 -- 172.21.15.87:0/974563235 shutdown_connections 2024-04-22T13:35:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.608+0000 7f8ec73b8740 1 -- 172.21.15.87:0/974563235 wait complete. 2024-04-22T13:35:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.608+0000 7f8ec73b8740 1 -- start start 2024-04-22T13:35:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 Processor -- start 2024-04-22T13:35:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b31464dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b314646090 0x55b31464a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000d280 con 0x55b31463d4b0 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000cbb0 con 0x55b314646090 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea000d950 con 0x55b31463cb90 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.609+0000 7f8ec73b8740 10 client.0 _ll_register_callbacks cb 0x7f8eb40098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.611+0000 7f8ec1d62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.611+0000 7f8ec1d62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54028/0 (socket says 172.21.15.87:54028) 2024-04-22T13:35:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.611+0000 7f8ec1d62640 1 -- 172.21.15.87:0/1439669261 learned_addr learned my addr 172.21.15.87:0/1439669261 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.611+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b31464dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec2563640 1 -- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b314646090 msgr2=0x55b31464a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b314646090 0x55b31464a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec2563640 1 -- 172.21.15.87:0/1439669261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 msgr2=0x55b31464e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec2563640 1 -- 172.21.15.87:0/1439669261 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b314636b20 con 0x55b31463cb90 2024-04-22T13:35:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.613+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.614+0000 7f8ec2563640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b31464dcb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400ac00 tx=0x7f8eb4014590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8719d7c946e69139 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.614+0000 7f8ea5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:22.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.623+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4016640 con 0x55b31463cb90 2024-04-22T13:35:22.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.623+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8eb4003a10 con 0x55b31463cb90 2024-04-22T13:35:22.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.624+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eac02f070 con 0x55b31463cb90 2024-04-22T13:35:22.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.635+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8eb4003c10 con 0x55b31463cb90 2024-04-22T13:35:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.636+0000 7f8ec73b8740 10 client.25873 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.636+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b3146bcc40 con 0x55b31463cb90 2024-04-22T13:35:22.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eac02f070 con 0x55b31463cb90 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8eb4028030 con 0x55b31463cb90 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 10 client.25873 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 10 client.25873 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 20 client.25873 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b3146bd3d0 0x55b3146bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 4 client.25873 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 71.2, "await": true}] 2024-04-22T13:35:22.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.641+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 71.2, "await": true}) -- 0x55b3145a8980 con 0x55b3146bd3d0 2024-04-22T13:35:22.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.648+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b3146bd3d0 0x55b3146bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.658+0000 7f8ec1d62640 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b3146bd3d0 0x55b3146bf7f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd97ad6b6b9934f server_cookie=e3bd6518bf7e7e52 in_seq=0 out_seq=0 2024-04-22T13:35:22.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.658+0000 7f8ea5ffb640 10 client.25873 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:22.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1814067495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1971562606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:22.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.827+0000 7fa1478d2640 1 -- 172.21.15.87:0/1546515741 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 msgr2=0x7fa14010ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.827+0000 7fa1478d2640 1 --2- 172.21.15.87:0/1546515741 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa14010ec50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009930 tx=0x7fa13c02bc60 comp rx=0 tx=0).stop 2024-04-22T13:35:22.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.828+0000 7fa1478d2640 1 -- 172.21.15.87:0/1546515741 shutdown_connections 2024-04-22T13:35:22.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.828+0000 7fa1478d2640 1 --2- 172.21.15.87:0/1546515741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 0x7fa14011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.828+0000 7fa1478d2640 1 --2- 172.21.15.87:0/1546515741 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa14010ec50 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.828+0000 7fa1478d2640 1 --2- 172.21.15.87:0/1546515741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa14010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.828+0000 7fa1478d2640 1 -- 172.21.15.87:0/1546515741 >> 172.21.15.87:0/1546515741 conn(0x7fa14006d060 msgr2=0x7fa14006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.833+0000 7fa1478d2640 1 -- 172.21.15.87:0/1546515741 shutdown_connections 2024-04-22T13:35:22.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.835+0000 7fa1478d2640 1 -- 172.21.15.87:0/1546515741 wait complete. 2024-04-22T13:35:22.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 Processor -- start 2024-04-22T13:35:22.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 -- start start 2024-04-22T13:35:22.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa1401a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 0x7fa1401abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa140121040 con 0x7fa14010e7d0 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa140120ec0 con 0x7fa1401107a0 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.837+0000 7fa1478d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa1401211c0 con 0x7fa14011b730 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.839+0000 7fa144e6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.840+0000 7fa144e6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48258/0 (socket says 172.21.15.87:48258) 2024-04-22T13:35:22.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.840+0000 7fa144e6d640 1 -- 172.21.15.87:0/910352096 learned_addr learned my addr 172.21.15.87:0/910352096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.842+0000 7fa145e6f640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 0x7fa1401abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.842+0000 7fa14566e640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa1401a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.846+0000 7fa144e6d640 1 -- 172.21.15.87:0/910352096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 msgr2=0x7fa1401abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.846+0000 7fa144e6d640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 0x7fa1401abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.847+0000 7fa144e6d640 1 -- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 msgr2=0x7fa1401a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.847+0000 7fa144e6d640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa1401a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.847+0000 7fa144e6d640 1 -- 172.21.15.87:0/910352096 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1401ac5d0 con 0x7fa1401107a0 2024-04-22T13:35:22.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.850+0000 7fa144e6d640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009430 tx=0x7fa13c003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.855+0000 7fa1367fc640 1 -- 172.21.15.87:0/910352096 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c042020 con 0x7fa1401107a0 2024-04-22T13:35:22.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.855+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1401ac860 con 0x7fa1401107a0 2024-04-22T13:35:22.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.855+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1401b4150 con 0x7fa1401107a0 2024-04-22T13:35:22.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.865+0000 7fa14a59f740 1 Processor -- start 2024-04-22T13:35:22.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.865+0000 7fa1367fc640 1 -- 172.21.15.87:0/910352096 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa13c005410 con 0x7fa1401107a0 2024-04-22T13:35:22.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.866+0000 7fa1367fc640 1 -- 172.21.15.87:0/910352096 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c02ca00 con 0x7fa1401107a0 2024-04-22T13:35:22.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.866+0000 7fa14a59f740 1 -- start start 2024-04-22T13:35:22.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.866+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c57e80 0x558285c60430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c60a00 0x558285c62e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa14a59f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558285bbc1f0 con 0x558285c574d0 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa145e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa145e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54046/0 (socket says 172.21.15.87:54046) 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa145e6f640 1 -- 172.21.15.87:0/2533159787 learned_addr learned my addr 172.21.15.87:0/2533159787 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558285b594b0 con 0x558285c57e80 2024-04-22T13:35:22.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.867+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558285bb81b0 con 0x558285c60a00 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa144e6d640 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c60a00 0x558285c62e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa145e6f640 1 -- 172.21.15.87:0/2533159787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c60a00 msgr2=0x558285c62e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa145e6f640 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c60a00 0x558285c62e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa145e6f640 1 -- 172.21.15.87:0/2533159787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c57e80 msgr2=0x558285c60430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa145e6f640 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c57e80 0x558285c60430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.868+0000 7fa145e6f640 1 -- 172.21.15.87:0/2533159787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558285b4b1b0 con 0x558285c574d0 2024-04-22T13:35:22.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.874+0000 7fa145e6f640 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa138009ee0 tx=0x7fa138010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f62ee7d72a3ed18 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.876+0000 7fa12b7fe640 1 -- 172.21.15.87:0/2533159787 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138016070 con 0x558285c574d0 2024-04-22T13:35:22.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.877+0000 7fa12b7fe640 1 -- 172.21.15.87:0/2533159787 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa138004d20 con 0x558285c574d0 2024-04-22T13:35:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.878+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 msgr2=0x558285c578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.878+0000 7fa14a59f740 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa138009ee0 tx=0x7fa138010740 comp rx=0 tx=0).stop 2024-04-22T13:35:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.879+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 shutdown_connections 2024-04-22T13:35:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.879+0000 7fa14a59f740 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c60a00 0x558285c62e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.879+0000 7fa14a59f740 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c57e80 0x558285c60430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.879+0000 7fa14a59f740 1 --2- 172.21.15.87:0/2533159787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c574d0 0x558285c578b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.879+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 >> 172.21.15.87:0/2533159787 conn(0x558285c51ba0 msgr2=0x558285c51f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.883+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 shutdown_connections 2024-04-22T13:35:22.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.883+0000 7fa14a59f740 1 -- 172.21.15.87:0/2533159787 wait complete. 2024-04-22T13:35:22.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.884+0000 7fa14a59f740 1 -- start start 2024-04-22T13:35:22.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.884+0000 7fa1367fc640 1 -- 172.21.15.87:0/910352096 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa13c045020 con 0x7fa1401107a0 2024-04-22T13:35:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.885+0000 7fa1367fc640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 0x7fa12c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.885+0000 7fa1367fc640 1 -- 172.21.15.87:0/910352096 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa13c0f2700 con 0x7fa1401107a0 2024-04-22T13:35:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.886+0000 7fa14a59f740 1 Processor -- start 2024-04-22T13:35:22.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.886+0000 7fa145e6f640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 0x7fa12c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.887+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c5dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14a59f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558285c639c0 con 0x558285c57e80 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14a59f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558285b75500 con 0x558285c60a00 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14566e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14566e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:46812/0 (socket says 172.21.15.87:46812) 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14566e640 1 -- 172.21.15.87:0/3160201552 learned_addr learned my addr 172.21.15.87:0/3160201552 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558285c63b40 con 0x558285c574d0 2024-04-22T13:35:22.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.888+0000 7fa145e6f640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c5dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa144e6d640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa145e6f640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 0x7fa12c0a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa1380058e0 tx=0x7fa1380059e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa145e6f640 1 -- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 msgr2=0x558285c60200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa145e6f640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa145e6f640 1 -- 172.21.15.87:0/3160201552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 msgr2=0x558285c5d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.889+0000 7fa145e6f640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa145e6f640 1 -- 172.21.15.87:0/3160201552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582858ce3f0 con 0x558285c60a00 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa14a59f740 10 client.0 _ll_register_callbacks cb 0x558285b43b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa138004d20 con 0x558285c60a00 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa14566e640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa145e6f640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c5dad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1380124f0 tx=0x7fa138010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3a14f8ff863f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.890+0000 7fa128ff9640 10 client.25844 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.892+0000 7fa128ff9640 1 -- 172.21.15.87:0/3160201552 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138034060 con 0x558285c60a00 2024-04-22T13:35:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.892+0000 7fa128ff9640 1 -- 172.21.15.87:0/3160201552 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa13802f430 con 0x558285c60a00 2024-04-22T13:35:22.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.892+0000 7fa128ff9640 1 -- 172.21.15.87:0/3160201552 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa138016050 con 0x558285c60a00 2024-04-22T13:35:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.894+0000 7fa14a59f740 10 client.25844 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.894+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({fsmap=21}) -- 0x5582859987d0 con 0x558285c60a00 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa128ff9640 1 -- 172.21.15.87:0/3160201552 <== mon.2 v2:172.21.15.120:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa13800f040 con 0x558285c60a00 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 10 client.25844 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 10 client.25844 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 20 client.25844 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285c707b0 0x558285c72ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 4 client.25844 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:22.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.901+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558285b5b930 con 0x558285c707b0 2024-04-22T13:35:22.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.902+0000 7fa14566e640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285c707b0 0x558285c72ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.911+0000 7fa128ff9640 10 client.25844 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:22.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.911+0000 7fa14566e640 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285c707b0 0x558285c72ba0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97712d5830edcbf8 server_cookie=1dcbd1a669332dd8 in_seq=0 out_seq=0 2024-04-22T13:35:22.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.917+0000 7fa128ff9640 1 -- 172.21.15.87:0/3160201552 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558285b5b930 con 0x558285c707b0 2024-04-22T13:35:22.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.917+0000 7fa128ff9640 10 client.25844 handle_command_reply: tid=0 2024-04-22T13:35:22.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.917+0000 7fa14a59f740 1 client.25844 shutdown 2024-04-22T13:35:22.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 msgr2=0x558285c5dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c5dad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1380124f0 tx=0x7fa138010740 comp rx=0 tx=0).stop 2024-04-22T13:35:22.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 shutdown_connections 2024-04-22T13:35:22.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285c707b0 0x558285c72ba0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c5dad0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 --2- 172.21.15.87:0/3160201552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c60200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.918+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 >> 172.21.15.87:0/3160201552 conn(0x558285c51ba0 msgr2=0x558285c538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.919+0000 7fa128ff9640 0 client.25844 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.919+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 shutdown_connections 2024-04-22T13:35:22.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.920+0000 7fa14a59f740 1 -- 172.21.15.87:0/3160201552 wait complete. 2024-04-22T13:35:22.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.921+0000 7fa14a59f740 20 client.25844 trim_cache size 0 max 16384 2024-04-22T13:35:22.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.958+0000 7fa14a59f740 1 Processor -- start 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.959+0000 7fa14a59f740 1 -- start start 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c60200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa124010fa0 con 0x558285c57e80 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa12400e870 con 0x558285c60a00 2024-04-22T13:35:22.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa14a59f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa12400c6c0 con 0x558285c574d0 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa144e6d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa144e6d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54068/0 (socket says 172.21.15.87:54068) 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.960+0000 7fa144e6d640 1 -- 172.21.15.87:0/1312041218 learned_addr learned my addr 172.21.15.87:0/1312041218 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 -- 172.21.15.87:0/1312041218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 msgr2=0x558285c6c960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c6c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 -- 172.21.15.87:0/1312041218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 msgr2=0x558285c60200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c60200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 -- 172.21.15.87:0/1312041218 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558285c4d470 con 0x558285c57e80 2024-04-22T13:35:22.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.961+0000 7fa144e6d640 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa13c006010 tx=0x7fa13c0be120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97abdb2cc90ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.962+0000 7fa12affd640 1 -- 172.21.15.87:0/1312041218 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0be7c0 con 0x558285c57e80 2024-04-22T13:35:22.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.962+0000 7fa12affd640 1 -- 172.21.15.87:0/1312041218 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa13c0c2b70 con 0x558285c57e80 2024-04-22T13:35:22.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.962+0000 7fa12affd640 1 -- 172.21.15.87:0/1312041218 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c02cba0 con 0x558285c57e80 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 -- 172.21.15.87:0/1312041218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 msgr2=0x558285c5fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa13c006010 tx=0x7fa13c0be120 comp rx=0 tx=0).stop 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 -- 172.21.15.87:0/1312041218 shutdown_connections 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c60a00 0x558285c60200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c57e80 0x558285c5fcc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 --2- 172.21.15.87:0/1312041218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c574d0 0x558285c6c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 -- 172.21.15.87:0/1312041218 >> 172.21.15.87:0/1312041218 conn(0x558285c51ba0 msgr2=0x558285c50c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.964+0000 7fa14a59f740 1 -- 172.21.15.87:0/1312041218 shutdown_connections 2024-04-22T13:35:22.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.965+0000 7fa14a59f740 1 -- 172.21.15.87:0/1312041218 wait complete. 2024-04-22T13:35:22.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.965+0000 7fa14a59f740 1 -- start start 2024-04-22T13:35:22.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.965+0000 7fa14a59f740 1 Processor -- start 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c574d0 0x558285c68e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c57e80 0x558285c693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa12400cd30 con 0x558285c60a00 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa12400e1a0 con 0x558285c574d0 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa12400d400 con 0x558285c57e80 2024-04-22T13:35:22.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.966+0000 7fa14a59f740 10 client.0 _ll_register_callbacks cb 0x7fa1380049e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:22.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.968+0000 7fa145e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.968+0000 7fa145e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54080/0 (socket says 172.21.15.87:54080) 2024-04-22T13:35:22.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.968+0000 7fa145e6f640 1 -- 172.21.15.87:0/4135997665 learned_addr learned my addr 172.21.15.87:0/4135997665 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:22.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 -- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c57e80 msgr2=0x558285c693b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c57e80 0x558285c693b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 -- 172.21.15.87:0/4135997665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c574d0 msgr2=0x558285c68e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:22.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c574d0 0x558285c68e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 -- 172.21.15.87:0/4135997665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558285c66820 con 0x558285c60a00 2024-04-22T13:35:22.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.969+0000 7fa145e6f640 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa1380059b0 tx=0x7fa138022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef15ea9b53a32c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:22.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.970+0000 7fa128ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.971+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138009580 con 0x558285c60a00 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.971+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa13801a620 con 0x558285c60a00 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.971+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa13c0c2b70 con 0x558285c60a00 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa138034030 con 0x558285c60a00 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 10 client.16656 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:22.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558285cd7770 con 0x558285c60a00 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa13c0c2b70 con 0x558285c60a00 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa13802fb00 con 0x558285c60a00 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 10 client.16656 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 10 client.16656 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 20 client.16656 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285cd7d20 0x558285cda110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 4 client.16656 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:22.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.972+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558285c5de60 con 0x558285cd7d20 2024-04-22T13:35:22.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.978+0000 7fa14566e640 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285cd7d20 0x558285cda110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:22.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.991+0000 7fa128ff9640 10 client.16656 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:22.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.991+0000 7fa14566e640 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285cd7d20 0x558285cda110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e30bf3eebf64ee5 server_cookie=c785896801ed4d36 in_seq=0 out_seq=0 2024-04-22T13:35:22.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.994+0000 7fa128ff9640 1 -- 172.21.15.87:0/4135997665 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558285c5de60 con 0x558285cd7d20 2024-04-22T13:35:22.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.994+0000 7fa128ff9640 10 client.16656 handle_command_reply: tid=0 2024-04-22T13:35:22.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.994+0000 7fa14a59f740 1 client.16656 shutdown 2024-04-22T13:35:22.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 msgr2=0x558285c65ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa1380059b0 tx=0x7fa138022000 comp rx=0 tx=0).stop 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 shutdown_connections 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558285cd7d20 0x558285cda110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558285c60a00 0x558285c65ff0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558285c57e80 0x558285c693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 --2- 172.21.15.87:0/4135997665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558285c574d0 0x558285c68e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 >> 172.21.15.87:0/4135997665 conn(0x558285c51ba0 msgr2=0x558285c58260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa128ff9640 0 client.16656 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 shutdown_connections 2024-04-22T13:35:22.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 1 -- 172.21.15.87:0/4135997665 wait complete. 2024-04-22T13:35:22.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.995+0000 7fa14a59f740 20 client.16656 trim_cache size 0 max 16384 2024-04-22T13:35:22.995 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 msgr2=0x7fa12c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 0x7fa12c0a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa1380058e0 tx=0x7fa1380059e0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 msgr2=0x7fa1401a7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009430 tx=0x7fa13c003e10 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 shutdown_connections 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa12c0a3dc0 0x7fa12c0a6280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa14011b730 0x7fa1401abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1401107a0 0x7fa1401a7b20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 --2- 172.21.15.87:0/910352096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa14010e7d0 0x7fa1401a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 >> 172.21.15.87:0/910352096 conn(0x7fa14006d060 msgr2=0x7fa140072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 shutdown_connections 2024-04-22T13:35:22.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:22.996+0000 7fa1478d2640 1 -- 172.21.15.87:0/910352096 wait complete. 2024-04-22T13:35:23.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 -- 172.21.15.87:0/2951599442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270102210 msgr2=0x7f72701025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 --2- 172.21.15.87:0/2951599442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270102210 0x7f72701025f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7260009900 tx=0x7f726002bc50 comp rx=0 tx=0).stop 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 -- 172.21.15.87:0/2951599442 shutdown_connections 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 --2- 172.21.15.87:0/2951599442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72701072e0 0x7f727010eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 --2- 172.21.15.87:0/2951599442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7270102bc0 0x7f7270106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 --2- 172.21.15.87:0/2951599442 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270102210 0x7f72701025f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 -- 172.21.15.87:0/2951599442 >> 172.21.15.87:0/2951599442 conn(0x7f72700fdab0 msgr2=0x7f72700ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 -- 172.21.15.87:0/2951599442 shutdown_connections 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.193+0000 7f727545b640 1 -- 172.21.15.87:0/2951599442 wait complete. 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 Processor -- start 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 -- start start 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7270102210 0x7f7270074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72701072e0 0x7f7270074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7270112340 con 0x7f7270074cd0 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72701121c0 con 0x7f7270102210 2024-04-22T13:35:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f727545b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f72701124c0 con 0x7f72701072e0 2024-04-22T13:35:23.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54094/0 (socket says 172.21.15.87:54094) 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.194+0000 7f726f7fe640 1 -- 172.21.15.87:0/3880582554 learned_addr learned my addr 172.21.15.87:0/3880582554 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f7267fff640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72701072e0 0x7f7270074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 -- 172.21.15.87:0/3880582554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72701072e0 msgr2=0x7f7270074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72701072e0 0x7f7270074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 -- 172.21.15.87:0/3880582554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7270102210 msgr2=0x7f7270074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7270102210 0x7f7270074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 -- 172.21.15.87:0/3880582554 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72701a7280 con 0x7f7270074cd0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726f7fe640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f726800b990 tx=0x7f726800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726cff9640 1 -- 172.21.15.87:0/3880582554 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268019070 con 0x7f7270074cd0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72701a7570 con 0x7f7270074cd0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726cff9640 1 -- 172.21.15.87:0/3880582554 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726800cba0 con 0x7f7270074cd0 2024-04-22T13:35:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.195+0000 7f726cff9640 1 -- 172.21.15.87:0/3880582554 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680075d0 con 0x7f7270074cd0 2024-04-22T13:35:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.196+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72701a7ac0 con 0x7f7270074cd0 2024-04-22T13:35:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.199+0000 7f7278045740 1 Processor -- start 2024-04-22T13:35:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.200+0000 7f726cff9640 1 -- 172.21.15.87:0/3880582554 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f726801c070 con 0x7f7270074cd0 2024-04-22T13:35:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.200+0000 7f7278045740 1 -- start start 2024-04-22T13:35:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ff58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f726cff9640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 0x7f725c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b8a2f5a1f0 con 0x55b8a2ffea00 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b8a2ef74b0 con 0x55b8a2ff54d0 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.201+0000 7f7278045740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b8a2f561b0 con 0x55b8a2ff5e80 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726cff9640 1 -- 172.21.15.87:0/3880582554 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f72680cfd20 con 0x7f7270074cd0 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54098/0 (socket says 172.21.15.87:54098) 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 -- 172.21.15.87:0/697867 learned_addr learned my addr 172.21.15.87:0/697867 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 -- 172.21.15.87:0/697867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ff5e80 msgr2=0x55b8a2ffe430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffe430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f7267fff640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 0x7f725c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 -- 172.21.15.87:0/697867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff54d0 msgr2=0x55b8a2ff58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ff58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.202+0000 7f726f7fe640 1 -- 172.21.15.87:0/697867 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a2ee91b0 con 0x55b8a2ffea00 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.203+0000 7f7267fff640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 0x7f725c0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7270075c90 tx=0x7f7258006f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.203+0000 7f726f7fe640 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7268098810 tx=0x7f726809b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576e0389119d80e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.203+0000 7f72657fa640 1 -- 172.21.15.87:0/697867 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726809e030 con 0x55b8a2ffea00 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.203+0000 7f72657fa640 1 -- 172.21.15.87:0/697867 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726801e440 con 0x55b8a2ffea00 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.203+0000 7f72657fa640 1 -- 172.21.15.87:0/697867 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726809a140 con 0x55b8a2ffea00 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 -- 172.21.15.87:0/697867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 msgr2=0x55b8a3000e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7268098810 tx=0x7f726809b980 comp rx=0 tx=0).stop 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 -- 172.21.15.87:0/697867 shutdown_connections 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3000e50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 --2- 172.21.15.87:0/697867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ff58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 -- 172.21.15.87:0/697867 >> 172.21.15.87:0/697867 conn(0x55b8a2fefba0 msgr2=0x55b8a2feff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 -- 172.21.15.87:0/697867 shutdown_connections 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.204+0000 7f7278045740 1 -- 172.21.15.87:0/697867 wait complete. 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 -- start start 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 Processor -- start 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2ffbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b8a3001970 con 0x55b8a2ff54d0 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b8a2f13500 con 0x55b8a2ff5e80 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b8a3001af0 con 0x55b8a2ffea00 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f7278045740 10 client.0 _ll_register_callbacks cb 0x55b8a2ee1b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f726effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f726effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54112/0 (socket says 172.21.15.87:54112) 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.205+0000 7f726effd640 1 -- 172.21.15.87:0/470162304 learned_addr learned my addr 172.21.15.87:0/470162304 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 -- 172.21.15.87:0/470162304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 msgr2=0x55b8a2ffbcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2ffbcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 -- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 msgr2=0x55b8a2ffb6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffb6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 -- 172.21.15.87:0/470162304 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a2c6c3f0 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f726effd640 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7260005ab0 tx=0x7f726002ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1412834146c04a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f72567fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260041020 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f726801e440 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7260005250 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f726004c050 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f7278045740 10 client.16668 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.206+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b8a2d367d0 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f726801e440 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f72600035f0 con 0x55b8a2ff54d0 2024-04-22T13:35:23.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 10 client.16668 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 20 client.16668 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a300ec80 0x55b8a3011120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 4 client.16668 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.207+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8a2ef9930 con 0x55b8a300ec80 2024-04-22T13:35:23.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.210+0000 7f7267fff640 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a300ec80 0x55b8a3011120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.213+0000 7f72567fc640 10 client.16668 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.213+0000 7f7267fff640 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a300ec80 0x55b8a3011120 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5d60128527ff17 server_cookie=8f0c809b3f6edc14 in_seq=0 out_seq=0 2024-04-22T13:35:23.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.215+0000 7f72567fc640 1 -- 172.21.15.87:0/470162304 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8a2ef9930 con 0x55b8a300ec80 2024-04-22T13:35:23.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.215+0000 7f72567fc640 10 client.16668 handle_command_reply: tid=0 2024-04-22T13:35:23.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 client.16668 shutdown 2024-04-22T13:35:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 msgr2=0x55b8a2ffe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7260005ab0 tx=0x7f726002ca30 comp rx=0 tx=0).stop 2024-04-22T13:35:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 shutdown_connections 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a300ec80 0x55b8a3011120 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2ffbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a2ffb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f72567fc640 0 client.16668 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 --2- 172.21.15.87:0/470162304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ff54d0 0x55b8a2ffe330 secure :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7260005ab0 tx=0x7f726002ca30 comp rx=0 tx=0).stop 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.216+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 >> 172.21.15.87:0/470162304 conn(0x55b8a2fefba0 msgr2=0x55b8a2ff18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.217+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 shutdown_connections 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.217+0000 7f7278045740 1 -- 172.21.15.87:0/470162304 wait complete. 2024-04-22T13:35:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.217+0000 7f7278045740 20 client.16668 trim_cache size 0 max 16384 2024-04-22T13:35:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.235+0000 7f7278045740 1 Processor -- start 2024-04-22T13:35:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.235+0000 7f7278045740 1 -- start start 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a300a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2feea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f72701b26d0 con 0x55b8a2ffddf0 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72701affa0 con 0x55b8a2ff5e80 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f7278045740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f72701ade30 con 0x55b8a2ffea00 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54118/0 (socket says 172.21.15.87:54118) 2024-04-22T13:35:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 -- 172.21.15.87:0/901626492 learned_addr learned my addr 172.21.15.87:0/901626492 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 -- 172.21.15.87:0/901626492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 msgr2=0x55b8a2feea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2feea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 -- 172.21.15.87:0/901626492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 msgr2=0x55b8a300a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a300a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.236+0000 7f726f7fe640 1 -- 172.21.15.87:0/901626492 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a2feb470 con 0x55b8a2ffddf0 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.237+0000 7f726f7fe640 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f72680cd6a0 tx=0x7f7268014c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595de1f3dfc48ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.237+0000 7f7264ff9640 1 -- 172.21.15.87:0/901626492 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680b4020 con 0x55b8a2ffddf0 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.237+0000 7f7264ff9640 1 -- 172.21.15.87:0/901626492 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7268099ee0 con 0x55b8a2ffddf0 2024-04-22T13:35:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.237+0000 7f7264ff9640 1 -- 172.21.15.87:0/901626492 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726809e030 con 0x55b8a2ffddf0 2024-04-22T13:35:23.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- 172.21.15.87:0/901626492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 msgr2=0x55b8a2ffe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f72680cd6a0 tx=0x7f7268014c50 comp rx=0 tx=0).stop 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- 172.21.15.87:0/901626492 shutdown_connections 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffddf0 0x55b8a2ffe250 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffea00 0x55b8a2feea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 --2- 172.21.15.87:0/901626492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a300a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- 172.21.15.87:0/901626492 >> 172.21.15.87:0/901626492 conn(0x55b8a2fefba0 msgr2=0x55b8a2fee370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- 172.21.15.87:0/901626492 shutdown_connections 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- 172.21.15.87:0/901626492 wait complete. 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.238+0000 7f7278045740 1 -- start start 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 Processor -- start 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a3007250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3002d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f72701ae480 con 0x55b8a2ffea00 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72701af8d0 con 0x55b8a2ff5e80 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f72701aeb30 con 0x55b8a2ffddf0 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7278045740 10 client.0 _ll_register_callbacks cb 0x7f7260004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:46912/0 (socket says 172.21.15.87:46912) 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 -- 172.21.15.87:0/925961301 learned_addr learned my addr 172.21.15.87:0/925961301 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f726f7fe640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3002d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f726effd640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a3007250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 -- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 msgr2=0x55b8a3007250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a3007250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 -- 172.21.15.87:0/925961301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 msgr2=0x55b8a3002d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3002d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.239+0000 7f7267fff640 1 -- 172.21.15.87:0/925961301 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a3074260 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f7267fff640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55b8a2ffaf20 tx=0x7f725800a720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3200d37ee0ae33d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f72567fc640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258019030 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7258013bf0 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7268099ee0 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7258033050 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.240+0000 7f7278045740 10 client.25891 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b8a3075410 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7258023e50 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f72580206a0 con 0x55b8a2ffddf0 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 10 client.25891 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 10 client.25891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:23.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 20 client.25891 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:23.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a3075b70 0x55b8a3077f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 4 client.25891 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:23.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.241+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7268099ee0 con 0x55b8a3075b70 2024-04-22T13:35:23.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.243+0000 7f726effd640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a3075b70 0x55b8a3077f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.247+0000 7f72567fc640 10 client.25891 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.247+0000 7f726effd640 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a3075b70 0x55b8a3077f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=106159e38d23b8b7 server_cookie=d8d752295d4c6b69 in_seq=0 out_seq=0 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f72567fc640 1 -- 172.21.15.87:0/925961301 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7268099ee0 con 0x55b8a3075b70 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f72567fc640 10 client.25891 handle_command_reply: tid=0 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 client.25891 shutdown 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 msgr2=0x55b8a30025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55b8a2ffaf20 tx=0x7f725800a720 comp rx=0 tx=0).stop 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 shutdown_connections 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b8a3075b70 0x55b8a3077f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b8a2ffea00 0x55b8a3002d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b8a2ffddf0 0x55b8a30025c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f72567fc640 0 client.25891 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 --2- 172.21.15.87:0/925961301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8a2ff5e80 0x55b8a3007250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 >> 172.21.15.87:0/925961301 conn(0x55b8a2fefba0 msgr2=0x55b8a2feec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 shutdown_connections 2024-04-22T13:35:23.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.251+0000 7f7278045740 1 -- 172.21.15.87:0/925961301 wait complete. 2024-04-22T13:35:23.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f7278045740 20 client.25891 trim_cache size 0 max 16384 2024-04-22T13:35:23.249 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 msgr2=0x7f725c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 0x7f725c0a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7270075c90 tx=0x7f7258006f90 comp rx=0 tx=0).stop 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 msgr2=0x7f72701a6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f726800b990 tx=0x7f726800be50 comp rx=0 tx=0).stop 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 shutdown_connections 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f725c0a3d20 0x7f725c0a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7270074cd0 0x7f72701a6be0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72701072e0 0x7f7270074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 --2- 172.21.15.87:0/3880582554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7270102210 0x7f7270074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 >> 172.21.15.87:0/3880582554 conn(0x7f72700fdab0 msgr2=0x7f72700fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 shutdown_connections 2024-04-22T13:35:23.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:23.252+0000 7f727545b640 1 -- 172.21.15.87:0/3880582554 wait complete. 2024-04-22T13:35:23.262 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:23 smithi087 ceph-mon[27999]: pgmap v108: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 659 MiB/s wr, 4.11k op/s 2024-04-22T13:35:23.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:23 smithi132 ceph-mon[31531]: pgmap v108: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 659 MiB/s wr, 4.11k op/s 2024-04-22T13:35:23.570 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:23 smithi120 ceph-mon[31101]: pgmap v108: 97 pgs: 97 active+clean; 26 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 659 MiB/s wr, 4.11k op/s 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 -- 172.21.15.87:0/4235463707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103200 msgr2=0x7fd2841035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 --2- 172.21.15.87:0/4235463707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103200 0x7fd2841035e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd274009900 tx=0x7fd27402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 -- 172.21.15.87:0/4235463707 shutdown_connections 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 --2- 172.21.15.87:0/4235463707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd28410c540 0x7fd28410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 --2- 172.21.15.87:0/4235463707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd284103b20 0x7fd28410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 --2- 172.21.15.87:0/4235463707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103200 0x7fd2841035e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 -- 172.21.15.87:0/4235463707 >> 172.21.15.87:0/4235463707 conn(0x7fd284077fd0 msgr2=0x7fd2840783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 -- 172.21.15.87:0/4235463707 shutdown_connections 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 -- 172.21.15.87:0/4235463707 wait complete. 2024-04-22T13:35:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.928+0000 7fd28a796640 1 Processor -- start 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 -- start start 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 0x7fd284100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd28410c540 0x7fd2841018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd284112280 con 0x7fd284103b20 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd284112100 con 0x7fd284103200 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd28a796640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd284112400 con 0x7fd28410c540 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54156/0 (socket says 172.21.15.87:54156) 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 -- 172.21.15.87:0/3937380906 learned_addr learned my addr 172.21.15.87:0/3937380906 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 -- 172.21.15.87:0/3937380906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd28410c540 msgr2=0x7fd2841018d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd28410c540 0x7fd2841018d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd2837fe640 1 -- 172.21.15.87:0/3937380906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 msgr2=0x7fd284100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.929+0000 7fd283fff640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 0x7fd284100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 0x7fd284100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2837fe640 1 -- 172.21.15.87:0/3937380906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2841a6d20 con 0x7fd284103b20 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd283fff640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 0x7fd284100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd27800b4a0 tx=0x7fd27800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2817fa640 1 -- 172.21.15.87:0/3937380906 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278014070 con 0x7fd284103b20 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2841a6fb0 con 0x7fd284103b20 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2817fa640 1 -- 172.21.15.87:0/3937380906 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd278010ac0 con 0x7fd284103b20 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd2817fa640 1 -- 172.21.15.87:0/3937380906 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278017c30 con 0x7fd284103b20 2024-04-22T13:35:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.930+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2841a7480 con 0x7fd284103b20 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.933+0000 7fd2817fa640 1 -- 172.21.15.87:0/3937380906 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd278017070 con 0x7fd284103b20 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.933+0000 7fd28d389740 1 Processor -- start 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 -- start start 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a727024620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a7270304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a726f898d0 con 0x55a727024b60 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a726f26e80 con 0x55a7270258d0 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd28d389740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a726f85890 con 0x55a72702e0e0 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54170/0 (socket says 172.21.15.87:54170) 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 -- 172.21.15.87:0/2146014355 learned_addr learned my addr 172.21.15.87:0/2146014355 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd288d33640 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a727024620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd2817fa640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 0x7fd26c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 -- 172.21.15.87:0/2146014355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 msgr2=0x55a7270304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a7270304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 -- 172.21.15.87:0/2146014355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 msgr2=0x55a727024620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd283fff640 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a727024620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.934+0000 7fd288d33640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 0x7fd26c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd283fff640 1 -- 172.21.15.87:0/2146014355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a726f48320 con 0x55a727024b60 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd2817fa640 1 -- 172.21.15.87:0/3937380906 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd2780cfac0 con 0x7fd284103b20 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd283fff640 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd274036040 tx=0x7fd274005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe63ddf4a3805f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd26a7fc640 1 -- 172.21.15.87:0/2146014355 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274041020 con 0x55a727024b60 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd288d33640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 0x7fd26c0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c4e0 tx=0x7fd27c004b50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd26a7fc640 1 -- 172.21.15.87:0/2146014355 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2740054f0 con 0x55a727024b60 2024-04-22T13:35:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.935+0000 7fd26a7fc640 1 -- 172.21.15.87:0/2146014355 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27402ce60 con 0x55a727024b60 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- 172.21.15.87:0/2146014355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 msgr2=0x55a727024fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd274036040 tx=0x7fd274005e80 comp rx=0 tx=0).stop 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- 172.21.15.87:0/2146014355 shutdown_connections 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a7270304d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727024fe0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 --2- 172.21.15.87:0/2146014355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a727024620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- 172.21.15.87:0/2146014355 >> 172.21.15.87:0/2146014355 conn(0x55a726febc90 msgr2=0x55a72701f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- 172.21.15.87:0/2146014355 shutdown_connections 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- 172.21.15.87:0/2146014355 wait complete. 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.936+0000 7fd28d389740 1 -- start start 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 Processor -- start 2024-04-22T13:35:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a72702ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a72702b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a727030db0 con 0x55a727024b60 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a726f11eb0 con 0x55a7270258d0 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a727030f30 con 0x55a72702e0e0 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd28d389740 10 client.0 _ll_register_callbacks cb 0x55a726f3eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54178/0 (socket says 172.21.15.87:54178) 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.937+0000 7fd283fff640 1 -- 172.21.15.87:0/1789461341 learned_addr learned my addr 172.21.15.87:0/1789461341 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd2837fe640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a72702ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 -- 172.21.15.87:0/1789461341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 msgr2=0x55a72702b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a72702b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 -- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 msgr2=0x55a72702ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a72702ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 -- 172.21.15.87:0/1789461341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a726c9c210 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd283fff640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd274005fd0 tx=0x7fd274005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd24deb88ac638a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd2637fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274041040 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd274003a00 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.938+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2740054f0 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd274054050 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 10 client.16686 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a726d68620 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2740054f0 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd274046c70 con 0x55a727024b60 2024-04-22T13:35:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 10 client.16686 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 10 client.16686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 20 client.16686 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.939+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a727040350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.940+0000 7fd28d389740 4 client.16686 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.940+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a726f9d880 con 0x55a72703df60 2024-04-22T13:35:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.940+0000 7fd2837fe640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a727040350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.941+0000 7fd2837fe640 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a727040350 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b505bf83c4acaf08 server_cookie=8cd20db2b1bca916 in_seq=0 out_seq=0 2024-04-22T13:35:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.941+0000 7fd2637fe640 10 client.16686 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd2637fe640 1 -- 172.21.15.87:0/1789461341 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a726f9d880 con 0x55a72703df60 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd2637fe640 10 client.16686 handle_command_reply: tid=0 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 client.16686 shutdown 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 msgr2=0x55a72702dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd274005fd0 tx=0x7fd274005d50 comp rx=0 tx=0).stop 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 shutdown_connections 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a727040350 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a72702b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd2637fe640 0 client.16686 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7270258d0 0x55a72702ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 --2- 172.21.15.87:0/1789461341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a72702dac0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 >> 172.21.15.87:0/1789461341 conn(0x55a726febc90 msgr2=0x55a7270215a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 shutdown_connections 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.944+0000 7fd28d389740 1 -- 172.21.15.87:0/1789461341 wait complete. 2024-04-22T13:35:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.945+0000 7fd28d389740 20 client.16686 trim_cache size 0 max 16384 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 Processor -- start 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 -- start start 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a7270258d0 0x55a727029fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a72702e0e0 0x55a72701e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841b2560 con 0x55a727024b60 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841afe30 con 0x55a72702e0e0 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd28d389740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841adcc0 con 0x55a7270258d0 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd283fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54194/0 (socket says 172.21.15.87:54194) 2024-04-22T13:35:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.971+0000 7fd283fff640 1 -- 172.21.15.87:0/1311362140 learned_addr learned my addr 172.21.15.87:0/1311362140 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 -- 172.21.15.87:0/1311362140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a7270258d0 msgr2=0x55a727029fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a7270258d0 0x55a727029fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 -- 172.21.15.87:0/1311362140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a72702e0e0 msgr2=0x55a72701e750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a72702e0e0 0x55a72701e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 -- 172.21.15.87:0/1311362140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7270293a0 con 0x55a727024b60 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd283fff640 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd27402cb50 tx=0x7fd27402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e1e69b6a4ee8af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd269ffb640 1 -- 172.21.15.87:0/1311362140 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274003800 con 0x55a727024b60 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd269ffb640 1 -- 172.21.15.87:0/1311362140 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd27404b070 con 0x55a727024b60 2024-04-22T13:35:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.972+0000 7fd269ffb640 1 -- 172.21.15.87:0/1311362140 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274034e60 con 0x55a727024b60 2024-04-22T13:35:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 -- 172.21.15.87:0/1311362140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 msgr2=0x55a727026530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd27402cb50 tx=0x7fd27402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 -- 172.21.15.87:0/1311362140 shutdown_connections 2024-04-22T13:35:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a72702e0e0 0x55a72701e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a7270258d0 0x55a727029fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 --2- 172.21.15.87:0/1311362140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a727024b60 0x55a727026530 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.973+0000 7fd28d389740 1 -- 172.21.15.87:0/1311362140 >> 172.21.15.87:0/1311362140 conn(0x55a726febc90 msgr2=0x55a72701e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.974+0000 7fd28d389740 1 -- 172.21.15.87:0/1311362140 shutdown_connections 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.974+0000 7fd28d389740 1 -- 172.21.15.87:0/1311362140 wait complete. 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.974+0000 7fd28d389740 1 -- start start 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.974+0000 7fd28d389740 1 Processor -- start 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a727024b60 0x55a7270364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a727031da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54204/0 (socket says 172.21.15.87:54204) 2024-04-22T13:35:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd2837fe640 1 -- 172.21.15.87:0/3764571164 learned_addr learned my addr 172.21.15.87:0/3764571164 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841ae310 con 0x55a7270258d0 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841af760 con 0x55a727024b60 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841ae9c0 con 0x55a72702e0e0 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.975+0000 7fd28d389740 10 client.0 _ll_register_callbacks cb 0x7fd27c0095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 -- 172.21.15.87:0/3764571164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 msgr2=0x55a727031da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a727031da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 -- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a727024b60 msgr2=0x55a7270364c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a727024b60 0x55a7270364c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7270325a0 con 0x55a7270258d0 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2837fe640 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd278098040 tx=0x7fd278010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66a65ccecdcc0af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2637fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27801e040 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd278010c70 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.976+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd27404b070 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd27809ff90 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 10 client.16692 fetch_fsmap learned FSMap version 21 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a7270a40c0 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2780a4850 con 0x55a7270258d0 2024-04-22T13:35:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd2780bc0b0 con 0x55a7270258d0 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 10 client.16692 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 20 client.16692 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a7270a6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 4 client.16692 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.977+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd27404b070 con 0x55a72703df60 2024-04-22T13:35:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.980+0000 7fd283fff640 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a7270a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.983+0000 7fd283fff640 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a7270a6830 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4d5e8e91c29a19 server_cookie=76c5840467a838e0 in_seq=0 out_seq=0 2024-04-22T13:35:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:23.983+0000 7fd2637fe640 10 client.16692 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.025+0000 7f8ea5ffb640 1 -- 172.21.15.87:0/1439669261 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b3145a8980 con 0x55b3146bd3d0 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.025+0000 7f8ea5ffb640 10 client.25873 handle_command_reply: tid=0 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.025+0000 7f8ec73b8740 1 client.25873 shutdown 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 msgr2=0x55b31464dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b31464dcb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400ac00 tx=0x7f8eb4014590 comp rx=0 tx=0).stop 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 shutdown_connections 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b3146bd3d0 0x55b3146bf7f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b314646090 0x55b31464a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b31463d4b0 0x55b31464e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 --2- 172.21.15.87:0/1439669261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b31463cb90 0x55b31464dcb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 >> 172.21.15.87:0/1439669261 conn(0x55b3146372c0 msgr2=0x55b3146386f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.026+0000 7f8ea5ffb640 0 client.25873 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.027+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 shutdown_connections 2024-04-22T13:35:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.028+0000 7f8ec73b8740 1 -- 172.21.15.87:0/1439669261 wait complete. 2024-04-22T13:35:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.028+0000 7f8ec73b8740 20 client.25873 trim_cache size 0 max 16384 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 21, 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 14, 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:35:24.024 INFO:teuthology.orchestra.run.smithi087.stdout: "34d09ee7": { 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 14, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 71.2, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:35:24.025 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.026 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 msgr2=0x7f8ea80a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 0x7f8ea80a6370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00ffd0 tx=0x7f8eac00e780 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 msgr2=0x7f8ebc119520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005ab0 tx=0x7f8eb8004a10 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 shutdown_connections 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ea80a3eb0 0x7f8ea80a6370 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ebc110550 0x7f8ebc119520 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc1099f0 0x7f8ebc11d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 --2- 172.21.15.87:0/3312151264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ebc108f50 0x7f8ebc11d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 >> 172.21.15.87:0/3312151264 conn(0x7f8ebc06ce90 msgr2=0x7f8ebc070990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.029+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 shutdown_connections 2024-04-22T13:35:24.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.030+0000 7f8ec47c7640 1 -- 172.21.15.87:0/3312151264 wait complete. 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.045+0000 7fd2637fe640 1 -- 172.21.15.87:0/3764571164 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd27404b070 con 0x55a72703df60 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.045+0000 7fd2637fe640 10 client.16692 handle_command_reply: tid=0 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.045+0000 7fd28d389740 1 client.16692 shutdown 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 msgr2=0x55a727031860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd278098040 tx=0x7fd278010f90 comp rx=0 tx=0).stop 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 shutdown_connections 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a72703df60 0x55a7270a6830 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a72702e0e0 0x55a727031da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd2637fe640 0 client.16692 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a7270258d0 0x55a727031860 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 --2- 172.21.15.87:0/3764571164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a727024b60 0x55a7270364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 >> 172.21.15.87:0/3764571164 conn(0x55a726febc90 msgr2=0x55a727029720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 shutdown_connections 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 1 -- 172.21.15.87:0/3764571164 wait complete. 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.046+0000 7fd28d389740 20 client.16692 trim_cache size 0 max 16384 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:35:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 msgr2=0x7fd26c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 0x7fd26c0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c4e0 tx=0x7fd27c004b50 comp rx=0 tx=0).stop 2024-04-22T13:35:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 msgr2=0x7fd284101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd27800b4a0 tx=0x7fd27800b960 comp rx=0 tx=0).stop 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 shutdown_connections 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd26c0a3dc0 0x7fd26c0a6280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd28410c540 0x7fd2841018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd284103b20 0x7fd284101390 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 --2- 172.21.15.87:0/3937380906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd284103200 0x7fd284100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 >> 172.21.15.87:0/3937380906 conn(0x7fd284077fd0 msgr2=0x7fd2840fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 shutdown_connections 2024-04-22T13:35:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:24.047+0000 7fd28a796640 1 -- 172.21.15.87:0/3937380906 wait complete. 2024-04-22T13:35:24.047 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 34d09ee7, quiesce took 3.5 seconds. Will release after: 11.2 2024-04-22T13:35:24.264 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:35:24.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:35:24.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 -- 172.21.15.87:0/1405027024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff280101a50 msgr2=0x7ff280105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 --2- 172.21.15.87:0/1405027024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff280101a50 0x7ff280105a90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7ff270009900 tx=0x7ff27002bc30 comp rx=0 tx=0).stop 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 -- 172.21.15.87:0/1405027024 shutdown_connections 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 --2- 172.21.15.87:0/1405027024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 0x7ff28010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 --2- 172.21.15.87:0/1405027024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff280101a50 0x7ff280105a90 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 --2- 172.21.15.87:0/1405027024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2801010a0 0x7ff280101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 -- 172.21.15.87:0/1405027024 >> 172.21.15.87:0/1405027024 conn(0x7ff280077eb0 msgr2=0x7ff2800782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 -- 172.21.15.87:0/1405027024 shutdown_connections 2024-04-22T13:35:24.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.372+0000 7ff2883a4640 1 -- 172.21.15.87:0/1405027024 wait complete. 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 Processor -- start 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 -- start start 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 0x7ff280197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 0x7ff28019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801110b0 con 0x7ff2801010a0 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff280110f30 con 0x7ff28019bed0 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff280111230 con 0x7ff280106170 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54210/0 (socket says 172.21.15.87:54210) 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 -- 172.21.15.87:0/3497427845 learned_addr learned my addr 172.21.15.87:0/3497427845 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff28593f640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 0x7ff280197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286941640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 0x7ff28019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 -- 172.21.15.87:0/3497427845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 msgr2=0x7ff280197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 0x7ff280197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 -- 172.21.15.87:0/3497427845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 msgr2=0x7ff28019c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 0x7ff28019c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 -- 172.21.15.87:0/3497427845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28019c9c0 con 0x7ff2801010a0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286941640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 0x7ff28019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff286140640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00d950 tx=0x7ff27c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2777fe640 1 -- 172.21.15.87:0/3497427845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c017030 con 0x7ff2801010a0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2777fe640 1 -- 172.21.15.87:0/3497427845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27c004020 con 0x7ff2801010a0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28019ccb0 con 0x7ff2801010a0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.373+0000 7ff2777fe640 1 -- 172.21.15.87:0/3497427845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0047a0 con 0x7ff2801010a0 2024-04-22T13:35:24.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.374+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2801a4470 con 0x7ff2801010a0 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff2777fe640 1 -- 172.21.15.87:0/3497427845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff27c009a30 con 0x7ff2801010a0 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 Processor -- start 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 -- start start 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878ee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.376+0000 7ff28af92740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e587849f00 con 0x55e5878eee10 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff28af92740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e5877e7590 con 0x55e5878e58e0 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff28af92740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e587845ec0 con 0x55e5878e6290 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878e5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48424/0 (socket says 172.21.15.87:48424) 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286941640 1 -- 172.21.15.87:0/3393859141 learned_addr learned my addr 172.21.15.87:0/3393859141 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff28593f640 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff2777fe640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 0x7ff26c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 -- 172.21.15.87:0/3393859141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 msgr2=0x55e5878e5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878e5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 -- 172.21.15.87:0/3393859141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 msgr2=0x55e5878f1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 -- 172.21.15.87:0/3393859141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5876247e0 con 0x55e5878e6290 2024-04-22T13:35:24.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286941640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 0x7ff26c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff286140640 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878ee840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0cd9d0 tx=0x7ff27c0c7330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=203a654ab2b70b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.377+0000 7ff274bfe640 1 -- 172.21.15.87:0/3393859141 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c09bff0 con 0x55e5878e6290 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff274bfe640 1 -- 172.21.15.87:0/3393859141 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27c09c920 con 0x55e5878e6290 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff274bfe640 1 -- 172.21.15.87:0/3393859141 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0a1760 con 0x55e5878e6290 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff2777fe640 1 -- 172.21.15.87:0/3497427845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff27c0d0210 con 0x7ff2801010a0 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff28af92740 1 -- 172.21.15.87:0/3393859141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 msgr2=0x55e5878ee840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff286941640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 0x7ff26c0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff278005c40 tx=0x7ff278004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.378+0000 7ff28af92740 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878ee840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0cd9d0 tx=0x7ff27c0c7330 comp rx=0 tx=0).stop 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 -- 172.21.15.87:0/3393859141 shutdown_connections 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878ee840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 --2- 172.21.15.87:0/3393859141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878e5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 -- 172.21.15.87:0/3393859141 >> 172.21.15.87:0/3393859141 conn(0x55e5878dffb0 msgr2=0x55e5878e0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 -- 172.21.15.87:0/3393859141 shutdown_connections 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.379+0000 7ff28af92740 1 -- 172.21.15.87:0/3393859141 wait complete. 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 -- start start 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 Processor -- start 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e5878ec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e5878f3ed0 con 0x55e5878e6290 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e587803580 con 0x55e5878e58e0 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e5878f4050 con 0x55e5878eee10 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28af92740 10 client.0 _ll_register_callbacks cb 0x55e587809650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54226/0 (socket says 172.21.15.87:54226) 2024-04-22T13:35:24.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 -- 172.21.15.87:0/1497746186 learned_addr learned my addr 172.21.15.87:0/1497746186 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff286140640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff286941640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e5878ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 -- 172.21.15.87:0/1497746186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 msgr2=0x55e5878ec800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e5878ec800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 -- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 msgr2=0x55e5878ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878ebd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 -- 172.21.15.87:0/1497746186 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e587684f70 con 0x55e5878e6290 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff286140640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff28593f640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff270009900 tx=0x7ff27002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827e0d8b39605c93 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.380+0000 7ff269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27002c610 con 0x55e5878e6290 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff270005030 con 0x55e5878e6290 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff27c09c920 con 0x55e5878e6290 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff270005230 con 0x55e5878e6290 2024-04-22T13:35:24.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 10 client.16707 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e58751c170 con 0x55e5878e6290 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27c09c920 con 0x55e5878e6290 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff270003680 con 0x55e5878e6290 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 10 client.16707 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 20 client.16707 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587901420 0x55e587903850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 4 client.16707 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e587857920 con 0x55e587901420 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.381+0000 7ff286140640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587901420 0x55e587903850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.382+0000 7ff286140640 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587901420 0x55e587903850 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ea8f7227366e5a server_cookie=4001b692e0ed0619 in_seq=0 out_seq=0 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.382+0000 7ff269ffb640 10 client.16707 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.382+0000 7ff269ffb640 1 -- 172.21.15.87:0/1497746186 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e587857920 con 0x55e587901420 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.382+0000 7ff269ffb640 10 client.16707 handle_command_reply: tid=0 2024-04-22T13:35:24.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.382+0000 7ff28af92740 1 client.16707 shutdown 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 msgr2=0x55e5878ec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff270009900 tx=0x7ff27002ca10 comp rx=0 tx=0).stop 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 shutdown_connections 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587901420 0x55e587903850 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e5878ec800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e6290 0x55e5878ec2c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff269ffb640 0 client.16707 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 --2- 172.21.15.87:0/1497746186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 >> 172.21.15.87:0/1497746186 conn(0x55e5878dffb0 msgr2=0x55e5878e1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 shutdown_connections 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 1 -- 172.21.15.87:0/1497746186 wait complete. 2024-04-22T13:35:24.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.383+0000 7ff28af92740 20 client.16707 trim_cache size 0 max 16384 2024-04-22T13:35:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 Processor -- start 2024-04-22T13:35:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 -- start start 2024-04-22T13:35:24.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 0x55e587909120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e587909660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801ae170 con 0x55e5878e58e0 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801aba40 con 0x55e5878e6290 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28af92740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801a9890 con 0x55e5878eee10 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff286140640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff286140640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54228/0 (socket says 172.21.15.87:54228) 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff286140640 1 -- 172.21.15.87:0/321424428 learned_addr learned my addr 172.21.15.87:0/321424428 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.400+0000 7ff28593f640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 0x55e587909120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286941640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e587909660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 -- 172.21.15.87:0/321424428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 msgr2=0x55e587909660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e587909660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 -- 172.21.15.87:0/321424428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 msgr2=0x55e587909120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 0x55e587909120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 -- 172.21.15.87:0/321424428 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5878f3b60 con 0x55e5878e58e0 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286941640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e587909660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28593f640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 0x55e587909120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff286140640 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0045f0 tx=0x7ff27c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2a68d98beb549e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff26bfff640 1 -- 172.21.15.87:0/321424428 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0a2dc0 con 0x55e5878e58e0 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff26bfff640 1 -- 172.21.15.87:0/321424428 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27c09c920 con 0x55e5878e58e0 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff26bfff640 1 -- 172.21.15.87:0/321424428 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c09bb80 con 0x55e5878e58e0 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 -- 172.21.15.87:0/321424428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 msgr2=0x55e5878fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0045f0 tx=0x7ff27c000bc0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 -- 172.21.15.87:0/321424428 shutdown_connections 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878eee10 0x55e587909660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e6290 0x55e587909120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 --2- 172.21.15.87:0/321424428 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878e58e0 0x55e5878fcd90 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.401+0000 7ff28af92740 1 -- 172.21.15.87:0/321424428 >> 172.21.15.87:0/321424428 conn(0x55e5878dffb0 msgr2=0x55e5878edd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- 172.21.15.87:0/321424428 shutdown_connections 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- 172.21.15.87:0/321424428 wait complete. 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- start start 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 Processor -- start 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801a9f00 con 0x55e5878eee10 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801ab370 con 0x55e5878e58e0 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801aa5d0 con 0x55e5878e6290 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28af92740 10 client.0 _ll_register_callbacks cb 0x7ff270009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54232/0 (socket says 172.21.15.87:54232) 2024-04-22T13:35:24.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 -- 172.21.15.87:0/1459958949 learned_addr learned my addr 172.21.15.87:0/1459958949 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff28593f640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286140640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 -- 172.21.15.87:0/1459958949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 msgr2=0x55e5878f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 -- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 msgr2=0x55e5878f9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878f9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286941640 1 -- 172.21.15.87:0/1459958949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5878f5ff0 con 0x55e5878eee10 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.402+0000 7ff286140640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff286941640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff278013ee0 tx=0x7ff278004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c47f1c63d8ba14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278011cd0 con 0x55e5878eee10 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff27801f420 con 0x55e5878eee10 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff27c09c920 con 0x55e5878eee10 2024-04-22T13:35:24.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff27801f620 con 0x55e5878eee10 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 10 client.16713 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e5879685c0 con 0x55e5878eee10 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff27c09c920 con 0x55e5878eee10 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff278032020 con 0x55e5878eee10 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 10 client.16713 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 10 client.16713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 20 client.16713 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587968cf0 0x55e58796b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 4 client.16713 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.403+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e587851790 con 0x55e587968cf0 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.404+0000 7ff286140640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587968cf0 0x55e58796b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.404+0000 7ff286140640 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587968cf0 0x55e58796b110 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e896865842c1fe server_cookie=4ca42bafa9d0c471 in_seq=0 out_seq=0 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.404+0000 7ff269ffb640 10 client.16713 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff269ffb640 1 -- 172.21.15.87:0/1459958949 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e587851790 con 0x55e587968cf0 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff269ffb640 10 client.16713 handle_command_reply: tid=0 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 client.16713 shutdown 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 msgr2=0x55e5878f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff278013ee0 tx=0x7ff278004ab0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 shutdown_connections 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e587968cf0 0x55e58796b110 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e5878eee10 0x55e5878f57f0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e5878e6290 0x55e5878f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff269ffb640 0 client.16713 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 --2- 172.21.15.87:0/1459958949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5878e58e0 0x55e5878f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 >> 172.21.15.87:0/1459958949 conn(0x55e5878dffb0 msgr2=0x55e5878ed380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 shutdown_connections 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.405+0000 7ff28af92740 1 -- 172.21.15.87:0/1459958949 wait complete. 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff28af92740 20 client.16713 trim_cache size 0 max 16384 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "6b48e44e-f626-45f3-baad-1a2b37809b4c", 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:35:24.408 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 msgr2=0x7ff26c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 0x7ff26c0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff278005c40 tx=0x7ff278004a40 comp rx=0 tx=0).stop 2024-04-22T13:35:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 msgr2=0x7ff280197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00d950 tx=0x7ff27c00de10 comp rx=0 tx=0).stop 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 shutdown_connections 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff26c0a3d20 0x7ff26c0a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff28019bed0 0x7ff28019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff280106170 0x7ff280197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 --2- 172.21.15.87:0/3497427845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2801010a0 0x7ff280197610 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 >> 172.21.15.87:0/3497427845 conn(0x7ff280077eb0 msgr2=0x7ff2800ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 shutdown_connections 2024-04-22T13:35:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.406+0000 7ff2883a4640 1 -- 172.21.15.87:0/3497427845 wait complete. 2024-04-22T13:35:24.411 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:24 smithi087 ceph-mon[27999]: daemon mds.f restarted 2024-04-22T13:35:24.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:35:24.413 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:35:24.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:24 smithi132 ceph-mon[31531]: daemon mds.f restarted 2024-04-22T13:35:24.463 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:35:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: ignoring --setuser ceph since I am not root 2024-04-22T13:35:24.463 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:35:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: ignoring --setgroup ceph since I am not root 2024-04-22T13:35:24.463 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:35:24 smithi132 ceph-mds[61948]: -- 172.21.15.132:0/3773944456 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f1b643a40 con 0x561f1b616800 2024-04-22T13:35:24.463 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:35:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: starting mds.f at 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- 172.21.15.87:0/1234469958 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0101080 msgr2=0x7fd9a0101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1234469958 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0101080 0x7fd9a0101460 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- 172.21.15.87:0/1234469958 shutdown_connections 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1234469958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0106150 0x7fd9a010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1234469958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0101a30 0x7fd9a0105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1234469958 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0101080 0x7fd9a0101460 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- 172.21.15.87:0/1234469958 >> 172.21.15.87:0/1234469958 conn(0x7fd9a0078000 msgr2=0x7fd9a0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- 172.21.15.87:0/1234469958 shutdown_connections 2024-04-22T13:35:24.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- 172.21.15.87:0/1234469958 wait complete. 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 Processor -- start 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.518+0000 7fd9a8448640 1 -- start start 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0101080 0x7fd9a0102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 0x7fd9a0103050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01110c0 con 0x7fd9a0103590 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0110f40 con 0x7fd9a0106150 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a8448640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0111240 con 0x7fd9a0101080 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54254/0 (socket says 172.21.15.87:54254) 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 -- 172.21.15.87:0/1763185448 learned_addr learned my addr 172.21.15.87:0/1763185448 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 0x7fd9a0103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0101080 0x7fd9a0102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 -- 172.21.15.87:0/1763185448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0101080 msgr2=0x7fd9a0102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0101080 0x7fd9a0102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 -- 172.21.15.87:0/1763185448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 msgr2=0x7fd9a0103050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 0x7fd9a0103050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 -- 172.21.15.87:0/1763185448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a01a7030 con 0x7fd9a0103590 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 0x7fd9a0103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9a69e5640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cad0 tx=0x7fd99800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.519+0000 7fd9977fe640 1 -- 172.21.15.87:0/1763185448 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99800ae20 con 0x7fd9a0103590 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.520+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a01a7320 con 0x7fd9a0103590 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.520+0000 7fd9977fe640 1 -- 172.21.15.87:0/1763185448 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd998005020 con 0x7fd9a0103590 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.520+0000 7fd9977fe640 1 -- 172.21.15.87:0/1763185448 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99800f6b0 con 0x7fd9a0103590 2024-04-22T13:35:24.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.520+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01a78c0 con 0x7fd9a0103590 2024-04-22T13:35:24.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9977fe640 1 -- 172.21.15.87:0/1763185448 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd9980051d0 con 0x7fd9a0103590 2024-04-22T13:35:24.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 Processor -- start 2024-04-22T13:35:24.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 -- start start 2024-04-22T13:35:24.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e3d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f825d991e0 con 0x55f825e34580 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f825d36390 con 0x55f825e34ea0 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9ab032740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f825d951a0 con 0x55f825e3da80 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54270/0 (socket says 172.21.15.87:54270) 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3689583662 learned_addr learned my addr 172.21.15.87:0/3689583662 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e3d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9977fe640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3689583662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 msgr2=0x55f825e3fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3689583662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 msgr2=0x55f825e3d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.523+0000 7fd9a69e5640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e3d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3689583662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f825b757d0 con 0x55f825e34580 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e3d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9977fe640 1 -- 172.21.15.87:0/1763185448 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd9980d06b0 con 0x7fd9a0103590 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd990013960 tx=0x7fd990009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82cc2eaecaeb1ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd994bfe640 1 -- 172.21.15.87:0/3689583662 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990004030 con 0x55f825e34580 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd9a69e5640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd9980ce470 tx=0x7fd998005890 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd994bfe640 1 -- 172.21.15.87:0/3689583662 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99000f940 con 0x55f825e34580 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.524+0000 7fd994bfe640 1 -- 172.21.15.87:0/3689583662 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9900039e0 con 0x55f825e34580 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- 172.21.15.87:0/3689583662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 msgr2=0x55f825e34960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd990013960 tx=0x7fd990009c00 comp rx=0 tx=0).stop 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- 172.21.15.87:0/3689583662 shutdown_connections 2024-04-22T13:35:24.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e3d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3689583662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e34960 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- 172.21.15.87:0/3689583662 >> 172.21.15.87:0/3689583662 conn(0x55f825e2ecb0 msgr2=0x55f825e2f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- 172.21.15.87:0/3689583662 shutdown_connections 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- 172.21.15.87:0/3689583662 wait complete. 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- start start 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 Processor -- start 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34ea0 0x55f825e3afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f825e40810 con 0x55f825e34580 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f825d522f0 con 0x55f825e3da80 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.525+0000 7fd9ab032740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f825e40990 con 0x55f825e34ea0 2024-04-22T13:35:24.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9ab032740 10 client.0 _ll_register_callbacks cb 0x55f825d5f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54284/0 (socket says 172.21.15.87:54284) 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3058499065 learned_addr learned my addr 172.21.15.87:0/3058499065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34ea0 0x55f825e3afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3058499065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34ea0 msgr2=0x55f825e3afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34ea0 0x55f825e3afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 msgr2=0x55f825e3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 -- 172.21.15.87:0/3058499065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f825aab3f0 con 0x55f825e34580 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd99000ff60 tx=0x7fd9900056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc17ba64033fdb08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990010900 con 0x55f825e34580 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd990010aa0 con 0x55f825e34580 2024-04-22T13:35:24.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99000f940 con 0x55f825e34580 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd990010ca0 con 0x55f825e34580 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9ab032740 10 client.16725 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f825a6b0f0 con 0x55f825e34580 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.526+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99000f940 con 0x55f825e34580 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd990034050 con 0x55f825e34580 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 10 client.16725 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 20 client.16725 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825e4d8f0 0x55f825e4fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 4 client.16725 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f825da6c00 con 0x55f825e4d8f0 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825e4d8f0 0x55f825e4fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd989ffb640 10 client.16725 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.527+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825e4d8f0 0x55f825e4fce0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d44ffd3c4dbcef5 server_cookie=a4fe66f5e03f6f6 in_seq=0 out_seq=0 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd989ffb640 1 -- 172.21.15.87:0/3058499065 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f825da6c00 con 0x55f825e4d8f0 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd989ffb640 10 client.16725 handle_command_reply: tid=0 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 client.16725 shutdown 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 msgr2=0x55f825e3aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd99000ff60 tx=0x7fd9900056d0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 shutdown_connections 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825e4d8f0 0x55f825e4fce0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34ea0 0x55f825e3afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd989ffb640 0 client.16725 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 --2- 172.21.15.87:0/3058499065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3aa80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 >> 172.21.15.87:0/3058499065 conn(0x55f825e2ecb0 msgr2=0x55f825e30a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 shutdown_connections 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 1 -- 172.21.15.87:0/3058499065 wait complete. 2024-04-22T13:35:24.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.528+0000 7fd9ab032740 20 client.16725 trim_cache size 0 max 16384 2024-04-22T13:35:24.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 Processor -- start 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 -- start start 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e2dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01b2530 con 0x55f825e34580 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01afe00 con 0x55f825e34ea0 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.545+0000 7fd9ab032740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01adc70 con 0x55f825e3da80 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54298/0 (socket says 172.21.15.87:54298) 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 -- 172.21.15.87:0/2209784857 learned_addr learned my addr 172.21.15.87:0/2209784857 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e2dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 -- 172.21.15.87:0/2209784857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 msgr2=0x55f825e3a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 -- 172.21.15.87:0/2209784857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 msgr2=0x55f825e2dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e2dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 -- 172.21.15.87:0/2209784857 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f825e34cd0 con 0x55f825e34580 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e2dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd99001f500 tx=0x7fd990009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee5749bb902ac29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd98bfff640 1 -- 172.21.15.87:0/2209784857 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99003a070 con 0x55f825e34580 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd98bfff640 1 -- 172.21.15.87:0/2209784857 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99000f790 con 0x55f825e34580 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.546+0000 7fd98bfff640 1 -- 172.21.15.87:0/2209784857 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99001fd10 con 0x55f825e34580 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- 172.21.15.87:0/2209784857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 msgr2=0x55f825e3d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd99001f500 tx=0x7fd990009b50 comp rx=0 tx=0).stop 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- 172.21.15.87:0/2209784857 shutdown_connections 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e3da80 0x55f825e3a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e34ea0 0x55f825e2dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2209784857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34580 0x55f825e3d1b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- 172.21.15.87:0/2209784857 >> 172.21.15.87:0/2209784857 conn(0x55f825e2ecb0 msgr2=0x55f825e2ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- 172.21.15.87:0/2209784857 shutdown_connections 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- 172.21.15.87:0/2209784857 wait complete. 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- start start 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 Processor -- start 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34580 0x55f825e460f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e37aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01ae2c0 con 0x55f825e34ea0 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01af730 con 0x55f825e3da80 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01ae990 con 0x55f825e34580 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9ab032740 10 client.0 _ll_register_callbacks cb 0x7fd99c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9a61e4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9a61e4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54300/0 (socket says 172.21.15.87:54300) 2024-04-22T13:35:24.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9a61e4640 1 -- 172.21.15.87:0/2789875916 learned_addr learned my addr 172.21.15.87:0/2789875916 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.547+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e37aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 -- 172.21.15.87:0/2789875916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34580 msgr2=0x55f825e460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34580 0x55f825e460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 -- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 msgr2=0x55f825e37aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e37aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 -- 172.21.15.87:0/2789875916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f825e382d0 con 0x55f825e34ea0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e37aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9a61e4640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3267c36dab38663 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c02c9f0 con 0x55f825e34ea0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd99c03a040 con 0x55f825e34ea0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99000f790 con 0x55f825e34ea0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd99c02cb90 con 0x55f825e34ea0 2024-04-22T13:35:24.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 10 client.16731 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f825eb3cc0 con 0x55f825e34ea0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99000f790 con 0x55f825e34ea0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd99c004a70 con 0x55f825e34ea0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 10 client.16731 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 20 client.16731 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825eb44f0 0x55f825eb6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 4 client.16731 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.548+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f825da0a70 con 0x55f825eb44f0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825eb44f0 0x55f825eb6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd989ffb640 10 client.16731 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9a59e3640 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825eb44f0 0x55f825eb6910 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ec2cf07146d6d1 server_cookie=ee684de4ca0679e1 in_seq=0 out_seq=0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd989ffb640 1 -- 172.21.15.87:0/2789875916 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f825da0a70 con 0x55f825eb44f0 2024-04-22T13:35:24.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd989ffb640 10 client.16731 handle_command_reply: tid=0 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 client.16731 shutdown 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 msgr2=0x55f825e37520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c004400 comp rx=0 tx=0).stop 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 shutdown_connections 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f825eb44f0 0x55f825eb6910 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f825e3da80 0x55f825e37aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f825e34ea0 0x55f825e37520 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 --2- 172.21.15.87:0/2789875916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f825e34580 0x55f825e460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 >> 172.21.15.87:0/2789875916 conn(0x55f825e2ecb0 msgr2=0x55f825e307c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd989ffb640 0 client.16731 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 shutdown_connections 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 1 -- 172.21.15.87:0/2789875916 wait complete. 2024-04-22T13:35:24.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.549+0000 7fd9ab032740 20 client.16731 trim_cache size 0 max 16384 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "6b48e44e-f626-45f3-baad-1a2b37809b4c": { 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "6b48e44e-f626-45f3-baad-1a2b37809b4c", 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:24.553 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:24.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 msgr2=0x7fd98c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd9980ce470 tx=0x7fd998005890 comp rx=0 tx=0).stop 2024-04-22T13:35:24.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 msgr2=0x7fd9a01a6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:24.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cad0 tx=0x7fd99800cf90 comp rx=0 tx=0).stop 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 shutdown_connections 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9a0103590 0x7fd9a01a6990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9a0106150 0x7fd9a0103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 --2- 172.21.15.87:0/1763185448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9a0101080 0x7fd9a0102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 >> 172.21.15.87:0/1763185448 conn(0x7fd9a0078000 msgr2=0x7fd9a0069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 shutdown_connections 2024-04-22T13:35:24.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:24.550+0000 7fd9a8448640 1 -- 172.21.15.87:0/1763185448 wait complete. 2024-04-22T13:35:24.556 INFO:tasks.cephfs.filesystem:scrub status for tag:6b48e44e-f626-45f3-baad-1a2b37809b4c - {'path': '/', 'tag': '6b48e44e-f626-45f3-baad-1a2b37809b4c', 'options': 'recursive,force'} 2024-04-22T13:35:24.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:24 smithi120 ceph-mon[31101]: daemon mds.f restarted 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: pgmap v109: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 954 MiB/s wr, 4.87k op/s 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6834/666537819,v1:172.21.15.132:6835/666537819] up:boot 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:35:25.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:25 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:25.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: pgmap v109: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 954 MiB/s wr, 4.87k op/s 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6834/666537819,v1:172.21.15.132:6835/666537819] up:boot 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:35:25.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:25 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:25.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: pgmap v109: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 954 MiB/s wr, 4.87k op/s 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6834/666537819,v1:172.21.15.132:6835/666537819] up:boot 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:35:25.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:25 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:35:26.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:26 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:35:26.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:26 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:35:26.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:26 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:35:27.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:27 smithi132 ceph-mon[31531]: pgmap v111: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 803 MiB/s wr, 2.44k op/s 2024-04-22T13:35:27.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:27 smithi087 ceph-mon[27999]: pgmap v111: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 803 MiB/s wr, 2.44k op/s 2024-04-22T13:35:27.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:27 smithi120 ceph-mon[31101]: pgmap v111: 97 pgs: 97 active+clean; 28 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 803 MiB/s wr, 2.44k op/s 2024-04-22T13:35:29.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:29 smithi132 ceph-mon[31531]: pgmap v112: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 602 MiB/s wr, 1.60k op/s 2024-04-22T13:35:29.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:29 smithi087 ceph-mon[27999]: pgmap v112: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 602 MiB/s wr, 1.60k op/s 2024-04-22T13:35:29.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:29 smithi120 ceph-mon[31101]: pgmap v112: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 602 MiB/s wr, 1.60k op/s 2024-04-22T13:35:31.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:31 smithi132 ceph-mon[31531]: pgmap v113: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:31.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:31 smithi087 ceph-mon[27999]: pgmap v113: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:31.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:31 smithi120 ceph-mon[31101]: pgmap v113: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:33.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:33 smithi087 ceph-mon[27999]: pgmap v114: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:33 smithi120 ceph-mon[31101]: pgmap v114: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:33.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:33 smithi132 ceph-mon[31531]: pgmap v114: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 454 MiB/s wr, 1.09k op/s 2024-04-22T13:35:35.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- 172.21.15.87:0/23978013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 msgr2=0x7f6f34069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- 172.21.15.87:0/23978013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f34069560 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009930 tx=0x7f6f1c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- 172.21.15.87:0/23978013 shutdown_connections 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- 172.21.15.87:0/23978013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f3410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- 172.21.15.87:0/23978013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f34069560 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- 172.21.15.87:0/23978013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f34108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- 172.21.15.87:0/23978013 >> 172.21.15.87:0/23978013 conn(0x7f6f34077e30 msgr2=0x7f6f34078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- 172.21.15.87:0/23978013 shutdown_connections 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- 172.21.15.87:0/23978013 wait complete. 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 Processor -- start 2024-04-22T13:35:35.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- start start 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f34198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f3419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34112140 con 0x7f6f340690e0 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34111fc0 con 0x7f6f34108030 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.363+0000 7f6f398f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f341122c0 con 0x7f6f34069aa0 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39480/0 (socket says 172.21.15.87:39480) 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 -- 172.21.15.87:0/940627476 learned_addr learned my addr 172.21.15.87:0/940627476 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f2bfff640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f34198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f337fe640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f3419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 -- 172.21.15.87:0/940627476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 msgr2=0x7f6f34198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f34198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 -- 172.21.15.87:0/940627476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 msgr2=0x7f6f3419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f3419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 -- 172.21.15.87:0/940627476 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3419d6b0 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f2bfff640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f34198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f337fe640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f3419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f32ffd640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400e9f0 tx=0x7f6f2400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400cd80 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f240108e0 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24018980 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3419d9a0 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.364+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f341a5230 con 0x7f6f340690e0 2024-04-22T13:35:35.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.365+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f34068de0 con 0x7f6f340690e0 2024-04-22T13:35:35.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.366+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6f24010430 con 0x7f6f340690e0 2024-04-22T13:35:35.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.367+0000 7f6f388f2640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 0x7f6f200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.367+0000 7f6f2bfff640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 0x7f6f200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.367+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f6f240d1000 con 0x7f6f340690e0 2024-04-22T13:35:35.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.368+0000 7f6f2bfff640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 0x7f6f200a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c006010 tx=0x7f6f1c009610 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.371+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f2409d710 con 0x7f6f340690e0 2024-04-22T13:35:35.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:35 smithi087 ceph-mon[27999]: pgmap v115: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 135 MiB/s wr, 302 op/s 2024-04-22T13:35:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.587+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f34199a10 con 0x7f6f340690e0 2024-04-22T13:35:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.588+0000 7f6f388f2640 1 -- 172.21.15.87:0/940627476 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f6f2409d010 con 0x7f6f340690e0 2024-04-22T13:35:35.584 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:35:35.584 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:35:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:35:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 msgr2=0x7f6f200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 0x7f6f200a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c006010 tx=0x7f6f1c009610 comp rx=0 tx=0).stop 2024-04-22T13:35:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 msgr2=0x7f6f341986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400e9f0 tx=0x7f6f2400eeb0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 shutdown_connections 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f200a3d20 0x7f6f200a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f34108030 0x7f6f3419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f34069aa0 0x7f6f34198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 --2- 172.21.15.87:0/940627476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f340690e0 0x7f6f341986c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 >> 172.21.15.87:0/940627476 conn(0x7f6f34077e30 msgr2=0x7f6f34100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 shutdown_connections 2024-04-22T13:35:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.590+0000 7f6f398f4640 1 -- 172.21.15.87:0/940627476 wait complete. 2024-04-22T13:35:35.597 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 34d09ee7 --release --await' 2024-04-22T13:35:35.598 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 34d09ee7 --release --await 2024-04-22T13:35:35.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:35 smithi120 ceph-mon[31101]: pgmap v115: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 135 MiB/s wr, 302 op/s 2024-04-22T13:35:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/4104576402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e0806a880 msgr2=0x7f2e0806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/4104576402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e0806a880 0x7f2e0806ac60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009900 tx=0x7f2df802bc30 comp rx=0 tx=0).stop 2024-04-22T13:35:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/4104576402 shutdown_connections 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/4104576402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e08107270 0x7f2e0810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/4104576402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0806b230 0x7f2e08106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/4104576402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e0806a880 0x7f2e0806ac60 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/4104576402 >> 172.21.15.87:0/4104576402 conn(0x7f2e08077eb0 msgr2=0x7f2e080782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/4104576402 shutdown_connections 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.707+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/4104576402 wait complete. 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 Processor -- start 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 -- start start 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e0806a880 0x7f2e08198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 0x7f2e0819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08112280 con 0x7f2e08107270 2024-04-22T13:35:35.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08112100 con 0x7f2e0819cfe0 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e0d8e0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08112400 con 0x7f2e0806a880 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39498/0 (socket says 172.21.15.87:39498) 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 -- 172.21.15.87:0/2949054252 learned_addr learned my addr 172.21.15.87:0/2949054252 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e0806a880 0x7f2e08198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 -- 172.21.15.87:0/2949054252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e0806a880 msgr2=0x7f2e08198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e077fe640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 0x7f2e0819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e0806a880 0x7f2e08198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 -- 172.21.15.87:0/2949054252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 msgr2=0x7f2e0819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 0x7f2e0819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2dfffff640 1 -- 172.21.15.87:0/2949054252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0819dad0 con 0x7f2e08107270 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.708+0000 7f2e077fe640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 0x7f2e0819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2dfffff640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2df000c970 tx=0x7f2df000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2e0c8de640 1 -- 172.21.15.87:0/2949054252 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0014070 con 0x7f2e08107270 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2e0c8de640 1 -- 172.21.15.87:0/2949054252 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df0004440 con 0x7f2e08107270 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0819ddc0 con 0x7f2e08107270 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2e0c8de640 1 -- 172.21.15.87:0/2949054252 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0003e50 con 0x7f2e08107270 2024-04-22T13:35:35.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.709+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e081a55b0 con 0x7f2e08107270 2024-04-22T13:35:35.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:35 smithi132 ceph-mon[31531]: pgmap v115: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 135 MiB/s wr, 302 op/s 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.711+0000 7f2e0c8de640 1 -- 172.21.15.87:0/2949054252 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2df0017070 con 0x7f2e08107270 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.711+0000 7f2e105ad740 1 Processor -- start 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.711+0000 7f2e105ad740 1 -- start start 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.711+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e105ad740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56533d0fa500 con 0x56533d195310 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e105ad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56533d0a06a0 con 0x56533d19e810 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e105ad740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56533d0f64c0 con 0x56533d195c30 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e0c8de640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 0x7f2df40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39506/0 (socket says 172.21.15.87:39506) 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 -- 172.21.15.87:0/496736223 learned_addr learned my addr 172.21.15.87:0/496736223 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 0x7f2df40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e0c8de640 1 -- 172.21.15.87:0/2949054252 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2df00d0430 con 0x7f2e08107270 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2dfffff640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 -- 172.21.15.87:0/496736223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 msgr2=0x56533d19e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 -- 172.21.15.87:0/496736223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 msgr2=0x56533d1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 -- 172.21.15.87:0/496736223 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56533d0bf5e0 con 0x56533d195310 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2dfffff640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2e00010a30 tx=0x7f2e00010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd82ee4cc803f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2dfdffb640 1 -- 172.21.15.87:0/496736223 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00020070 con 0x56533d195310 2024-04-22T13:35:35.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2dfdffb640 1 -- 172.21.15.87:0/496736223 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0001bbd0 con 0x56533d195310 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2dfdffb640 1 -- 172.21.15.87:0/496736223 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0000e350 con 0x56533d195310 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.712+0000 7f2e077fe640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 0x7f2df40a6380 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2e0001d000 tx=0x7f2e00019000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- 172.21.15.87:0/496736223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 msgr2=0x56533d1956f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2e00010a30 tx=0x7f2e00010ef0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- 172.21.15.87:0/496736223 shutdown_connections 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 --2- 172.21.15.87:0/496736223 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1956f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- 172.21.15.87:0/496736223 >> 172.21.15.87:0/496736223 conn(0x56533d18fa40 msgr2=0x56533d18fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- 172.21.15.87:0/496736223 shutdown_connections 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- 172.21.15.87:0/496736223 wait complete. 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 -- start start 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.713+0000 7f2e105ad740 1 Processor -- start 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195310 0x56533d19e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56533d1a1600 con 0x56533d19e810 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56533d0b3a60 con 0x56533d195310 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56533d1a1780 con 0x56533d195c30 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 10 client.0 _ll_register_callbacks cb 0x56533ce0c110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39510/0 (socket says 172.21.15.87:39510) 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 -- 172.21.15.87:0/579403240 learned_addr learned my addr 172.21.15.87:0/579403240 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2dfffff640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195310 0x56533d19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 -- 172.21.15.87:0/579403240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 msgr2=0x56533d19b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 -- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195310 msgr2=0x56533d19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195310 0x56533d19e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 -- 172.21.15.87:0/579403240 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56533ce0c920 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e077fe640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2e00004a80 tx=0x7f2e0000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9208db26c0fd10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00032060 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2e0000d5b0 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.714+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e0001bbd0 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e00020040 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 10 client.16749 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56533ced8d30 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0001bbd0 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2e00035030 con 0x56533d19e810 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 10 client.16749 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 10 client.16749 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 20 client.16749 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:35.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d1ae710 0x56533d1b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 4 client.16749 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56533d0cda10 con 0x56533d1ae710 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.715+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d1ae710 0x56533d1b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.716+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d1ae710 0x56533d1b0b00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58885cfedf02c01f server_cookie=9390b5aeb4cf747f in_seq=0 out_seq=0 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.716+0000 7f2de6ffd640 10 client.16749 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.716+0000 7f2de6ffd640 1 -- 172.21.15.87:0/579403240 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56533d0cda10 con 0x56533d1ae710 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.716+0000 7f2de6ffd640 10 client.16749 handle_command_reply: tid=0 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 client.16749 shutdown 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 msgr2=0x56533d19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2e00004a80 tx=0x7f2e0000d9e0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 shutdown_connections 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d1ae710 0x56533d1b0b00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d19e810 0x56533d19bd10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195c30 0x56533d19b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 --2- 172.21.15.87:0/579403240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195310 0x56533d19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 >> 172.21.15.87:0/579403240 conn(0x56533d18fa40 msgr2=0x56533d191790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2de6ffd640 0 client.16749 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:35.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 shutdown_connections 2024-04-22T13:35:35.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 1 -- 172.21.15.87:0/579403240 wait complete. 2024-04-22T13:35:35.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.717+0000 7f2e105ad740 20 client.16749 trim_cache size 0 max 16384 2024-04-22T13:35:35.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 Processor -- start 2024-04-22T13:35:35.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 -- start start 2024-04-22T13:35:35.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195c30 0x56533d1b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d19e810 0x56533d1b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b0320 con 0x56533d195310 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081adbf0 con 0x56533d195c30 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e105ad740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081aba40 con 0x56533d19e810 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39522/0 (socket says 172.21.15.87:39522) 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 -- 172.21.15.87:0/39034770 learned_addr learned my addr 172.21.15.87:0/39034770 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e077fe640 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d19e810 0x56533d1b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 -- 172.21.15.87:0/39034770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d19e810 msgr2=0x56533d1b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d19e810 0x56533d1b68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 -- 172.21.15.87:0/39034770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195c30 msgr2=0x56533d1b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195c30 0x56533d1b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 -- 172.21.15.87:0/39034770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56533d18f280 con 0x56533d195310 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009900 tx=0x7f2df8005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72414d04718335a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2dfd7fa640 1 -- 172.21.15.87:0/39034770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df8004360 con 0x56533d195310 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2dfd7fa640 1 -- 172.21.15.87:0/39034770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df802cc90 con 0x56533d195310 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.734+0000 7f2dfd7fa640 1 -- 172.21.15.87:0/39034770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df80046c0 con 0x56533d195310 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- 172.21.15.87:0/39034770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 msgr2=0x56533d1aa4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009900 tx=0x7f2df8005a70 comp rx=0 tx=0).stop 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- 172.21.15.87:0/39034770 shutdown_connections 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d19e810 0x56533d1b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d195c30 0x56533d1b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 --2- 172.21.15.87:0/39034770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195310 0x56533d1aa4c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- 172.21.15.87:0/39034770 >> 172.21.15.87:0/39034770 conn(0x56533d18fa40 msgr2=0x56533d19a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- 172.21.15.87:0/39034770 shutdown_connections 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- 172.21.15.87:0/39034770 wait complete. 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 -- start start 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.735+0000 7f2e105ad740 1 Processor -- start 2024-04-22T13:35:35.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195310 0x56533d19f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081ac0b0 con 0x56533d195c30 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081ad520 con 0x56533d19e810 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081ac780 con 0x56533d195310 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 10 client.0 _ll_register_callbacks cb 0x7f2e0001e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39538/0 (socket says 172.21.15.87:39538) 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 -- 172.21.15.87:0/300196373 learned_addr learned my addr 172.21.15.87:0/300196373 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195310 0x56533d19f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e077fe640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 -- 172.21.15.87:0/300196373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195310 msgr2=0x56533d19f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195310 0x56533d19f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 -- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 msgr2=0x56533d1a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 -- 172.21.15.87:0/300196373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56533d1a5080 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e077fe640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2dfffff640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2df009c370 tx=0x7f2df009cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e014deb7f403ff87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df00a0790 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2df00cde10 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2df802cc90 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2df00ce010 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 10 client.16755 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.736+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56533d2151a0 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df802cc90 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2df00b5bf0 con 0x56533d195c30 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 10 client.16755 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 10 client.16755 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 20 client.16755 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d2158a0 0x56533d217cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 4 client.16755 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "34d09ee7", "release": true, "await": true}] 2024-04-22T13:35:35.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "34d09ee7", "release": true, "await": true}) -- 0x56533d107f20 con 0x56533d2158a0 2024-04-22T13:35:35.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d2158a0 0x56533d217cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:35.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2e06ffd640 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d2158a0 0x56533d217cc0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=578b5b1c1de90a7d server_cookie=51d7babb0ceb059a in_seq=0 out_seq=0 2024-04-22T13:35:35.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.737+0000 7f2de6ffd640 10 client.16755 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 16, 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:35:35.740 INFO:teuthology.orchestra.run.smithi087.stdout: "34d09ee7": { 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 16, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 71.2, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:35:35.741 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:35:35.742 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.743+0000 7f2de6ffd640 1 -- 172.21.15.87:0/300196373 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56533d107f20 con 0x56533d2158a0 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.743+0000 7f2de6ffd640 10 client.16755 handle_command_reply: tid=0 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.743+0000 7f2e105ad740 1 client.16755 shutdown 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.743+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 msgr2=0x56533d19fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.743+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2df009c370 tx=0x7f2df009cba0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 shutdown_connections 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56533d2158a0 0x56533d217cc0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56533d19e810 0x56533d1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56533d195c30 0x56533d19fe50 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 --2- 172.21.15.87:0/300196373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56533d195310 0x56533d19f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 >> 172.21.15.87:0/300196373 conn(0x56533d18fa40 msgr2=0x56533d18e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2de6ffd640 0 client.16755 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 shutdown_connections 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 1 -- 172.21.15.87:0/300196373 wait complete. 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.744+0000 7f2e105ad740 20 client.16755 trim_cache size 0 max 16384 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 msgr2=0x7f2df40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 0x7f2df40a6380 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2e0001d000 tx=0x7f2e00019000 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 msgr2=0x7f2e08198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2df000c970 tx=0x7f2df000ce30 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 shutdown_connections 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2df40a3ec0 0x7f2df40a6380 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e0819cfe0 0x7f2e0819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2e08107270 0x7f2e08198c60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 --2- 172.21.15.87:0/2949054252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e0806a880 0x7f2e08198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.745+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 >> 172.21.15.87:0/2949054252 conn(0x7f2e08077eb0 msgr2=0x7f2e080fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.746+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 shutdown_connections 2024-04-22T13:35:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:35.746+0000 7f2e0d8e0640 1 -- 172.21.15.87:0/2949054252 wait complete. 2024-04-22T13:35:35.781 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 34d09ee7, seconds elapsed: 15.2 2024-04-22T13:35:36.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:36 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/940627476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:36.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:36 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/940627476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:36.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:36 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/940627476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.401+0000 7fe548632640 1 -- 172.21.15.87:0/1417894325 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe54011b730 msgr2=0x7fe54011db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.401+0000 7fe548632640 1 --2- 172.21.15.87:0/1417894325 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe54011b730 0x7fe54011db20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009930 tx=0x7fe53c02bcd0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.402+0000 7fe548632640 1 -- 172.21.15.87:0/1417894325 shutdown_connections 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.402+0000 7fe548632640 1 --2- 172.21.15.87:0/1417894325 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe54011b730 0x7fe54011db20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.402+0000 7fe548632640 1 --2- 172.21.15.87:0/1417894325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110b20 0x7fe540109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.402+0000 7fe548632640 1 --2- 172.21.15.87:0/1417894325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe540110740 0x7fe540108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.402+0000 7fe548632640 1 -- 172.21.15.87:0/1417894325 >> 172.21.15.87:0/1417894325 conn(0x7fe54006d020 msgr2=0x7fe54006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.408+0000 7fe548632640 1 -- 172.21.15.87:0/1417894325 shutdown_connections 2024-04-22T13:35:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.408+0000 7fe548632640 1 -- 172.21.15.87:0/1417894325 wait complete. 2024-04-22T13:35:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 Processor -- start 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 -- start start 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110740 0x7fe5401a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe5401abe60 0x7fe5401ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe540121080 con 0x7fe5401abe60 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe540120f00 con 0x7fe54011b730 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.409+0000 7fe548632640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe540121200 con 0x7fe540110740 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.410+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.410+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:53824/0 (socket says 172.21.15.87:53824) 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.410+0000 7fe545bcd640 1 -- 172.21.15.87:0/1506802629 learned_addr learned my addr 172.21.15.87:0/1506802629 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.410+0000 7fe5463ce640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110740 0x7fe5401a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.411+0000 7fe545bcd640 1 -- 172.21.15.87:0/1506802629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110740 msgr2=0x7fe5401a75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.411+0000 7fe545bcd640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110740 0x7fe5401a75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.411+0000 7fe545bcd640 1 -- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe5401abe60 msgr2=0x7fe5401ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.411+0000 7fe545bcd640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe5401abe60 0x7fe5401ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.411+0000 7fe545bcd640 1 -- 172.21.15.87:0/1506802629 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5401ac950 con 0x7fe54011b730 2024-04-22T13:35:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.412+0000 7fe545bcd640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b3f0 tx=0x7fe53000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.416+0000 7fe5377fe640 1 -- 172.21.15.87:0/1506802629 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530037020 con 0x7fe54011b730 2024-04-22T13:35:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.416+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5401acbe0 con 0x7fe54011b730 2024-04-22T13:35:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.416+0000 7fe5377fe640 1 -- 172.21.15.87:0/1506802629 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe530033af0 con 0x7fe54011b730 2024-04-22T13:35:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.416+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe540074470 con 0x7fe54011b730 2024-04-22T13:35:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.418+0000 7fe5377fe640 1 -- 172.21.15.87:0/1506802629 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530032c40 con 0x7fe54011b730 2024-04-22T13:35:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.421+0000 7fe54b21b740 1 Processor -- start 2024-04-22T13:35:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.422+0000 7fe54b21b740 1 -- start start 2024-04-22T13:35:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.423+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ec620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.423+0000 7fe546bcf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.424+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.424+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.424+0000 7fe54b21b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559a4e9518d0 con 0x559a4e9ed8d0 2024-04-22T13:35:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.424+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.426+0000 7fe5377fe640 1 -- 172.21.15.87:0/1506802629 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe53003a020 con 0x7fe54011b730 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.426+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:53832/0 (socket says 172.21.15.87:53832) 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.426+0000 7fe5463ce640 1 -- 172.21.15.87:0/1959093583 learned_addr learned my addr 172.21.15.87:0/1959093583 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.426+0000 7fe546bcf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39578/0 (socket says 172.21.15.87:39578) 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.427+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559a4e8eee80 con 0x559a4e9ecb60 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.427+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559a4e94d890 con 0x559a4e9f60e0 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.426+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.428+0000 7fe5377fe640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 0x7fe52c0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5463ce640 1 -- 172.21.15.87:0/1959093583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 msgr2=0x559a4e9f84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5463ce640 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5463ce640 1 -- 172.21.15.87:0/1959093583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 msgr2=0x559a4e9ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5463ce640 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5463ce640 1 -- 172.21.15.87:0/1959093583 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a4e910320 con 0x559a4e9ecb60 2024-04-22T13:35:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.429+0000 7fe5377fe640 1 -- 172.21.15.87:0/1506802629 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe5300f19e0 con 0x7fe54011b730 2024-04-22T13:35:37.426 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.431+0000 7fe5463ce640 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe538009f50 tx=0x7fe538010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe5bca0b7ef020de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.436+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 0x7fe52c0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.441+0000 7fe534bfe640 1 -- 172.21.15.87:0/1959093583 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538016070 con 0x559a4e9ecb60 2024-04-22T13:35:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.441+0000 7fe534bfe640 1 -- 172.21.15.87:0/1959093583 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe53801bb10 con 0x559a4e9ecb60 2024-04-22T13:35:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.441+0000 7fe534bfe640 1 -- 172.21.15.87:0/1959093583 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538012c10 con 0x559a4e9ecb60 2024-04-22T13:35:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.442+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 msgr2=0x559a4e9ecfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.442+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe538009f50 tx=0x7fe538010800 comp rx=0 tx=0).stop 2024-04-22T13:35:37.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.442+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 shutdown_connections 2024-04-22T13:35:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.443+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.443+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ecb60 0x559a4e9ecfe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.443+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1959093583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.443+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 >> 172.21.15.87:0/1959093583 conn(0x559a4e9b3c90 msgr2=0x559a4e9e7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.451+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 shutdown_connections 2024-04-22T13:35:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.451+0000 7fe54b21b740 1 -- 172.21.15.87:0/1959093583 wait complete. 2024-04-22T13:35:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.451+0000 7fe54b21b740 1 -- start start 2024-04-22T13:35:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.452+0000 7fe54b21b740 1 Processor -- start 2024-04-22T13:35:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4e9f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9f60e0 0x559a4e9f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559a4e9f8e10 con 0x559a4e9ecb60 2024-04-22T13:35:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559a4e8d9eb0 con 0x559a4e9f60e0 2024-04-22T13:35:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559a4e9f8f90 con 0x559a4e9ed8d0 2024-04-22T13:35:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.453+0000 7fe54b21b740 10 client.0 _ll_register_callbacks cb 0x559a4e906b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.454+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.454+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:58094/0 (socket says 172.21.15.87:58094) 2024-04-22T13:35:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.454+0000 7fe545bcd640 1 -- 172.21.15.87:0/890051578 learned_addr learned my addr 172.21.15.87:0/890051578 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.455+0000 7fe5463ce640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4e9f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.455+0000 7fe546bcf640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9f60e0 0x559a4e9f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.462+0000 7fe545bcd640 1 -- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9f60e0 msgr2=0x559a4e9f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.468+0000 7fe545bcd640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9f60e0 0x559a4e9f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.468+0000 7fe545bcd640 1 -- 172.21.15.87:0/890051578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 msgr2=0x559a4e9f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.468+0000 7fe545bcd640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4e9f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.468+0000 7fe545bcd640 1 -- 172.21.15.87:0/890051578 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a4e664210 con 0x559a4e9ed8d0 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.468+0000 7fe5463ce640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4e9f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.469+0000 7fe545bcd640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe530006fd0 tx=0x7fe5300bd700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=710d14f1df81a029 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.469+0000 7fe529ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.469+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5300bdb00 con 0x559a4e9ed8d0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.469+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe5300ba170 con 0x559a4e9ed8d0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.469+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe53801bb10 con 0x559a4e9ed8d0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.470+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 0x7fe52c0a6430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009850 tx=0x7fe53c02e040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.470+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe5300c1eb0 con 0x559a4e9ed8d0 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.470+0000 7fe54b21b740 10 client.25945 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.470+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559a4e730620 con 0x559a4e9ed8d0 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe53801bb10 con 0x559a4e9ed8d0 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe530032070 con 0x559a4e9ed8d0 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 10 client.25945 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 10 client.25945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 20 client.25945 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea06170 0x559a4ea08580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 4 client.25945 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.471+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a4e965880 con 0x559a4ea06170 2024-04-22T13:35:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.472+0000 7fe5463ce640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea06170 0x559a4ea08580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.474+0000 7fe5463ce640 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea06170 0x559a4ea08580 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31e00a8a10286059 server_cookie=ac46832f6185f822 in_seq=0 out_seq=0 2024-04-22T13:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.475+0000 7fe529ffb640 10 client.25945 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.478+0000 7fe529ffb640 1 -- 172.21.15.87:0/890051578 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a4e965880 con 0x559a4ea06170 2024-04-22T13:35:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.478+0000 7fe529ffb640 10 client.25945 handle_command_reply: tid=0 2024-04-22T13:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 client.25945 shutdown 2024-04-22T13:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 msgr2=0x559a4e9f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe530006fd0 tx=0x7fe5300bd700 comp rx=0 tx=0).stop 2024-04-22T13:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 shutdown_connections 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea06170 0x559a4ea08580 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9f60e0 0x559a4e9f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f2db0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 --2- 172.21.15.87:0/890051578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4e9f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 >> 172.21.15.87:0/890051578 conn(0x559a4e9b3c90 msgr2=0x559a4e9e95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.479+0000 7fe529ffb640 0 client.25945 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.480+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 shutdown_connections 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.480+0000 7fe54b21b740 1 -- 172.21.15.87:0/890051578 wait complete. 2024-04-22T13:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.481+0000 7fe54b21b740 20 client.25945 trim_cache size 0 max 16384 2024-04-22T13:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.517+0000 7fe54b21b740 1 Processor -- start 2024-04-22T13:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.517+0000 7fe54b21b740 1 -- start start 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.517+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4ea01d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.518+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.518+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.518+0000 7fe54b21b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe524010df0 con 0x559a4e9ecb60 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.518+0000 7fe54b21b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe52400e6c0 con 0x559a4e9ed8d0 2024-04-22T13:35:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.518+0000 7fe54b21b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe52400c510 con 0x559a4e9f60e0 2024-04-22T13:35:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.519+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4ea01d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.519+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4ea01d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39594/0 (socket says 172.21.15.87:39594) 2024-04-22T13:35:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.519+0000 7fe5463ce640 1 -- 172.21.15.87:0/1736958932 learned_addr learned my addr 172.21.15.87:0/1736958932 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.519+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.519+0000 7fe545bcd640 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.520+0000 7fe546bcf640 1 -- 172.21.15.87:0/1736958932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 msgr2=0x559a4e9f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.520+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.520+0000 7fe546bcf640 1 -- 172.21.15.87:0/1736958932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 msgr2=0x559a4ea01d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.520+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4ea01d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.520+0000 7fe546bcf640 1 -- 172.21.15.87:0/1736958932 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a4e9e2f30 con 0x559a4e9f60e0 2024-04-22T13:35:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.521+0000 7fe546bcf640 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f5a20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009230 tx=0x7fe53c045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f002832da9ddfbfa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.522+0000 7fe52bfff640 1 -- 172.21.15.87:0/1736958932 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c050070 con 0x559a4e9f60e0 2024-04-22T13:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.522+0000 7fe52bfff640 1 -- 172.21.15.87:0/1736958932 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe53c005500 con 0x559a4e9f60e0 2024-04-22T13:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.523+0000 7fe54b21b740 1 -- 172.21.15.87:0/1736958932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 msgr2=0x559a4e9f5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.523+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f5a20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009230 tx=0x7fe53c045000 comp rx=0 tx=0).stop 2024-04-22T13:35:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.536+0000 7fe54b21b740 1 -- 172.21.15.87:0/1736958932 shutdown_connections 2024-04-22T13:35:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.536+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9f60e0 0x559a4e9f5a20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.536+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.536+0000 7fe54b21b740 1 --2- 172.21.15.87:0/1736958932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9ecb60 0x559a4ea01d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.536+0000 7fe54b21b740 1 -- 172.21.15.87:0/1736958932 >> 172.21.15.87:0/1736958932 conn(0x559a4e9b3c90 msgr2=0x559a4e9e7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.539+0000 7fe54b21b740 1 -- 172.21.15.87:0/1736958932 shutdown_connections 2024-04-22T13:35:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.539+0000 7fe54b21b740 1 -- 172.21.15.87:0/1736958932 wait complete. 2024-04-22T13:35:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.539+0000 7fe54b21b740 1 -- start start 2024-04-22T13:35:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.539+0000 7fe54b21b740 1 Processor -- start 2024-04-22T13:35:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 0x559a4e9efb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe52400cb80 con 0x559a4e9f60e0 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe52400dff0 con 0x559a4e9ed8d0 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe52400d250 con 0x559a4e9ecb60 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.540+0000 7fe54b21b740 10 client.0 _ll_register_callbacks cb 0x7fe5300f7240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.541+0000 7fe545bcd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.541+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.541+0000 7fe5463ce640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:58114/0 (socket says 172.21.15.87:58114) 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.541+0000 7fe5463ce640 1 -- 172.21.15.87:0/256239848 learned_addr learned my addr 172.21.15.87:0/256239848 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.541+0000 7fe546bcf640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 0x559a4e9efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe5463ce640 1 -- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 msgr2=0x559a4e9ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe5463ce640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe5463ce640 1 -- 172.21.15.87:0/256239848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 msgr2=0x559a4e9efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe5463ce640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 0x559a4e9efb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe5463ce640 1 -- 172.21.15.87:0/256239848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a4e9f8840 con 0x559a4e9ecb60 2024-04-22T13:35:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.542+0000 7fe546bcf640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 0x559a4e9efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.543+0000 7fe5463ce640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe538009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512ba2bdcae59ec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.546+0000 7fe529ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.546+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538016040 con 0x559a4e9ecb60 2024-04-22T13:35:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.546+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe538004be0 con 0x559a4e9ecb60 2024-04-22T13:35:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.546+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe53c005500 con 0x559a4e9ecb60 2024-04-22T13:35:37.543 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.547+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe538004de0 con 0x559a4e9ecb60 2024-04-22T13:35:37.543 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.548+0000 7fe54b21b740 10 client.25951 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:37.543 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.548+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559a4ea7d2c0 con 0x559a4e9ecb60 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.548+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe53c005500 con 0x559a4e9ecb60 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe53800f040 con 0x559a4e9ecb60 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 10 client.25951 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 10 client.25951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 20 client.25951 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea7d660 0x559a4ea7fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 4 client.25951 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:35:37.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.549+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a4e8f13d0 con 0x559a4ea7d660 2024-04-22T13:35:37.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.550+0000 7fe545bcd640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea7d660 0x559a4ea7fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:37.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.553+0000 7fe529ffb640 10 client.25951 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:37.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:37.553+0000 7fe545bcd640 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea7d660 0x559a4ea7fa80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f2cf763e59906b8 server_cookie=83e8687d0ea87dfb in_seq=0 out_seq=0 2024-04-22T13:35:37.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:37 smithi120 ceph-mon[31101]: pgmap v116: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 113 MiB/s wr, 253 op/s 2024-04-22T13:35:37.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:37 smithi132 ceph-mon[31531]: pgmap v116: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 113 MiB/s wr, 253 op/s 2024-04-22T13:35:37.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:37 smithi087 ceph-mon[27999]: pgmap v116: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 113 MiB/s wr, 253 op/s 2024-04-22T13:35:39.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:39 smithi120 ceph-mon[31101]: pgmap v117: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 185 MiB/s wr, 704 op/s 2024-04-22T13:35:39.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:39 smithi132 ceph-mon[31531]: pgmap v117: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 185 MiB/s wr, 704 op/s 2024-04-22T13:35:39.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:39 smithi087 ceph-mon[27999]: pgmap v117: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 185 MiB/s wr, 704 op/s 2024-04-22T13:35:41.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:41 smithi120 ceph-mon[31101]: pgmap v118: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 245 MiB/s wr, 1.09k op/s 2024-04-22T13:35:41.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:41 smithi132 ceph-mon[31531]: pgmap v118: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 245 MiB/s wr, 1.09k op/s 2024-04-22T13:35:41.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:41 smithi087 ceph-mon[27999]: pgmap v118: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 245 MiB/s wr, 1.09k op/s 2024-04-22T13:35:43.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:43 smithi120 ceph-mon[31101]: pgmap v119: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 237 MiB/s wr, 1.07k op/s 2024-04-22T13:35:43.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:43 smithi132 ceph-mon[31531]: pgmap v119: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 237 MiB/s wr, 1.07k op/s 2024-04-22T13:35:43.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:43 smithi087 ceph-mon[27999]: pgmap v119: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 237 MiB/s wr, 1.07k op/s 2024-04-22T13:35:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.110+0000 7fe529ffb640 1 -- 172.21.15.87:0/256239848 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a4e8f13d0 con 0x559a4ea7d660 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.110+0000 7fe529ffb640 10 client.25951 handle_command_reply: tid=0 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.110+0000 7fe54b21b740 1 client.25951 shutdown 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 msgr2=0x559a4e9fe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe538009a00 comp rx=0 tx=0).stop 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 shutdown_connections 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559a4ea7d660 0x559a4ea7fa80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559a4e9f60e0 0x559a4e9efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a4e9ed8d0 0x559a4e9ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe529ffb640 0 client.25951 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 --2- 172.21.15.87:0/256239848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a4e9ecb60 0x559a4e9fe440 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.111+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 >> 172.21.15.87:0/256239848 conn(0x559a4e9b3c90 msgr2=0x559a4e9e6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.112+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 shutdown_connections 2024-04-22T13:35:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.112+0000 7fe54b21b740 1 -- 172.21.15.87:0/256239848 wait complete. 2024-04-22T13:35:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.112+0000 7fe54b21b740 20 client.25951 trim_cache size 0 max 16384 2024-04-22T13:35:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:35:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:35:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:35:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.113+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 msgr2=0x7fe52c0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.113+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 0x7fe52c0a6430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009850 tx=0x7fe53c02e040 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.113+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 msgr2=0x7fe5401a7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.113+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b3f0 tx=0x7fe53000b8b0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 shutdown_connections 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe52c0a3f70 0x7fe52c0a6430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe5401abe60 0x7fe5401ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe54011b730 0x7fe5401a7ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 --2- 172.21.15.87:0/1506802629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe540110740 0x7fe5401a75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 >> 172.21.15.87:0/1506802629 conn(0x7fe54006d020 msgr2=0x7fe54010e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 shutdown_connections 2024-04-22T13:35:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:45.114+0000 7fe548632640 1 -- 172.21.15.87:0/1506802629 wait complete. 2024-04-22T13:35:45.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:45 smithi120 ceph-mon[31101]: pgmap v120: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 541 MiB/s wr, 2.07k op/s 2024-04-22T13:35:45.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:45 smithi132 ceph-mon[31531]: pgmap v120: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 541 MiB/s wr, 2.07k op/s 2024-04-22T13:35:45.741 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:45 smithi087 ceph-mon[27999]: pgmap v120: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 541 MiB/s wr, 2.07k op/s 2024-04-22T13:35:47.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:47 smithi120 ceph-mon[31101]: pgmap v121: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 560 MiB/s wr, 2.10k op/s 2024-04-22T13:35:47.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:47 smithi132 ceph-mon[31531]: pgmap v121: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 560 MiB/s wr, 2.10k op/s 2024-04-22T13:35:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:47 smithi087 ceph-mon[27999]: pgmap v121: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 560 MiB/s wr, 2.10k op/s 2024-04-22T13:35:49.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:49 smithi132 ceph-mon[31531]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 790 MiB/s wr, 2.49k op/s 2024-04-22T13:35:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:49 smithi087 ceph-mon[27999]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 790 MiB/s wr, 2.49k op/s 2024-04-22T13:35:49.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:49 smithi120 ceph-mon[31101]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 790 MiB/s wr, 2.49k op/s 2024-04-22T13:35:51.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:51 smithi132 ceph-mon[31531]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 817 MiB/s wr, 2.29k op/s 2024-04-22T13:35:51.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:51 smithi087 ceph-mon[27999]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 817 MiB/s wr, 2.29k op/s 2024-04-22T13:35:51.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:51 smithi120 ceph-mon[31101]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 817 MiB/s wr, 2.29k op/s 2024-04-22T13:35:53.721 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:53 smithi120 ceph-mon[31101]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 722 MiB/s wr, 1.88k op/s 2024-04-22T13:35:53.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:53 smithi087 ceph-mon[27999]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 722 MiB/s wr, 1.88k op/s 2024-04-22T13:35:53.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:53 smithi132 ceph-mon[31531]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 722 MiB/s wr, 1.88k op/s 2024-04-22T13:35:54.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:35:54.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:35:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.935+0000 7ff27359e640 1 -- 172.21.15.87:0/2544778261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 msgr2=0x7ff2741093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:54.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.935+0000 7ff27359e640 1 --2- 172.21.15.87:0/2544778261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 0x7ff2741093b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff268009860 tx=0x7ff26802bc30 comp rx=0 tx=0).stop 2024-04-22T13:35:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.936+0000 7ff27359e640 1 -- 172.21.15.87:0/2544778261 shutdown_connections 2024-04-22T13:35:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.936+0000 7ff27359e640 1 --2- 172.21.15.87:0/2544778261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff27411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.936+0000 7ff27359e640 1 --2- 172.21.15.87:0/2544778261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 0x7ff2741093b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.936+0000 7ff27359e640 1 --2- 172.21.15.87:0/2544778261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2741105f0 0x7ff2741109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.936+0000 7ff27359e640 1 -- 172.21.15.87:0/2544778261 >> 172.21.15.87:0/2544778261 conn(0x7ff27406ced0 msgr2=0x7ff27406d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:54.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.943+0000 7ff27359e640 1 -- 172.21.15.87:0/2544778261 shutdown_connections 2024-04-22T13:35:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.944+0000 7ff27359e640 1 -- 172.21.15.87:0/2544778261 wait complete. 2024-04-22T13:35:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.947+0000 7ff27359e640 1 Processor -- start 2024-04-22T13:35:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.951+0000 7ff27359e640 1 -- start start 2024-04-22T13:35:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.952+0000 7ff27359e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 0x7ff2741a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.952+0000 7ff27359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.952+0000 7ff27359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2741105f0 0x7ff2741abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.952+0000 7ff27359e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff274120ee0 con 0x7ff2741105f0 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.953+0000 7ff27359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff274120d60 con 0x7ff2741099d0 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.953+0000 7ff27359e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff274121060 con 0x7ff274108f50 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.953+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.953+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37368/0 (socket says 172.21.15.87:37368) 2024-04-22T13:35:54.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.953+0000 7ff271d9b640 1 -- 172.21.15.87:0/1183275079 learned_addr learned my addr 172.21.15.87:0/1183275079 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:54.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.954+0000 7ff271d9b640 1 -- 172.21.15.87:0/1183275079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 msgr2=0x7ff2741a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:54.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.954+0000 7ff271d9b640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 0x7ff2741a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.954+0000 7ff271d9b640 1 -- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2741105f0 msgr2=0x7ff2741abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:54.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.954+0000 7ff271d9b640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2741105f0 0x7ff2741abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.954+0000 7ff271d9b640 1 -- 172.21.15.87:0/1183275079 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2741ac3d0 con 0x7ff2741099d0 2024-04-22T13:35:54.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.955+0000 7ff271d9b640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff268009460 tx=0x7ff2680349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:54.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.959+0000 7ff2637fe640 1 -- 172.21.15.87:0/1183275079 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268039040 con 0x7ff2741099d0 2024-04-22T13:35:54.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.959+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2741ac660 con 0x7ff2741099d0 2024-04-22T13:35:54.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.960+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2741b3f50 con 0x7ff2741099d0 2024-04-22T13:35:54.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.960+0000 7ff2637fe640 1 -- 172.21.15.87:0/1183275079 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff268042070 con 0x7ff2741099d0 2024-04-22T13:35:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.970+0000 7ff27b84e740 1 Processor -- start 2024-04-22T13:35:54.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.970+0000 7ff27b84e740 1 -- start start 2024-04-22T13:35:54.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.975+0000 7ff2637fe640 1 -- 172.21.15.87:0/1183275079 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268032800 con 0x7ff2741099d0 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0318700 0x55daf0318ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf03216c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55daf027d1e0 con 0x55daf0319020 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55daf021a390 con 0x55daf0321c00 2024-04-22T13:35:54.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.977+0000 7ff27b84e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55daf02791a0 con 0x55daf0318700 2024-04-22T13:35:54.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.979+0000 7ff2637fe640 1 -- 172.21.15.87:0/1183275079 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff2680035f0 con 0x7ff2741099d0 2024-04-22T13:35:54.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.980+0000 7ff27259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf03216c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:54.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.980+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:54.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.980+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37374/0 (socket says 172.21.15.87:37374) 2024-04-22T13:35:54.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.980+0000 7ff271d9b640 1 -- 172.21.15.87:0/2426209454 learned_addr learned my addr 172.21.15.87:0/2426209454 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:54.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.982+0000 7ff2637fe640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 0x7ff25c0a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:54.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.983+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0318700 0x55daf0318ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.984+0000 7ff2637fe640 1 -- 172.21.15.87:0/1183275079 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff2680f1f20 con 0x7ff2741099d0 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff272d9d640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 0x7ff25c0a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff27259c640 1 -- 172.21.15.87:0/2426209454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0318700 msgr2=0x55daf0318ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff27259c640 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0318700 0x55daf0318ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff27259c640 1 -- 172.21.15.87:0/2426209454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 msgr2=0x55daf0323ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff27259c640 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff27259c640 1 -- 172.21.15.87:0/2426209454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daf00597d0 con 0x55daf0319020 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.985+0000 7ff271d9b640 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.986+0000 7ff27259c640 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf03216c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff26400b9a0 tx=0x7ff26400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77dd3184d6705ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.987+0000 7ff260ff9640 1 -- 172.21.15.87:0/2426209454 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264016070 con 0x55daf0319020 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.987+0000 7ff260ff9640 1 -- 172.21.15.87:0/2426209454 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff26400de30 con 0x55daf0319020 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.988+0000 7ff27b84e740 1 -- 172.21.15.87:0/2426209454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 msgr2=0x55daf03216c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.988+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf03216c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff26400b9a0 tx=0x7ff26400be60 comp rx=0 tx=0).stop 2024-04-22T13:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.990+0000 7ff272d9d640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 0x7ff25c0a61c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00bd50 tx=0x7ff26c004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.991+0000 7ff27b84e740 1 -- 172.21.15.87:0/2426209454 shutdown_connections 2024-04-22T13:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.992+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0321c00 0x55daf0323ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.992+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf03216c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.992+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2426209454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0318700 0x55daf0318ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.992+0000 7ff27b84e740 1 -- 172.21.15.87:0/2426209454 >> 172.21.15.87:0/2426209454 conn(0x55daf0312e30 msgr2=0x55daf0313210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:54.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:54.997+0000 7ff27b84e740 1 -- 172.21.15.87:0/2426209454 shutdown_connections 2024-04-22T13:35:54.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.003+0000 7ff27b84e740 1 -- 172.21.15.87:0/2426209454 wait complete. 2024-04-22T13:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.004+0000 7ff27b84e740 1 -- start start 2024-04-22T13:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.004+0000 7ff27b84e740 1 Processor -- start 2024-04-22T13:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.004+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf031f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55daf0324a40 con 0x55daf0319020 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55daf02362f0 con 0x55daf0318700 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55daf0324bc0 con 0x55daf0321c00 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.005+0000 7ff27b84e740 10 client.0 _ll_register_callbacks cb 0x55daf0243580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.006+0000 7ff27259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.006+0000 7ff27259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37382/0 (socket says 172.21.15.87:37382) 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.006+0000 7ff27259c640 1 -- 172.21.15.87:0/2008948795 learned_addr learned my addr 172.21.15.87:0/2008948795 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.006+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf031f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 -- 172.21.15.87:0/2008948795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 msgr2=0x55daf031f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf031f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 -- 172.21.15.87:0/2008948795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 msgr2=0x55daf031f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 -- 172.21.15.87:0/2008948795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daeff8f3f0 con 0x55daf0318700 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.008+0000 7ff27259c640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff2640057c0 tx=0x7ff26400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd03afc826078a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.009+0000 7ff259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.009+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264016070 con 0x55daf0318700 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.009+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff264004e60 con 0x55daf0318700 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.009+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff26400de30 con 0x55daf0318700 2024-04-22T13:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.010+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff264005060 con 0x55daf0318700 2024-04-22T13:35:55.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.012+0000 7ff27b84e740 10 client.25922 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:55.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.012+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55daeff4f0f0 con 0x55daf0318700 2024-04-22T13:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.017+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff26400de30 con 0x55daf0318700 2024-04-22T13:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.018+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff264003ba0 con 0x55daf0318700 2024-04-22T13:35:55.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 10 client.25922 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:55.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 10 client.25922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 20 client.25922 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0331910 0x55daf0333d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 4 client.25922 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.020+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daf028ac00 con 0x55daf0331910 2024-04-22T13:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.025+0000 7ff271d9b640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0331910 0x55daf0333d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.031+0000 7ff259ffb640 10 client.25922 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:55.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.031+0000 7ff271d9b640 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0331910 0x55daf0333d00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=989d74d5b04faff3 server_cookie=3dbe01b113cb1471 in_seq=0 out_seq=0 2024-04-22T13:35:55.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.036+0000 7ff259ffb640 1 -- 172.21.15.87:0/2008948795 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55daf028ac00 con 0x55daf0331910 2024-04-22T13:35:55.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.036+0000 7ff259ffb640 10 client.25922 handle_command_reply: tid=0 2024-04-22T13:35:55.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.037+0000 7ff27b84e740 1 client.25922 shutdown 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.041+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 msgr2=0x55daf031eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.041+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff2640057c0 tx=0x7ff26400dbb0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 shutdown_connections 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0331910 0x55daf0333d00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf031f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2008948795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf031eb80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.042+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 >> 172.21.15.87:0/2008948795 conn(0x55daf0312e30 msgr2=0x55daf0314b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:55.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.043+0000 7ff259ffb640 0 client.25922 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:55.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.043+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 shutdown_connections 2024-04-22T13:35:55.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.044+0000 7ff27b84e740 1 -- 172.21.15.87:0/2008948795 wait complete. 2024-04-22T13:35:55.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.044+0000 7ff27b84e740 20 client.25922 trim_cache size 0 max 16384 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.107+0000 7ff27b84e740 1 Processor -- start 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.107+0000 7ff27b84e740 1 -- start start 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0339470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37384/0 (socket says 172.21.15.87:37384) 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27259c640 1 -- 172.21.15.87:0/2689686324 learned_addr learned my addr 172.21.15.87:0/2689686324 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff254010fb0 con 0x55daf0319020 2024-04-22T13:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff25400e880 con 0x55daf0318700 2024-04-22T13:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.108+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff25400c6d0 con 0x55daf0321c00 2024-04-22T13:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0339470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff271d9b640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 -- 172.21.15.87:0/2689686324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 msgr2=0x55daf0339470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0339470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 -- 172.21.15.87:0/2689686324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 msgr2=0x55daf031f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 -- 172.21.15.87:0/2689686324 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daf0312c60 con 0x55daf0318700 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff271d9b640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff27259c640 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff264000c00 tx=0x7ff26400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7664f51ae29d6c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff25bfff640 1 -- 172.21.15.87:0/2689686324 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264016040 con 0x55daf0318700 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff25bfff640 1 -- 172.21.15.87:0/2689686324 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff264033070 con 0x55daf0318700 2024-04-22T13:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.109+0000 7ff25bfff640 1 -- 172.21.15.87:0/2689686324 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26402ee00 con 0x55daf0318700 2024-04-22T13:35:55.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 msgr2=0x55daf032d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff264000c00 tx=0x7ff26400be60 comp rx=0 tx=0).stop 2024-04-22T13:35:55.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 shutdown_connections 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0339470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf031f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2689686324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032d9b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 >> 172.21.15.87:0/2689686324 conn(0x55daf0312e30 msgr2=0x55daf0311ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.111+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 shutdown_connections 2024-04-22T13:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.112+0000 7ff27b84e740 1 -- 172.21.15.87:0/2689686324 wait complete. 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.112+0000 7ff27b84e740 1 -- start start 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 Processor -- start 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf0325670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0325bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff25400cd40 con 0x55daf0319020 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff25400e1b0 con 0x55daf0318700 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff25400d410 con 0x55daf0321c00 2024-04-22T13:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff27b84e740 10 client.0 _ll_register_callbacks cb 0x7ff2680ba390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf0325670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff272d9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0325bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff271d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf0325670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57532/0 (socket says 172.21.15.87:57532) 2024-04-22T13:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.113+0000 7ff271d9b640 1 -- 172.21.15.87:0/2558313069 learned_addr learned my addr 172.21.15.87:0/2558313069 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff27259c640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff272d9d640 1 -- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 msgr2=0x55daf032a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff272d9d640 1 -- 172.21.15.87:0/2558313069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 msgr2=0x55daf0325670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf0325670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.114+0000 7ff272d9d640 1 -- 172.21.15.87:0/2558313069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daf0326400 con 0x55daf0321c00 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.115+0000 7ff272d9d640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0325bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x55daf0243600 tx=0x7ff26c005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bf78c2fc24c7acf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.115+0000 7ff259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.115+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c01ed30 con 0x55daf0321c00 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.115+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff26c028470 con 0x55daf0321c00 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.115+0000 7ff27259c640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.116+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff264033070 con 0x55daf0321c00 2024-04-22T13:35:55.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.116+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff26c011e50 con 0x55daf0321c00 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.116+0000 7ff27b84e740 10 client.25963 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.116+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55daf0397d50 con 0x55daf0321c00 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff264033070 con 0x55daf0321c00 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff26c00f040 con 0x55daf0321c00 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 10 client.25963 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:55.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 20 client.25963 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0398480 0x55daf039a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 4 client.25963 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:35:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.117+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55daf0284a70 con 0x55daf0398480 2024-04-22T13:35:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.118+0000 7ff27259c640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0398480 0x55daf039a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.123+0000 7ff27259c640 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0398480 0x55daf039a8a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91cfb40a6f2dcba5 server_cookie=1473f89a41bd42cb in_seq=0 out_seq=0 2024-04-22T13:35:55.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.123+0000 7ff259ffb640 10 client.25963 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.126+0000 7ff259ffb640 1 -- 172.21.15.87:0/2558313069 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55daf0284a70 con 0x55daf0398480 2024-04-22T13:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.126+0000 7ff259ffb640 10 client.25963 handle_command_reply: tid=0 2024-04-22T13:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.126+0000 7ff27b84e740 1 client.25963 shutdown 2024-04-22T13:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 msgr2=0x55daf0325bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0325bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x55daf0243600 tx=0x7ff26c005ec0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 shutdown_connections 2024-04-22T13:35:55.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55daf0398480 0x55daf039a8a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55daf0321c00 0x55daf0325bd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55daf0319020 0x55daf0325670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 --2- 172.21.15.87:0/2558313069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55daf0318700 0x55daf032a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 >> 172.21.15.87:0/2558313069 conn(0x55daf0312e30 msgr2=0x55daf0313e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff259ffb640 0 client.25963 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 shutdown_connections 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.127+0000 7ff27b84e740 1 -- 172.21.15.87:0/2558313069 wait complete. 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.128+0000 7ff27b84e740 20 client.25963 trim_cache size 0 max 16384 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:35:55.124 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:35:55.125 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:35:55.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.128+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 msgr2=0x7ff25c0a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.128+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 0x7ff25c0a61c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00bd50 tx=0x7ff26c004a40 comp rx=0 tx=0).stop 2024-04-22T13:35:55.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.128+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 msgr2=0x7ff2741a7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.128+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff268009460 tx=0x7ff2680349e0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 shutdown_connections 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff25c0a3d00 0x7ff25c0a61c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2741105f0 0x7ff2741abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2741099d0 0x7ff2741a7920 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 --2- 172.21.15.87:0/1183275079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff274108f50 0x7ff2741a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 >> 172.21.15.87:0/1183275079 conn(0x7ff27406ced0 msgr2=0x7ff274070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 shutdown_connections 2024-04-22T13:35:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.129+0000 7ff27359e640 1 -- 172.21.15.87:0/1183275079 wait complete. 2024-04-22T13:35:55.140 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:35:55.140 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.600+0000 7f38c9792640 1 -- 172.21.15.87:0/2185691820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 msgr2=0x7f38c4101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.600+0000 7f38c9792640 1 --2- 172.21.15.87:0/2185691820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4101750 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009ca0 tx=0x7f38bc030200 comp rx=0 tx=0).stop 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 -- 172.21.15.87:0/2185691820 shutdown_connections 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 --2- 172.21.15.87:0/2185691820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 0x7f38c410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 --2- 172.21.15.87:0/2185691820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c4101d20 0x7f38c4105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 --2- 172.21.15.87:0/2185691820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4101750 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 -- 172.21.15.87:0/2185691820 >> 172.21.15.87:0/2185691820 conn(0x7f38c4077a90 msgr2=0x7f38c4077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:55.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 -- 172.21.15.87:0/2185691820 shutdown_connections 2024-04-22T13:35:55.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.601+0000 7f38c9792640 1 -- 172.21.15.87:0/2185691820 wait complete. 2024-04-22T13:35:55.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.604+0000 7f38c9792640 1 Processor -- start 2024-04-22T13:35:55.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.604+0000 7f38c9792640 1 -- start start 2024-04-22T13:35:55.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.605+0000 7f38c9792640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.605+0000 7f38c9792640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 0x7f38c4196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.605+0000 7f38c9792640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 0x7f38c419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c9792640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f38c410ff00 con 0x7f38c4101370 2024-04-22T13:35:55.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c9792640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38c410fd80 con 0x7f38c4106030 2024-04-22T13:35:55.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c9792640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38c4110080 con 0x7f38c419add0 2024-04-22T13:35:55.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 0x7f38c419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 0x7f38c419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39798/0 (socket says 172.21.15.87:39798) 2024-04-22T13:35:55.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.606+0000 7f38c3fff640 1 -- 172.21.15.87:0/1247497557 learned_addr learned my addr 172.21.15.87:0/1247497557 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:55.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.607+0000 7f38c27fc640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 0x7f38c4196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 -- 172.21.15.87:0/1247497557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 msgr2=0x7f38c419b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 0x7f38c419b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 -- 172.21.15.87:0/1247497557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 msgr2=0x7f38c4196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 0x7f38c4196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:55.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.608+0000 7f38c2ffd640 1 -- 172.21.15.87:0/1247497557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c419b8c0 con 0x7f38c4101370 2024-04-22T13:35:55.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.609+0000 7f38c2ffd640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4196560 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f38bc030740 tx=0x7f38bc039d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:55.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.609+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc048030 con 0x7f38c4101370 2024-04-22T13:35:55.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.609+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38bc036b10 con 0x7f38c4101370 2024-04-22T13:35:55.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.609+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc003c80 con 0x7f38c4101370 2024-04-22T13:35:55.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.611+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c419bb50 con 0x7f38c4101370 2024-04-22T13:35:55.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.611+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c41a3260 con 0x7f38c4101370 2024-04-22T13:35:55.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.615+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f38bc04b030 con 0x7f38c4101370 2024-04-22T13:35:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.617+0000 7f38abfff640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 0x7f38ac0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.618+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f38bc0fa430 con 0x7f38c4101370 2024-04-22T13:35:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.618+0000 7f38c27fc640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 0x7f38ac0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.618+0000 7f38c27fc640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 0x7f38ac0a62e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38c4100dc0 tx=0x7f38b4008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.619+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4068d70 con 0x7f38c4101370 2024-04-22T13:35:55.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:55.630+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38bc003e20 con 0x7f38c4101370 2024-04-22T13:35:55.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:55 smithi087 ceph-mon[27999]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.0 GiB/s wr, 2.65k op/s 2024-04-22T13:35:55.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:55 smithi120 ceph-mon[31101]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.0 GiB/s wr, 2.65k op/s 2024-04-22T13:35:55.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:55 smithi132 ceph-mon[31531]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.0 GiB/s wr, 2.65k op/s 2024-04-22T13:35:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.373+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38c41091e0 con 0x7f38c4101370 2024-04-22T13:35:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.375+0000 7f38abfff640 1 -- 172.21.15.87:0/1247497557 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f38bc0c6bf0 con 0x7f38c4101370 2024-04-22T13:35:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:35:56.373 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:35:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:35:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.382+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 msgr2=0x7f38ac0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.382+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 0x7f38ac0a62e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38c4100dc0 tx=0x7f38b4008040 comp rx=0 tx=0).stop 2024-04-22T13:35:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.382+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 msgr2=0x7f38c4196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:56.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.382+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4196560 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f38bc030740 tx=0x7f38bc039d50 comp rx=0 tx=0).stop 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.382+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 shutdown_connections 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38ac0a3e20 0x7f38ac0a62e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38c419add0 0x7f38c419b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38c4106030 0x7f38c4196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 --2- 172.21.15.87:0/1247497557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38c4101370 0x7f38c4196560 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 >> 172.21.15.87:0/1247497557 conn(0x7f38c4077a90 msgr2=0x7f38c40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 shutdown_connections 2024-04-22T13:35:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.383+0000 7f38c9792640 1 -- 172.21.15.87:0/1247497557 wait complete. 2024-04-22T13:35:56.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:35:56.582 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T13:35:56.582 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-22T13:35:56.582 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:35:56.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1247497557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:56.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1247497557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:56.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1247497557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:56.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 -- 172.21.15.87:0/2960108772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 msgr2=0x7f9a6411d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:56.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 --2- 172.21.15.87:0/2960108772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a6411d740 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009860 tx=0x7f9a5402bc00 comp rx=0 tx=0).stop 2024-04-22T13:35:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 -- 172.21.15.87:0/2960108772 shutdown_connections 2024-04-22T13:35:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 --2- 172.21.15.87:0/2960108772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a6411d740 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 --2- 172.21.15.87:0/2960108772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a64108f50 0x7f9a641093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 --2- 172.21.15.87:0/2960108772 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a641105f0 0x7f9a641109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:56.993+0000 7f9a6a658640 1 -- 172.21.15.87:0/2960108772 >> 172.21.15.87:0/2960108772 conn(0x7f9a6406ced0 msgr2=0x7f9a6406d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:56.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.000+0000 7f9a6a658640 1 -- 172.21.15.87:0/2960108772 shutdown_connections 2024-04-22T13:35:56.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.001+0000 7f9a6a658640 1 -- 172.21.15.87:0/2960108772 wait complete. 2024-04-22T13:35:56.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.002+0000 7f9a6a658640 1 Processor -- start 2024-04-22T13:35:56.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.002+0000 7f9a6a658640 1 -- start start 2024-04-22T13:35:56.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.002+0000 7f9a6a658640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 0x7f9a641a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:56.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.003+0000 7f9a69656640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 0x7f9a641a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.003+0000 7f9a69656640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 0x7f9a641a73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57564/0 (socket says 172.21.15.87:57564) 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.003+0000 7f9a6a658640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a641a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.004+0000 7f9a6a658640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a641105f0 0x7f9a641abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.004+0000 7f9a6a658640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9a64120e90 con 0x7f9a64108f50 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.004+0000 7f9a6a658640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9a64120d10 con 0x7f9a641105f0 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.004+0000 7f9a6a658640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9a64121010 con 0x7f9a641099d0 2024-04-22T13:35:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.004+0000 7f9a69656640 1 -- 172.21.15.87:0/3653654095 learned_addr learned my addr 172.21.15.87:0/3653654095 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.005+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a641a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 -- 172.21.15.87:0/3653654095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a641105f0 msgr2=0x7f9a641abc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:35:57.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a641105f0 0x7f9a641abc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 -- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 msgr2=0x7f9a641a73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 0x7f9a641a73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 -- 172.21.15.87:0/3653654095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a641ac3c0 con 0x7f9a641099d0 2024-04-22T13:35:57.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.006+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a641a7910 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c970 tx=0x7f9a6000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.012+0000 7f9a5a7fc640 1 -- 172.21.15.87:0/3653654095 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60007bb0 con 0x7f9a641099d0 2024-04-22T13:35:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.012+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a641ac6b0 con 0x7f9a641099d0 2024-04-22T13:35:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.012+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a641b3f40 con 0x7f9a641099d0 2024-04-22T13:35:57.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.017+0000 7f9a6e589740 1 Processor -- start 2024-04-22T13:35:57.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 -- start start 2024-04-22T13:35:57.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a608bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6097050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a1a5ff01f0 con 0x55a1a6094c00 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1a5f8d4b0 con 0x55a1a608b6d0 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a6e589740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a1a5fec1b0 con 0x55a1a608c080 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a5a7fc640 1 -- 172.21.15.87:0/3653654095 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a60007d50 con 0x7f9a641099d0 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.020+0000 7f9a5a7fc640 1 -- 172.21.15.87:0/3653654095 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60004dd0 con 0x7f9a641099d0 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.021+0000 7f9a5a7fc640 1 -- 172.21.15.87:0/3653654095 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9a6000f4d0 con 0x7f9a641099d0 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.021+0000 7f9a69656640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.021+0000 7f9a69656640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39830/0 (socket says 172.21.15.87:39830) 2024-04-22T13:35:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.021+0000 7f9a69656640 1 -- 172.21.15.87:0/2899676659 learned_addr learned my addr 172.21.15.87:0/2899676659 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 -- 172.21.15.87:0/2899676659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 msgr2=0x55a1a608bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a608bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 -- 172.21.15.87:0/2899676659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 msgr2=0x55a1a6097050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6097050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 -- 172.21.15.87:0/2899676659 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1a5f7f1b0 con 0x55a1a608c080 2024-04-22T13:35:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.022+0000 7f9a69656640 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00f400 tx=0x7f9a5c00f8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f22a9883cc5f6cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.028+0000 7f9a4f7fe640 1 -- 172.21.15.87:0/2899676659 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c003970 con 0x55a1a608c080 2024-04-22T13:35:57.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.028+0000 7f9a4f7fe640 1 -- 172.21.15.87:0/2899676659 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a5c004020 con 0x55a1a608c080 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.029+0000 7f9a6e589740 1 -- 172.21.15.87:0/2899676659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 msgr2=0x55a1a6094630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00f400 tx=0x7f9a5c00f8c0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 -- 172.21.15.87:0/2899676659 shutdown_connections 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6097050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608c080 0x55a1a6094630 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2899676659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a608bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.030+0000 7f9a6e589740 1 -- 172.21.15.87:0/2899676659 >> 172.21.15.87:0/2899676659 conn(0x55a1a6085da0 msgr2=0x55a1a6086180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.031+0000 7f9a6e589740 1 -- 172.21.15.87:0/2899676659 shutdown_connections 2024-04-22T13:35:57.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.031+0000 7f9a6e589740 1 -- 172.21.15.87:0/2899676659 wait complete. 2024-04-22T13:35:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.033+0000 7f9a6e589740 1 -- start start 2024-04-22T13:35:57.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.033+0000 7f9a6e589740 1 Processor -- start 2024-04-22T13:35:57.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.035+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a6094230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.035+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6091520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.038+0000 7f9a69656640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6091520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.038+0000 7f9a69e57640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a6094230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.038+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a6091ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.039+0000 7f9a6e589740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a1a6097bc0 con 0x55a1a6094c00 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.039+0000 7f9a6e589740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1a5fa9500 con 0x55a1a608b6d0 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.039+0000 7f9a6e589740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a1a6097d40 con 0x55a1a6091a60 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.039+0000 7f9a6e589740 10 client.0 _ll_register_callbacks cb 0x55a1a5f77b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:57.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.040+0000 7f9a69e57640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a6094230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37444/0 (socket says 172.21.15.87:37444) 2024-04-22T13:35:57.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.040+0000 7f9a69e57640 1 -- 172.21.15.87:0/3975062382 learned_addr learned my addr 172.21.15.87:0/3975062382 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.040+0000 7f9a5a7fc640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 0x7f9a500a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.041+0000 7f9a5a7fc640 1 -- 172.21.15.87:0/3653654095 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9a600d13b0 con 0x7f9a641099d0 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 -- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 msgr2=0x55a1a6091ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a6091ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 -- 172.21.15.87:0/3975062382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 msgr2=0x55a1a6094230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a6094230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 -- 172.21.15.87:0/3975062382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1a5d023f0 con 0x55a1a6094c00 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a69656640 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6091520 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x55a1a6086e10 tx=0x7f9a5c0189b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896b8bb04193c960 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.042+0000 7f9a4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.043+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c018d30 con 0x55a1a6094c00 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.043+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a1a5f8f930 con 0x55a1a6094c00 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.043+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9a5c015070 con 0x55a1a6094c00 2024-04-22T13:35:57.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.044+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9a5c003970 con 0x55a1a6094c00 2024-04-22T13:35:57.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.045+0000 7f9a6e589740 10 client.16794 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:57.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.045+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a1a5dcc7d0 con 0x55a1a6094c00 2024-04-22T13:35:57.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.047+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a1a5f8f930 con 0x55a1a6094c00 2024-04-22T13:35:57.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.048+0000 7f9a69e57640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 0x7f9a500a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9a5c00b040 con 0x55a1a6094c00 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 10 client.16794 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 10 client.16794 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 20 client.16794 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a60a5150 0x55a1a60a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 4 client.16794 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:57.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.049+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1a5ffdc10 con 0x55a1a60a5150 2024-04-22T13:35:57.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.053+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a60a5150 0x55a1a60a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.058+0000 7f9a4cff9640 10 client.16794 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:57.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.058+0000 7f9a68e55640 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a60a5150 0x55a1a60a75d0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3961109141b3c97 server_cookie=78da4ff3a606bcbe in_seq=0 out_seq=0 2024-04-22T13:35:57.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.059+0000 7f9a4cff9640 1 -- 172.21.15.87:0/3975062382 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1a5ffdc10 con 0x55a1a60a5150 2024-04-22T13:35:57.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.059+0000 7f9a4cff9640 10 client.16794 handle_command_reply: tid=0 2024-04-22T13:35:57.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.059+0000 7f9a6e589740 1 client.16794 shutdown 2024-04-22T13:35:57.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.060+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 msgr2=0x55a1a6091520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.060+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6091520 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x55a1a6086e10 tx=0x7f9a5c0189b0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 shutdown_connections 2024-04-22T13:35:57.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a60a5150 0x55a1a60a75d0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a6091ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6091520 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 --2- 172.21.15.87:0/3975062382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a608b6d0 0x55a1a6094230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a4cff9640 0 client.16794 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 >> 172.21.15.87:0/3975062382 conn(0x55a1a6085da0 msgr2=0x55a1a6087af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.067+0000 7f9a69e57640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 0x7f9a500a6320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c210 tx=0x7f9a5402bc00 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.082+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 shutdown_connections 2024-04-22T13:35:57.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.082+0000 7f9a6e589740 1 -- 172.21.15.87:0/3975062382 wait complete. 2024-04-22T13:35:57.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.082+0000 7f9a6e589740 20 client.16794 trim_cache size 0 max 16384 2024-04-22T13:35:57.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.102+0000 7f7965192640 1 -- 172.21.15.87:0/778491328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 msgr2=0x7f796010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.102+0000 7f795dd9b640 1 -- 172.21.15.87:0/778491328 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950039070 con 0x7f796010e0c0 2024-04-22T13:35:57.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.102+0000 7f7965192640 1 --2- 172.21.15.87:0/778491328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796010e520 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7950009860 tx=0x7f795002bc50 comp rx=0 tx=0).stop 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.112+0000 7f7965192640 1 -- 172.21.15.87:0/778491328 shutdown_connections 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 --2- 172.21.15.87:0/778491328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f796010eb40 0x7f796011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 --2- 172.21.15.87:0/778491328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796010e520 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 --2- 172.21.15.87:0/778491328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7960110550 0x7f7960110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 -- 172.21.15.87:0/778491328 >> 172.21.15.87:0/778491328 conn(0x7f796006ce80 msgr2=0x7f796006d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 -- 172.21.15.87:0/778491328 shutdown_connections 2024-04-22T13:35:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.113+0000 7f7965192640 1 -- 172.21.15.87:0/778491328 wait complete. 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.114+0000 7f7965192640 1 Processor -- start 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 -- start start 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796011cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 0x7f796011d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7960110550 0x7f7960116fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f79601113c0 con 0x7f796010eb40 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7960111240 con 0x7f796010e0c0 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.115+0000 7f7965192640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7960111540 con 0x7f7960110550 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.116+0000 7f795e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 0x7f796011d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.116+0000 7f795e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 0x7f796011d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57604/0 (socket says 172.21.15.87:57604) 2024-04-22T13:35:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.116+0000 7f795e59c640 1 -- 172.21.15.87:0/302805476 learned_addr learned my addr 172.21.15.87:0/302805476 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.123+0000 7f795ed9d640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796011cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.123+0000 7f795f59e640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7960110550 0x7f7960116fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.125+0000 7f795ed9d640 1 -- 172.21.15.87:0/302805476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7960110550 msgr2=0x7f7960116fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.125+0000 7f795ed9d640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7960110550 0x7f7960116fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.125+0000 7f795ed9d640 1 -- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 msgr2=0x7f796011d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.125+0000 7f795ed9d640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 0x7f796011d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.125+0000 7f795ed9d640 1 -- 172.21.15.87:0/302805476 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7960117780 con 0x7f796010e0c0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.126+0000 7f795ed9d640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796011cf40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f795800ef60 tx=0x7f795800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.127+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795801b070 con 0x7f796010e0c0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.127+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f795800ba20 con 0x7f796010e0c0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.127+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958016b20 con 0x7f796010e0c0 2024-04-22T13:35:57.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.127+0000 7f7965192640 1 -- 172.21.15.87:0/302805476 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7960117a70 con 0x7f796010e0c0 2024-04-22T13:35:57.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.127+0000 7f7965192640 1 -- 172.21.15.87:0/302805476 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796007a1b0 con 0x7f796010e0c0 2024-04-22T13:35:57.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.143+0000 7f7965192640 1 -- 172.21.15.87:0/302805476 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7944005290 con 0x7f796010e0c0 2024-04-22T13:35:57.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.145+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f795801e070 con 0x7f796010e0c0 2024-04-22T13:35:57.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.146+0000 7f794ffff640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 0x7f79480a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.147+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f79580d0510 con 0x7f796010e0c0 2024-04-22T13:35:57.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.149+0000 7f795e59c640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 0x7f79480a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.156+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f795809c570 con 0x7f796010e0c0 2024-04-22T13:35:57.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.185+0000 7f795e59c640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 0x7f79480a62a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79500095f0 tx=0x7f795002c170 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.190+0000 7f9a6e589740 1 Processor -- start 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 -- start start 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a60a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6094c00 0x55a1a6093d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9a48010e20 con 0x55a1a608b6d0 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800e6f0 con 0x55a1a6094c00 2024-04-22T13:35:57.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.191+0000 7f9a6e589740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800c5f0 con 0x55a1a6091a60 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.192+0000 7f9a69656640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.192+0000 7f9a69656640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57620/0 (socket says 172.21.15.87:57620) 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.192+0000 7f9a69656640 1 -- 172.21.15.87:0/1255864996 learned_addr learned my addr 172.21.15.87:0/1255864996 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 -- 172.21.15.87:0/1255864996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 msgr2=0x55a1a60a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a60a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 -- 172.21.15.87:0/1255864996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6094c00 msgr2=0x55a1a6093d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6094c00 0x55a1a6093d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 -- 172.21.15.87:0/1255864996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1a6092250 con 0x55a1a608b6d0 2024-04-22T13:35:57.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.193+0000 7f9a69656640 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c02d040 tx=0x7f9a5c00f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b747e1b243485e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.194+0000 7f9a4effd640 1 -- 172.21.15.87:0/1255864996 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c009540 con 0x55a1a608b6d0 2024-04-22T13:35:57.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.194+0000 7f9a4effd640 1 -- 172.21.15.87:0/1255864996 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a5c0050d0 con 0x55a1a608b6d0 2024-04-22T13:35:57.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 -- 172.21.15.87:0/1255864996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 msgr2=0x55a1a60914c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c02d040 tx=0x7f9a5c00f8c0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 -- 172.21.15.87:0/1255864996 shutdown_connections 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6094c00 0x55a1a6093d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a6091a60 0x55a1a60a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 --2- 172.21.15.87:0/1255864996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a608b6d0 0x55a1a60914c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 -- 172.21.15.87:0/1255864996 >> 172.21.15.87:0/1255864996 conn(0x55a1a6085da0 msgr2=0x55a1a6090db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 -- 172.21.15.87:0/1255864996 shutdown_connections 2024-04-22T13:35:57.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.196+0000 7f9a6e589740 1 -- 172.21.15.87:0/1255864996 wait complete. 2024-04-22T13:35:57.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.197+0000 7f9a6e589740 1 -- start start 2024-04-22T13:35:57.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.197+0000 7f9a6e589740 1 Processor -- start 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608b6d0 0x55a1a60967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6091a60 0x55a1a6094fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800cbb0 con 0x55a1a6094c00 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800e020 con 0x55a1a6091a60 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800d280 con 0x55a1a608b6d0 2024-04-22T13:35:57.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.198+0000 7f9a6e589740 10 client.0 _ll_register_callbacks cb 0x7f9a5c018640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:57.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.200+0000 7f9a69e57640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.200+0000 7f9a69e57640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57622/0 (socket says 172.21.15.87:57622) 2024-04-22T13:35:57.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.200+0000 7f9a69e57640 1 -- 172.21.15.87:0/2915358527 learned_addr learned my addr 172.21.15.87:0/2915358527 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.200+0000 7f9a69656640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608b6d0 0x55a1a60967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.200+0000 7f9a68e55640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6091a60 0x55a1a6094fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.201+0000 7f9a69e57640 1 -- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608b6d0 msgr2=0x55a1a60967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.201+0000 7f9a69e57640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608b6d0 0x55a1a60967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.201+0000 7f9a69e57640 1 -- 172.21.15.87:0/2915358527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6091a60 msgr2=0x55a1a6094fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.201+0000 7f9a69e57640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6091a60 0x55a1a6094fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.201+0000 7f9a69e57640 1 -- 172.21.15.87:0/2915358527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1a6095c50 con 0x55a1a6094c00 2024-04-22T13:35:57.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.202+0000 7f9a69e57640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9a54034580 tx=0x7f9a540345b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5811923972e9120e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.202+0000 7f9a4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:57.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.202+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5404f060 con 0x55a1a6094c00 2024-04-22T13:35:57.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.213+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9a54005460 con 0x55a1a6094c00 2024-04-22T13:35:57.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.214+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a1a5ff7a80 con 0x55a1a6094c00 2024-04-22T13:35:57.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.218+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9a540046c0 con 0x55a1a6094c00 2024-04-22T13:35:57.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.218+0000 7f9a6e589740 10 client.16806 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:57.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.218+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a1a609d070 con 0x55a1a6094c00 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.219+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a1a5ff7a80 con 0x55a1a6094c00 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.219+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9a54003a40 con 0x55a1a6094c00 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 10 client.16806 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 10 client.16806 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 20 client.16806 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:57.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a609d750 0x55a1a611e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 4 client.16806 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:57.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.220+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1a5f736e0 con 0x55a1a609d750 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.221+0000 7f9a69656640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a609d750 0x55a1a611e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.223+0000 7f9a69656640 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a609d750 0x55a1a611e760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec649973cf9db74 server_cookie=134e0d3c0555dccc in_seq=0 out_seq=0 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a4cff9640 10 client.16806 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a4cff9640 1 -- 172.21.15.87:0/2915358527 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1a5f736e0 con 0x55a1a609d750 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a4cff9640 10 client.16806 handle_command_reply: tid=0 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 client.16806 shutdown 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 msgr2=0x55a1a6095520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9a54034580 tx=0x7f9a540345b0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 shutdown_connections 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55a1a609d750 0x55a1a611e760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a1a6094c00 0x55a1a6095520 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1a6091a60 0x55a1a6094fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a4cff9640 0 client.16806 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:35:57.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 --2- 172.21.15.87:0/2915358527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a1a608b6d0 0x55a1a60967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.224+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 >> 172.21.15.87:0/2915358527 conn(0x55a1a6085da0 msgr2=0x55a1a608c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.225+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 shutdown_connections 2024-04-22T13:35:57.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.225+0000 7f9a6e589740 1 -- 172.21.15.87:0/2915358527 wait complete. 2024-04-22T13:35:57.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.225+0000 7f9a6e589740 20 client.16806 trim_cache size 0 max 16384 2024-04-22T13:35:57.223 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 msgr2=0x7f9a500a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 0x7f9a500a6320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c210 tx=0x7f9a5402bc00 comp rx=0 tx=0).stop 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 msgr2=0x7f9a641a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a641a7910 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000c970 tx=0x7f9a6000ce30 comp rx=0 tx=0).stop 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 shutdown_connections 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9a500a3e60 0x7f9a500a6320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a641105f0 0x7f9a641abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a641099d0 0x7f9a641a7910 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 --2- 172.21.15.87:0/3653654095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a64108f50 0x7f9a641a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.227+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 >> 172.21.15.87:0/3653654095 conn(0x7f9a6406ced0 msgr2=0x7f9a64070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.237+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 shutdown_connections 2024-04-22T13:35:57.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.238+0000 7f9a6a658640 1 -- 172.21.15.87:0/3653654095 wait complete. 2024-04-22T13:35:57.318 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:35:57.739 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:57 smithi087 ceph-mon[27999]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 780 MiB/s wr, 1.70k op/s 2024-04-22T13:35:57.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.742+0000 7f69f2c9a640 1 -- 172.21.15.87:0/1980786243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 msgr2=0x7f69ec1109a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.742+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/1980786243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec1109a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f69e401ace0 tx=0x7f69e403d130 comp rx=0 tx=0).stop 2024-04-22T13:35:57.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.745+0000 7f69f2c9a640 1 -- 172.21.15.87:0/1980786243 shutdown_connections 2024-04-22T13:35:57.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.745+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/1980786243 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69ec11b730 0x7f69ec11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.745+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/1980786243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.745+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/1980786243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec1109a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.745+0000 7f69f2c9a640 1 -- 172.21.15.87:0/1980786243 >> 172.21.15.87:0/1980786243 conn(0x7f69ec06cf30 msgr2=0x7f69ec06d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.758+0000 7f69f2c9a640 1 -- 172.21.15.87:0/1980786243 shutdown_connections 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.758+0000 7f69f2c9a640 1 -- 172.21.15.87:0/1980786243 wait complete. 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 Processor -- start 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 -- start start 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec11ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69ec11b730 0x7f69ec117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113880 con 0x7f69ec11b730 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113700 con 0x7f69ec1105c0 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69f2c9a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113a00 con 0x7f69ec108fe0 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69ebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.759+0000 7f69ebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37480/0 (socket says 172.21.15.87:37480) 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 -- 172.21.15.87:0/160392146 learned_addr learned my addr 172.21.15.87:0/160392146 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69f0a36640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec11ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 -- 172.21.15.87:0/160392146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 msgr2=0x7f69ec11ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec11ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 -- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69ec11b730 msgr2=0x7f69ec117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69ec11b730 0x7f69ec117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69ebfff640 1 -- 172.21.15.87:0/160392146 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec117bb0 con 0x7f69ec1105c0 2024-04-22T13:35:57.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.760+0000 7f69f0a36640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec11ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:35:57.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.768+0000 7f69ebfff640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00b450 tx=0x7f69dc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.768+0000 7f69e9ffb640 1 -- 172.21.15.87:0/160392146 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc015020 con 0x7f69ec1105c0 2024-04-22T13:35:57.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.769+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ec07a1b0 con 0x7f69ec1105c0 2024-04-22T13:35:57.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.769+0000 7f69e9ffb640 1 -- 172.21.15.87:0/160392146 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69dc011d40 con 0x7f69ec1105c0 2024-04-22T13:35:57.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.769+0000 7f69e9ffb640 1 -- 172.21.15.87:0/160392146 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc015020 con 0x7f69ec1105c0 2024-04-22T13:35:57.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.769+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ec07a700 con 0x7f69ec1105c0 2024-04-22T13:35:57.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.773+0000 7f69e9ffb640 1 -- 172.21.15.87:0/160392146 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f69dc018020 con 0x7f69ec1105c0 2024-04-22T13:35:57.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.774+0000 7f69e9ffb640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 0x7f69d40a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.775+0000 7f69e9ffb640 1 -- 172.21.15.87:0/160392146 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f69dc011690 con 0x7f69ec1105c0 2024-04-22T13:35:57.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.775+0000 7f69f0a36640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 0x7f69d40a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.781+0000 7f69f5969740 1 Processor -- start 2024-04-22T13:35:57.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.781+0000 7f69f5969740 1 -- start start 2024-04-22T13:35:57.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.782+0000 7f69f5969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e630336d0 0x559e63033ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.782+0000 7f69f0a36640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e630336d0 0x559e63033ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69f5969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69f5969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e6303f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69f5969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e62f981f0 con 0x559e630336d0 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69f5969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e62f354b0 con 0x559e63034080 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69f5969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e62f941b0 con 0x559e6303cc00 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69ebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69ebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37482/0 (socket says 172.21.15.87:37482) 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.783+0000 7f69ebfff640 1 -- 172.21.15.87:0/877273724 learned_addr learned my addr 172.21.15.87:0/877273724 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69f0a36640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 0x7f69d40a6620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69e4006010 tx=0x7f69e4000f40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69ebfff640 1 -- 172.21.15.87:0/877273724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 msgr2=0x559e6303f050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69ebfff640 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e6303f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69ebfff640 1 -- 172.21.15.87:0/877273724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e630336d0 msgr2=0x559e63033ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69ebfff640 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e630336d0 0x559e63033ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.784+0000 7f69ebfff640 1 -- 172.21.15.87:0/877273724 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e62f271b0 con 0x559e63034080 2024-04-22T13:35:57.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.785+0000 7f69ebfff640 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f69dc010920 tx=0x7f69dc0a1180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=241f9a021cf232ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.788+0000 7f69d2ffd640 1 -- 172.21.15.87:0/877273724 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0986e0 con 0x559e63034080 2024-04-22T13:35:57.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.788+0000 7f69d2ffd640 1 -- 172.21.15.87:0/877273724 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69dc09c3b0 con 0x559e63034080 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.788+0000 7f69d2ffd640 1 -- 172.21.15.87:0/877273724 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc098880 con 0x559e63034080 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 -- 172.21.15.87:0/877273724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 msgr2=0x559e6303c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f69dc010920 tx=0x7f69dc0a1180 comp rx=0 tx=0).stop 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 -- 172.21.15.87:0/877273724 shutdown_connections 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e6303f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e6303c630 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 --2- 172.21.15.87:0/877273724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e630336d0 0x559e63033ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.789+0000 7f69f5969740 1 -- 172.21.15.87:0/877273724 >> 172.21.15.87:0/877273724 conn(0x559e6302dda0 msgr2=0x559e6302e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.790+0000 7f69f5969740 1 -- 172.21.15.87:0/877273724 shutdown_connections 2024-04-22T13:35:57.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.790+0000 7f69f5969740 1 -- 172.21.15.87:0/877273724 wait complete. 2024-04-22T13:35:57.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.790+0000 7f69f5969740 1 -- start start 2024-04-22T13:35:57.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.790+0000 7f69f5969740 1 Processor -- start 2024-04-22T13:35:57.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.791+0000 7f69f5969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.791+0000 7f69f5969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e63034080 0x559e630399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.791+0000 7f69f5969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.792+0000 7f69f5969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e6303fa80 con 0x559e63034080 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.792+0000 7f69f5969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e62f51500 con 0x559e6303cc00 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.792+0000 7f69f5969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e6303fc00 con 0x559e630336d0 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.792+0000 7f69f5969740 10 client.0 _ll_register_callbacks cb 0x559e62f1fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.793+0000 7f69f0a36640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.793+0000 7f69f0a36640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39886/0 (socket says 172.21.15.87:39886) 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.793+0000 7f69f0a36640 1 -- 172.21.15.87:0/687631639 learned_addr learned my addr 172.21.15.87:0/687631639 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.793+0000 7f69f1237640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.794+0000 7f69f0a36640 1 -- 172.21.15.87:0/687631639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 msgr2=0x559e6303a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.794+0000 7f69f0a36640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.794+0000 7f69f0a36640 1 -- 172.21.15.87:0/687631639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e63034080 msgr2=0x559e630399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.794+0000 7f69f0a36640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e63034080 0x559e630399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.794+0000 7f69f0a36640 1 -- 172.21.15.87:0/687631639 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e62caa3f0 con 0x559e630336d0 2024-04-22T13:35:57.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.795+0000 7f69f0a36640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69e401ace0 tx=0x7f69e401a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4cbaff25df3d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.796+0000 7f69cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:57.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.799+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4060070 con 0x559e630336d0 2024-04-22T13:35:57.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.799+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69dc09c3b0 con 0x559e630336d0 2024-04-22T13:35:57.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.799+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f69e4045440 con 0x559e630336d0 2024-04-22T13:35:57.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.800+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f69e404a040 con 0x559e630336d0 2024-04-22T13:35:57.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.800+0000 7f69f5969740 10 client.25990 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:57.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.800+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e62d747d0 con 0x559e630336d0 2024-04-22T13:35:57.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.803+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69dc09c3b0 con 0x559e630336d0 2024-04-22T13:35:57.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f69e406d020 con 0x559e630336d0 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 10 client.25990 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 10 client.25990 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 20 client.25990 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e6304d190 0x559e6304f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 4 client.25990 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:57.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.805+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e62f37930 con 0x559e6304d190 2024-04-22T13:35:57.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.807+0000 7f69ebfff640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e6304d190 0x559e6304f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.817+0000 7f69ebfff640 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e6304d190 0x559e6304f630 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adb419fddf5f3cca server_cookie=8c342ecbbb0a04ae in_seq=0 out_seq=0 2024-04-22T13:35:57.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.817+0000 7f69cbfff640 10 client.25990 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:57.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.822+0000 7f69cbfff640 1 -- 172.21.15.87:0/687631639 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e62f37930 con 0x559e6304d190 2024-04-22T13:35:57.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.822+0000 7f69cbfff640 10 client.25990 handle_command_reply: tid=0 2024-04-22T13:35:57.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.822+0000 7f69f5969740 1 client.25990 shutdown 2024-04-22T13:35:57.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.826+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 msgr2=0x559e6303c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.826+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69e401ace0 tx=0x7f69e401a5d0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 shutdown_connections 2024-04-22T13:35:57.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e6304d190 0x559e6304f630 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e63034080 0x559e630399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 --2- 172.21.15.87:0/687631639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e630336d0 0x559e6303c620 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69e401ace0 tx=0x7f69e401a5d0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 >> 172.21.15.87:0/687631639 conn(0x559e6302dda0 msgr2=0x559e6302faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.827+0000 7f69cbfff640 0 client.25990 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.828+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 shutdown_connections 2024-04-22T13:35:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.828+0000 7f69f5969740 1 -- 172.21.15.87:0/687631639 wait complete. 2024-04-22T13:35:57.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.828+0000 7f69f5969740 20 client.25990 trim_cache size 0 max 16384 2024-04-22T13:35:57.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.868+0000 7f69f5969740 1 Processor -- start 2024-04-22T13:35:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.868+0000 7f69f5969740 1 -- start start 2024-04-22T13:35:57.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.868+0000 7f69f5969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e63034080 0x559e63048940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.868+0000 7f69f5969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.869+0000 7f69f5969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.869+0000 7f69f5969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec084d90 con 0x559e6303c620 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.869+0000 7f69f5969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec1c7650 con 0x559e6303cc00 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.869+0000 7f69f5969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec082d30 con 0x559e63034080 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.870+0000 7f69f1237640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.870+0000 7f69f1237640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57666/0 (socket says 172.21.15.87:57666) 2024-04-22T13:35:57.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.870+0000 7f69f1237640 1 -- 172.21.15.87:0/3445555579 learned_addr learned my addr 172.21.15.87:0/3445555579 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.871+0000 7f69f1237640 1 -- 172.21.15.87:0/3445555579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e63034080 msgr2=0x559e63048940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:57.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.871+0000 7f69f1237640 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e63034080 0x559e63048940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.871+0000 7f69f1237640 1 -- 172.21.15.87:0/3445555579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 msgr2=0x559e6303c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.871+0000 7f69f1237640 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.871+0000 7f69f1237640 1 -- 172.21.15.87:0/3445555579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e63037f00 con 0x559e6303c620 2024-04-22T13:35:57.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.873+0000 7f69f1237640 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f69e000b5e0 tx=0x7f69e000baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53633a17e1a4e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.886+0000 7f69d27fc640 1 -- 172.21.15.87:0/3445555579 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e000d670 con 0x559e6303c620 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.886+0000 7f69d27fc640 1 -- 172.21.15.87:0/3445555579 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e0018070 con 0x559e6303c620 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.886+0000 7f69d27fc640 1 -- 172.21.15.87:0/3445555579 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0013d40 con 0x559e6303c620 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.887+0000 7f69f5969740 1 -- 172.21.15.87:0/3445555579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 msgr2=0x559e63054f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.887+0000 7f69f5969740 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f69e000b5e0 tx=0x7f69e000baa0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.888+0000 7f69f5969740 1 -- 172.21.15.87:0/3445555579 shutdown_connections 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.888+0000 7f69f5969740 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63054f40 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.888+0000 7f69f5969740 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e6303cc00 0x559e6303c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.888+0000 7f69f5969740 1 --2- 172.21.15.87:0/3445555579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e63034080 0x559e63048940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.888+0000 7f69f5969740 1 -- 172.21.15.87:0/3445555579 >> 172.21.15.87:0/3445555579 conn(0x559e6302dda0 msgr2=0x559e63034e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.900+0000 7f69f5969740 1 -- 172.21.15.87:0/3445555579 shutdown_connections 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.900+0000 7f69f5969740 1 -- 172.21.15.87:0/3445555579 wait complete. 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.900+0000 7f69f5969740 1 -- start start 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.900+0000 7f69f5969740 1 Processor -- start 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63040ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e63041400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec081210 con 0x559e6303c620 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec082660 con 0x559e63034080 2024-04-22T13:35:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec0818c0 con 0x559e6303cc00 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.901+0000 7f69f5969740 10 client.0 _ll_register_callbacks cb 0x7f69e4003a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37516/0 (socket says 172.21.15.87:37516) 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 -- 172.21.15.87:0/1817296885 learned_addr learned my addr 172.21.15.87:0/1817296885 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f1237640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e63041400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 -- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 msgr2=0x559e63041400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e63041400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 -- 172.21.15.87:0/1817296885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 msgr2=0x559e63040ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:57.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63040ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.902+0000 7f69f0a36640 1 -- 172.21.15.87:0/1817296885 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e630b2880 con 0x559e63034080 2024-04-22T13:35:57.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.903+0000 7f69f0a36640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69e4044000 tx=0x7f69e406fbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f535ba255fd5851 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:57.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.905+0000 7f69cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:35:57.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.905+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4060040 con 0x559e63034080 2024-04-22T13:35:57.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.905+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69e0018070 con 0x559e63034080 2024-04-22T13:35:57.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.905+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f69e4045650 con 0x559e63034080 2024-04-22T13:35:57.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.906+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f69e4004860 con 0x559e63034080 2024-04-22T13:35:57.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.907+0000 7f69f5969740 10 client.25958 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:57.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.907+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e630b3b60 con 0x559e63034080 2024-04-22T13:35:57.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.908+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e0018070 con 0x559e63034080 2024-04-22T13:35:57.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.914+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f69e406d040 con 0x559e63034080 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 10 client.25958 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 10 client.25958 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 20 client.25958 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e630b42b0 0x559e630b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 4 client.25958 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.917+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e62fa5c10 con 0x559e630b42b0 2024-04-22T13:35:57.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.918+0000 7f69ebfff640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e630b42b0 0x559e630b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:57.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.920+0000 7f69cbfff640 10 client.25958 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:57.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.920+0000 7f69ebfff640 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e630b42b0 0x559e630b66d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19d1b941486c6d92 server_cookie=fc46ebefb73fd849 in_seq=0 out_seq=0 2024-04-22T13:35:57.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.921+0000 7f69cbfff640 1 -- 172.21.15.87:0/1817296885 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e62fa5c10 con 0x559e630b42b0 2024-04-22T13:35:57.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.921+0000 7f69cbfff640 10 client.25958 handle_command_reply: tid=0 2024-04-22T13:35:57.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.921+0000 7f69f5969740 1 client.25958 shutdown 2024-04-22T13:35:57.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.922+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 msgr2=0x559e63040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.922+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69e4044000 tx=0x7f69e406fbe0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:57 smithi120 ceph-mon[31101]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 780 MiB/s wr, 1.70k op/s 2024-04-22T13:35:57.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 shutdown_connections 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559e630b42b0 0x559e630b66d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e6303cc00 0x559e63041400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e6303c620 0x559e63040ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 --2- 172.21.15.87:0/1817296885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e63034080 0x559e63040980 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.925+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 >> 172.21.15.87:0/1817296885 conn(0x559e6302dda0 msgr2=0x559e6302c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.926+0000 7f69cbfff640 0 client.25958 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.926+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 shutdown_connections 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.926+0000 7f69f5969740 1 -- 172.21.15.87:0/1817296885 wait complete. 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.926+0000 7f69f5969740 20 client.25958 trim_cache size 0 max 16384 2024-04-22T13:35:57.923 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 msgr2=0x7f69d40a6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 0x7f69d40a6620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69e4006010 tx=0x7f69e4000f40 comp rx=0 tx=0).stop 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 msgr2=0x7f69ec116f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00b450 tx=0x7f69dc00b910 comp rx=0 tx=0).stop 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 shutdown_connections 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f69d40a4160 0x7f69d40a6620 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f69ec11b730 0x7f69ec117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69ec1105c0 0x7f69ec116f40 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00b450 tx=0x7f69dc00b910 comp rx=0 tx=0).stop 2024-04-22T13:35:57.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 --2- 172.21.15.87:0/160392146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec108fe0 0x7f69ec11ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:57.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 >> 172.21.15.87:0/160392146 conn(0x7f69ec06cf30 msgr2=0x7f69ec11bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:57.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 shutdown_connections 2024-04-22T13:35:57.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:57.930+0000 7f69f2c9a640 1 -- 172.21.15.87:0/160392146 wait complete. 2024-04-22T13:35:57.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:35:57.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:57 smithi132 ceph-mon[31531]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 780 MiB/s wr, 1.70k op/s 2024-04-22T13:35:58.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.030+0000 7f7965192640 1 -- 172.21.15.87:0/302805476 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7944005850 con 0x7f796010e0c0 2024-04-22T13:35:58.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:35:58.032 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:35:58.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.033+0000 7f794ffff640 1 -- 172.21.15.87:0/302805476 <== mon.2 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f79580a0870 con 0x7f796010e0c0 2024-04-22T13:35:58.033 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:35:58.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 msgr2=0x7f79480a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 0x7f79480a62a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79500095f0 tx=0x7f795002c170 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 msgr2=0x7f796011cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796011cf40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f795800ef60 tx=0x7f795800ef90 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 shutdown_connections 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f79480a3de0 0x7f79480a62a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7960110550 0x7f7960116fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.041+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f796010eb40 0x7f796011d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.042+0000 7f794dffb640 1 --2- 172.21.15.87:0/302805476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f796010e0c0 0x7f796011cf40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.042+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 >> 172.21.15.87:0/302805476 conn(0x7f796006ce80 msgr2=0x7f7960071660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.042+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 shutdown_connections 2024-04-22T13:35:58.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.042+0000 7f794dffb640 1 -- 172.21.15.87:0/302805476 wait complete. 2024-04-22T13:35:58.131 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 90.0 --await' 2024-04-22T13:35:58.131 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 90.0 --await 2024-04-22T13:35:58.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 -- 172.21.15.87:0/199321711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc11b730 msgr2=0x7f30bc11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 --2- 172.21.15.87:0/199321711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc11b730 0x7f30bc11db20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009cd0 tx=0x7f30b402c120 comp rx=0 tx=0).stop 2024-04-22T13:35:58.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 -- 172.21.15.87:0/199321711 shutdown_connections 2024-04-22T13:35:58.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 --2- 172.21.15.87:0/199321711 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc11b730 0x7f30bc11db20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 --2- 172.21.15.87:0/199321711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc10e7d0 0x7f30bc10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 --2- 172.21.15.87:0/199321711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30bc1107a0 0x7f30bc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.325+0000 7f30c2f75640 1 -- 172.21.15.87:0/199321711 >> 172.21.15.87:0/199321711 conn(0x7f30bc06d060 msgr2=0x7f30bc06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.332+0000 7f30c2f75640 1 -- 172.21.15.87:0/199321711 shutdown_connections 2024-04-22T13:35:58.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.338+0000 7f30c2f75640 1 -- 172.21.15.87:0/199321711 wait complete. 2024-04-22T13:35:58.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 Processor -- start 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 -- start start 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30bc10e7d0 0x7f30bc1a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc1abb90 0x7f30bc1ac040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc121030 con 0x7f30bc1abb90 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc120eb0 con 0x7f30bc1107a0 2024-04-22T13:35:58.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.340+0000 7f30c2f75640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc1211b0 con 0x7f30bc10e7d0 2024-04-22T13:35:58.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.341+0000 7f30bbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.341+0000 7f30bbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37544/0 (socket says 172.21.15.87:37544) 2024-04-22T13:35:58.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.341+0000 7f30bbfff640 1 -- 172.21.15.87:0/4042839162 learned_addr learned my addr 172.21.15.87:0/4042839162 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.342+0000 7f30bbfff640 1 -- 172.21.15.87:0/4042839162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30bc10e7d0 msgr2=0x7f30bc1a72d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:35:58.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.342+0000 7f30bbfff640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30bc10e7d0 0x7f30bc1a72d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.342+0000 7f30bbfff640 1 -- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc1abb90 msgr2=0x7f30bc1ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.342+0000 7f30bbfff640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc1abb90 0x7f30bc1ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.342+0000 7f30bbfff640 1 -- 172.21.15.87:0/4042839162 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc1ac770 con 0x7f30bc1107a0 2024-04-22T13:35:58.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.343+0000 7f30bbfff640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00c890 tx=0x7f30ac00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.350+0000 7f30b9ffb640 1 -- 172.21.15.87:0/4042839162 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac004220 con 0x7f30bc1107a0 2024-04-22T13:35:58.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.350+0000 7f30b9ffb640 1 -- 172.21.15.87:0/4042839162 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0043c0 con 0x7f30bc1107a0 2024-04-22T13:35:58.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.350+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc1b3e40 con 0x7f30bc1107a0 2024-04-22T13:35:58.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.351+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc1b41f0 con 0x7f30bc1107a0 2024-04-22T13:35:58.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.351+0000 7f30b9ffb640 1 -- 172.21.15.87:0/4042839162 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac004a00 con 0x7f30bc1107a0 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 Processor -- start 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 -- start start 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b11e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8a6b1f0 con 0x55b2b8b0fa00 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.355+0000 7f30c5b65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8a084b0 con 0x55b2b8b06e80 2024-04-22T13:35:58.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.356+0000 7f30c5b65740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8a671b0 con 0x55b2b8b064d0 2024-04-22T13:35:58.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.358+0000 7f30c1512640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.359+0000 7f30c1512640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39922/0 (socket says 172.21.15.87:39922) 2024-04-22T13:35:58.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.359+0000 7f30c1512640 1 -- 172.21.15.87:0/336862392 learned_addr learned my addr 172.21.15.87:0/336862392 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.360+0000 7f30bbfff640 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b11e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.361+0000 7f30c1512640 1 -- 172.21.15.87:0/336862392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 msgr2=0x55b2b8b0f430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.361+0000 7f30c1512640 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0f430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.361+0000 7f30c1512640 1 -- 172.21.15.87:0/336862392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 msgr2=0x55b2b8b11e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.361+0000 7f30c1512640 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b11e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.361+0000 7f30c1512640 1 -- 172.21.15.87:0/336862392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2b89fa1b0 con 0x55b2b8b064d0 2024-04-22T13:35:58.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.363+0000 7f30b9ffb640 1 -- 172.21.15.87:0/4042839162 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f30ac005220 con 0x7f30bc1107a0 2024-04-22T13:35:58.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.366+0000 7f30b9ffb640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 0x7f30a40a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.367+0000 7f30c0d11640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 0x7f30a40a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.368+0000 7f30b9ffb640 1 -- 172.21.15.87:0/4042839162 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f30ac0d23f0 con 0x7f30bc1107a0 2024-04-22T13:35:58.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.369+0000 7f30c1512640 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009bf0 tx=0x7f30b4009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8872b95d26068363 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.370+0000 7f30a2ffd640 1 -- 172.21.15.87:0/336862392 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4041020 con 0x55b2b8b064d0 2024-04-22T13:35:58.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.371+0000 7f30a2ffd640 1 -- 172.21.15.87:0/336862392 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30b40327f0 con 0x55b2b8b064d0 2024-04-22T13:35:58.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 -- 172.21.15.87:0/336862392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 msgr2=0x55b2b8b068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c0d11640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 0x7f30a40a63f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55b2b8b01c10 tx=0x7f30b0006bd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009bf0 tx=0x7f30b4009870 comp rx=0 tx=0).stop 2024-04-22T13:35:58.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 -- 172.21.15.87:0/336862392 shutdown_connections 2024-04-22T13:35:58.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b11e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 --2- 172.21.15.87:0/336862392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b064d0 0x55b2b8b068b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.372+0000 7f30c5b65740 1 -- 172.21.15.87:0/336862392 >> 172.21.15.87:0/336862392 conn(0x55b2b8b00ba0 msgr2=0x55b2b8b00f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.373+0000 7f30c5b65740 1 -- 172.21.15.87:0/336862392 shutdown_connections 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.373+0000 7f30c5b65740 1 -- 172.21.15.87:0/336862392 wait complete. 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.373+0000 7f30c5b65740 1 -- start start 2024-04-22T13:35:58.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.373+0000 7f30c5b65740 1 Processor -- start 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b0c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8b23970 con 0x55b2b8b0fa00 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8a24500 con 0x55b2b8b0c810 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b2b8b23af0 con 0x55b2b8b06e80 2024-04-22T13:35:58.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.374+0000 7f30c5b65740 10 client.0 _ll_register_callbacks cb 0x55b2b89f2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:58.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.375+0000 7f30c0d11640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.375+0000 7f30c0d11640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39926/0 (socket says 172.21.15.87:39926) 2024-04-22T13:35:58.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.375+0000 7f30c0d11640 1 -- 172.21.15.87:0/1426090361 learned_addr learned my addr 172.21.15.87:0/1426090361 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.376+0000 7f30c0d11640 1 -- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0c810 msgr2=0x55b2b8b0cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:58.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.376+0000 7f30c0d11640 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.376+0000 7f30c0d11640 1 -- 172.21.15.87:0/1426090361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 msgr2=0x55b2b8b0c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.376+0000 7f30c0d11640 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b0c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.376+0000 7f30c0d11640 1 -- 172.21.15.87:0/1426090361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2b877d3f0 con 0x55b2b8b06e80 2024-04-22T13:35:58.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.377+0000 7f30c0d11640 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30b0013f40 tx=0x7f30b00046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2db7d21647cc2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.377+0000 7f309bfff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:58.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.378+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b001d5f0 con 0x55b2b8b06e80 2024-04-22T13:35:58.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.386+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f30b001d790 con 0x55b2b8b06e80 2024-04-22T13:35:58.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.386+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2b8a0a930 con 0x55b2b8b06e80 2024-04-22T13:35:58.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.387+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f30b002b050 con 0x55b2b8b06e80 2024-04-22T13:35:58.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.387+0000 7f30c5b65740 10 client.25999 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:58.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.387+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b2b88477d0 con 0x55b2b8b06e80 2024-04-22T13:35:58.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b2b8a0a930 con 0x55b2b8b06e80 2024-04-22T13:35:58.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f30b0012070 con 0x55b2b8b06e80 2024-04-22T13:35:58.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 10 client.25999 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:58.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 10 client.25999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:58.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 20 client.25999 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:58.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b30f10 0x55b2b8b333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 4 client.25999 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:58.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.390+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2b8a78c10 con 0x55b2b8b30f10 2024-04-22T13:35:58.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.392+0000 7f30bbfff640 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b30f10 0x55b2b8b333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.393+0000 7f309bfff640 10 client.25999 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.393+0000 7f30bbfff640 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b30f10 0x55b2b8b333d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139ff71907af8e02 server_cookie=2797334250f042fc in_seq=0 out_seq=0 2024-04-22T13:35:58.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.397+0000 7f309bfff640 1 -- 172.21.15.87:0/1426090361 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2b8a78c10 con 0x55b2b8b30f10 2024-04-22T13:35:58.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.397+0000 7f309bfff640 10 client.25999 handle_command_reply: tid=0 2024-04-22T13:35:58.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.397+0000 7f30c5b65740 1 client.25999 shutdown 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 msgr2=0x55b2b8b0efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f30b0013f40 tx=0x7f30b00046e0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 shutdown_connections 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b30f10 0x55b2b8b333d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b0c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 --2- 172.21.15.87:0/1426090361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0efe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 >> 172.21.15.87:0/1426090361 conn(0x55b2b8b00ba0 msgr2=0x55b2b8b02060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f309bfff640 0 client.25999 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 shutdown_connections 2024-04-22T13:35:58.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.398+0000 7f30c5b65740 1 -- 172.21.15.87:0/1426090361 wait complete. 2024-04-22T13:35:58.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.399+0000 7f30c5b65740 20 client.25999 trim_cache size 0 max 16384 2024-04-22T13:35:58.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.437+0000 7f30c5b65740 1 Processor -- start 2024-04-22T13:35:58.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 -- start start 2024-04-22T13:35:58.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b38ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8010fb0 con 0x55b2b8b0c810 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30a800e880 con 0x55b2b8b0fa00 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.438+0000 7f30c5b65740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30a800c6d0 con 0x55b2b8b06e80 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f30bbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f30bbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57718/0 (socket says 172.21.15.87:57718) 2024-04-22T13:35:58.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f30bbfff640 1 -- 172.21.15.87:0/2558567007 learned_addr learned my addr 172.21.15.87:0/2558567007 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f30bbfff640 1 -- 172.21.15.87:0/2558567007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 msgr2=0x55b2b8b0bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f30bbfff640 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f30bbfff640 1 -- 172.21.15.87:0/2558567007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0fa00 msgr2=0x55b2b8b38ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f30bbfff640 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b38ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f30bbfff640 1 -- 172.21.15.87:0/2558567007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2b8afc230 con 0x55b2b8b0c810 2024-04-22T13:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 -- 172.21.15.87:0/400637254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0110b20 msgr2=0x7f10a0109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 --2- 172.21.15.87:0/400637254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0110b20 0x7f10a0109490 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f109401a910 tx=0x7f109403cd40 comp rx=0 tx=0).stop 2024-04-22T13:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 -- 172.21.15.87:0/400637254 shutdown_connections 2024-04-22T13:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 --2- 172.21.15.87:0/400637254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a011b730 0x7f10a011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 --2- 172.21.15.87:0/400637254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0110b20 0x7f10a0109490 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 --2- 172.21.15.87:0/400637254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10a0110740 0x7f10a0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.439+0000 7f10a51b6640 1 -- 172.21.15.87:0/400637254 >> 172.21.15.87:0/400637254 conn(0x7f10a006d020 msgr2=0x7f10a006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.440+0000 7f10a51b6640 1 -- 172.21.15.87:0/400637254 shutdown_connections 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.440+0000 7f10a51b6640 1 -- 172.21.15.87:0/400637254 wait complete. 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 Processor -- start 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 -- start start 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a0110740 0x7f10a011ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10a011b730 0x7f10a0116ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01138a0 con 0x7f10a011b730 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0113720 con 0x7f10a0110740 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.441+0000 7f10a51b6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0113a20 con 0x7f10a0117400 2024-04-22T13:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.442+0000 7f109f59e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.442+0000 7f109f59e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39960/0 (socket says 172.21.15.87:39960) 2024-04-22T13:35:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.442+0000 7f109f59e640 1 -- 172.21.15.87:0/3990822275 learned_addr learned my addr 172.21.15.87:0/3990822275 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.442+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a0110740 0x7f10a011ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.443+0000 7f109f59e640 1 -- 172.21.15.87:0/3990822275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a0110740 msgr2=0x7f10a011ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.443+0000 7f109f59e640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a0110740 0x7f10a011ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.443+0000 7f109f59e640 1 -- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10a011b730 msgr2=0x7f10a0116ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.443+0000 7f109f59e640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10a011b730 0x7f10a0116ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.443+0000 7f109f59e640 1 -- 172.21.15.87:0/3990822275 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a007c1d0 con 0x7f10a0117400 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.444+0000 7f109f59e640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f109804f9e0 tx=0x7f109804fea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.444+0000 7f1087fff640 1 -- 172.21.15.87:0/3990822275 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109805d070 con 0x7f10a0117400 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.444+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a007c4c0 con 0x7f10a0117400 2024-04-22T13:35:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.445+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a007c9c0 con 0x7f10a0117400 2024-04-22T13:35:58.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.445+0000 7f30bbfff640 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f30ac09e0a0 tx=0x7f30ac01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e649cdfd55c1c07c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.447+0000 7f30a27fc640 1 -- 172.21.15.87:0/2558567007 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac01d040 con 0x55b2b8b0c810 2024-04-22T13:35:58.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.447+0000 7f30a27fc640 1 -- 172.21.15.87:0/2558567007 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0b7070 con 0x55b2b8b0c810 2024-04-22T13:35:58.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.447+0000 7f30a27fc640 1 -- 172.21.15.87:0/2558567007 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac09aa10 con 0x55b2b8b0c810 2024-04-22T13:35:58.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.449+0000 7f30c5b65740 1 -- 172.21.15.87:0/2558567007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 msgr2=0x55b2b8b0eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.449+0000 7f30c5b65740 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f30ac09e0a0 tx=0x7f30ac01f000 comp rx=0 tx=0).stop 2024-04-22T13:35:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.448+0000 7f1087fff640 1 -- 172.21.15.87:0/3990822275 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1098058440 con 0x7f10a0117400 2024-04-22T13:35:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.449+0000 7f1087fff640 1 -- 172.21.15.87:0/3990822275 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098056580 con 0x7f10a0117400 2024-04-22T13:35:58.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 -- 172.21.15.87:0/2558567007 shutdown_connections 2024-04-22T13:35:58.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b38ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b0eb10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 --2- 172.21.15.87:0/2558567007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b06e80 0x55b2b8b0bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 -- 172.21.15.87:0/2558567007 >> 172.21.15.87:0/2558567007 conn(0x55b2b8b00ba0 msgr2=0x55b2b8affc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f10a7dab740 1 Processor -- start 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f10a7dab740 1 -- start start 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c07afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c0864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56012bfdf8d0 con 0x56012c07b8d0 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56012bf7ce80 con 0x56012c07ab60 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.451+0000 7f10a7dab740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56012bfdb890 con 0x56012c0840e0 2024-04-22T13:35:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f1087fff640 1 -- 172.21.15.87:0/3990822275 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1098060070 con 0x7f10a0117400 2024-04-22T13:35:58.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.450+0000 7f30c5b65740 1 -- 172.21.15.87:0/2558567007 shutdown_connections 2024-04-22T13:35:58.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.454+0000 7f30c5b65740 1 -- 172.21.15.87:0/2558567007 wait complete. 2024-04-22T13:35:58.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f30c5b65740 1 -- start start 2024-04-22T13:35:58.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f30c5b65740 1 Processor -- start 2024-04-22T13:35:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b28e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b29360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30a800cd40 con 0x55b2b8b0c810 2024-04-22T13:35:58.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30a800e1b0 con 0x55b2b8b06e80 2024-04-22T13:35:58.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30a800d410 con 0x55b2b8b0fa00 2024-04-22T13:35:58.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f30c5b65740 10 client.0 _ll_register_callbacks cb 0x7f30b0013bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f109e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f109e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57740/0 (socket says 172.21.15.87:57740) 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f109e59c640 1 -- 172.21.15.87:0/2091268284 learned_addr learned my addr 172.21.15.87:0/2091268284 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.455+0000 7f109e59c640 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c0864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.456+0000 7f1087fff640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 0x7f10880a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.456+0000 7f109f59e640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 0x7f10880a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f1087fff640 1 -- 172.21.15.87:0/3990822275 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f1098113b30 con 0x7f10a0117400 2024-04-22T13:35:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f109f59e640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 0x7f10880a63d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f10980dfb80 tx=0x7f10980b1630 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.457+0000 7f109ed9d640 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c07afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f109e59c640 1 -- 172.21.15.87:0/2091268284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 msgr2=0x56012c07afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f109e59c640 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c07afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f109e59c640 1 -- 172.21.15.87:0/2091268284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 msgr2=0x56012c07a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f109e59c640 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f109e59c640 1 -- 172.21.15.87:0/2091268284 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56012bf9e320 con 0x56012c0840e0 2024-04-22T13:35:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f30c1512640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f30c1512640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:39984/0 (socket says 172.21.15.87:39984) 2024-04-22T13:35:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f30c1512640 1 -- 172.21.15.87:0/3515375461 learned_addr learned my addr 172.21.15.87:0/3515375461 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.461+0000 7f30bbfff640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b29360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.462+0000 7f30c0d11640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b28e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.464+0000 7f30c1512640 1 -- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 msgr2=0x55b2b8b28e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.464+0000 7f30c1512640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b28e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.464+0000 7f30c1512640 1 -- 172.21.15.87:0/3515375461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 msgr2=0x55b2b8b29360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.464+0000 7f30c1512640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b29360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.464+0000 7f30c1512640 1 -- 172.21.15.87:0/3515375461 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2b8b09ee0 con 0x55b2b8b0fa00 2024-04-22T13:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.468+0000 7f109e59c640 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c0864d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f109403d350 tx=0x7f1094005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ebdf4d5f75a99d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.486+0000 7f30c1512640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30b402c730 tx=0x7f30b40347e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa067841959dd9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.487+0000 7f10857fa640 1 -- 172.21.15.87:0/2091268284 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094059070 con 0x56012c0840e0 2024-04-22T13:35:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.487+0000 7f10857fa640 1 -- 172.21.15.87:0/2091268284 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109403dda0 con 0x56012c0840e0 2024-04-22T13:35:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.487+0000 7f10857fa640 1 -- 172.21.15.87:0/2091268284 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094003d40 con 0x56012c0840e0 2024-04-22T13:35:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 -- 172.21.15.87:0/2091268284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 msgr2=0x56012c0864d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c0864d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f109403d350 tx=0x7f1094005fd0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.487+0000 7f309bfff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 -- 172.21.15.87:0/2091268284 shutdown_connections 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c0864d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c07afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 --2- 172.21.15.87:0/2091268284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.488+0000 7f10a7dab740 1 -- 172.21.15.87:0/2091268284 >> 172.21.15.87:0/2091268284 conn(0x56012c041c90 msgr2=0x56012c075850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.489+0000 7f10a7dab740 1 -- 172.21.15.87:0/2091268284 shutdown_connections 2024-04-22T13:35:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.489+0000 7f10a7dab740 1 -- 172.21.15.87:0/2091268284 wait complete. 2024-04-22T13:35:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.490+0000 7f10a7dab740 1 -- start start 2024-04-22T13:35:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.490+0000 7f10a7dab740 1 Processor -- start 2024-04-22T13:35:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07ab60 0x56012c083ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c0840e0 0x56012c081470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56012c086db0 con 0x56012c07b8d0 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56012bf67eb0 con 0x56012c0840e0 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56012c086f30 con 0x56012c07ab60 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.492+0000 7f10a7dab740 10 client.0 _ll_register_callbacks cb 0x56012bf94b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.494+0000 7f109ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.494+0000 7f109ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57768/0 (socket says 172.21.15.87:57768) 2024-04-22T13:35:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.494+0000 7f109ed9d640 1 -- 172.21.15.87:0/36214324 learned_addr learned my addr 172.21.15.87:0/36214324 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.498+0000 7f109ed9d640 1 -- 172.21.15.87:0/36214324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07ab60 msgr2=0x56012c083ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.498+0000 7f109ed9d640 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07ab60 0x56012c083ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.498+0000 7f109ed9d640 1 -- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c0840e0 msgr2=0x56012c081470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.498+0000 7f109ed9d640 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c0840e0 0x56012c081470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.498+0000 7f109ed9d640 1 -- 172.21.15.87:0/36214324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56012bcf2210 con 0x56012c07b8d0 2024-04-22T13:35:58.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.500+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b404b060 con 0x55b2b8b0fa00 2024-04-22T13:35:58.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.500+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f30b4034a80 con 0x55b2b8b0fa00 2024-04-22T13:35:58.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.501+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ac0b7070 con 0x55b2b8b0fa00 2024-04-22T13:35:58.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.501+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f30b4034c80 con 0x55b2b8b0fa00 2024-04-22T13:35:58.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.507+0000 7f30c5b65740 10 client.26011 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:58.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.507+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b2b8b10dd0 con 0x55b2b8b0fa00 2024-04-22T13:35:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.513+0000 7f109ed9d640 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f10900104d0 tx=0x7f1090010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfd4a86145c77df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.513+0000 7f1082ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.515+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30ac0b7070 con 0x55b2b8b0fa00 2024-04-22T13:35:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.515+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f30b403c030 con 0x55b2b8b0fa00 2024-04-22T13:35:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 10 client.26011 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:58.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 10 client.26011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 20 client.26011 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b97510 0x55b2b8b99900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 4 client.26011 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:35:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.516+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b2b8a72a80 con 0x55b2b8b97510 2024-04-22T13:35:58.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.517+0000 7f30c0d11640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b97510 0x55b2b8b99900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.520+0000 7f30c0d11640 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b97510 0x55b2b8b99900 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e5f528aeee42d8 server_cookie=486aaddc85cf681e in_seq=0 out_seq=0 2024-04-22T13:35:58.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.521+0000 7f309bfff640 10 client.26011 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.518 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.522+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109001a020 con 0x56012c07b8d0 2024-04-22T13:35:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.532+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109403dda0 con 0x56012c07b8d0 2024-04-22T13:35:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.532+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f1090016690 con 0x56012c07b8d0 2024-04-22T13:35:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.539+0000 7f309bfff640 1 -- 172.21.15.87:0/3515375461 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b2b8a72a80 con 0x55b2b8b97510 2024-04-22T13:35:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.539+0000 7f309bfff640 10 client.26011 handle_command_reply: tid=0 2024-04-22T13:35:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.539+0000 7f30c5b65740 1 client.26011 shutdown 2024-04-22T13:35:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.542+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1090025050 con 0x56012c07b8d0 2024-04-22T13:35:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.542+0000 7f10a7dab740 10 client.16830 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.542+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56012bdbe620 con 0x56012c07b8d0 2024-04-22T13:35:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 msgr2=0x55b2b8b096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30b402c730 tx=0x7f30b40347e0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 shutdown_connections 2024-04-22T13:35:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b2b8b97510 0x55b2b8b99900 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b2b8b0fa00 0x55b2b8b096b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b2b8b0c810 0x55b2b8b29360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 --2- 172.21.15.87:0/3515375461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2b8b06e80 0x55b2b8b28e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 >> 172.21.15.87:0/3515375461 conn(0x55b2b8b00ba0 msgr2=0x55b2b8b01d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f309bfff640 0 client.26011 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 shutdown_connections 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.543+0000 7f30c5b65740 1 -- 172.21.15.87:0/3515375461 wait complete. 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.544+0000 7f30c5b65740 20 client.26011 trim_cache size 0 max 16384 2024-04-22T13:35:58.541 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:35:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.554+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109403dda0 con 0x56012c07b8d0 2024-04-22T13:35:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.554+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1090014a80 con 0x56012c07b8d0 2024-04-22T13:35:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 10 client.16830 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 10 client.16830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:58.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 msgr2=0x7f30a40a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 0x7f30a40a63f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55b2b8b01c10 tx=0x7f30b0006bd0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 msgr2=0x7f30bc1a7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 20 client.16830 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c094260 0x56012c096670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 4 client.16830 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56012bff3880 con 0x56012c094260 2024-04-22T13:35:58.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.556+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00c890 tx=0x7f30ac00cd50 comp rx=0 tx=0).stop 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 shutdown_connections 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30a40a3f30 0x7f30a40a63f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30bc1abb90 0x7f30bc1ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc1107a0 0x7f30bc1a7810 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 --2- 172.21.15.87:0/4042839162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30bc10e7d0 0x7f30bc1a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 >> 172.21.15.87:0/4042839162 conn(0x7f30bc06d060 msgr2=0x7f30bc071720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 shutdown_connections 2024-04-22T13:35:58.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.562+0000 7f30c2f75640 1 -- 172.21.15.87:0/4042839162 wait complete. 2024-04-22T13:35:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.564+0000 7f109e59c640 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c094260 0x56012c096670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.584+0000 7f109e59c640 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c094260 0x56012c096670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f4714382a0f12 server_cookie=80044c4c344b40d3 in_seq=0 out_seq=0 2024-04-22T13:35:58.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.584+0000 7f1082ff5640 10 client.16830 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.595+0000 7f1082ff5640 1 -- 172.21.15.87:0/36214324 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56012bff3880 con 0x56012c094260 2024-04-22T13:35:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.595+0000 7f1082ff5640 10 client.16830 handle_command_reply: tid=0 2024-04-22T13:35:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.595+0000 7f10a7dab740 1 client.16830 shutdown 2024-04-22T13:35:58.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.600+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 msgr2=0x56012c080e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.600+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f10900104d0 tx=0x7f1090010990 comp rx=0 tx=0).stop 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.604+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 shutdown_connections 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.604+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c094260 0x56012c096670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.605+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c0840e0 0x56012c081470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.605+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c080e60 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.605+0000 7f10a7dab740 1 --2- 172.21.15.87:0/36214324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07ab60 0x56012c083ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.605+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 >> 172.21.15.87:0/36214324 conn(0x56012c041c90 msgr2=0x56012c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.605+0000 7f1082ff5640 0 client.16830 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.605 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/302805476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.608+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 shutdown_connections 2024-04-22T13:35:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.608+0000 7f10a7dab740 1 -- 172.21.15.87:0/36214324 wait complete. 2024-04-22T13:35:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.608+0000 7f10a7dab740 20 client.16830 trim_cache size 0 max 16384 2024-04-22T13:35:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.714+0000 7f10a7dab740 1 Processor -- start 2024-04-22T13:35:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.715+0000 7f10a7dab740 1 -- start start 2024-04-22T13:35:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.715+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07b8d0 0x56012c08fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c0840e0 0x56012c083590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f109e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f109e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37622/0 (socket says 172.21.15.87:37622) 2024-04-22T13:35:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f109e59c640 1 -- 172.21.15.87:0/3050421907 learned_addr learned my addr 172.21.15.87:0/3050421907 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f108c010fa0 con 0x56012c0840e0 2024-04-22T13:35:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.716+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f108c00e870 con 0x56012c07ab60 2024-04-22T13:35:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.717+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f108c00c6c0 con 0x56012c07b8d0 2024-04-22T13:35:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.721+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07b8d0 0x56012c08fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.721+0000 7f109e59c640 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c0840e0 0x56012c083590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.724+0000 7f109e59c640 1 -- 172.21.15.87:0/3050421907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07b8d0 msgr2=0x56012c08fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.724+0000 7f109e59c640 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07b8d0 0x56012c08fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.724+0000 7f109e59c640 1 -- 172.21.15.87:0/3050421907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c0840e0 msgr2=0x56012c083590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.724+0000 7f109e59c640 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c0840e0 0x56012c083590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.724+0000 7f109e59c640 1 -- 172.21.15.87:0/3050421907 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56012c07ffc0 con 0x56012c07ab60 2024-04-22T13:35:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.730+0000 7f109e59c640 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1094004320 tx=0x7f10940040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=889e3653f38b07f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.736+0000 7f1084ff9640 1 -- 172.21.15.87:0/3050421907 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094059030 con 0x56012c07ab60 2024-04-22T13:35:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.736+0000 7f1084ff9640 1 -- 172.21.15.87:0/3050421907 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109405a040 con 0x56012c07ab60 2024-04-22T13:35:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.736+0000 7f1084ff9640 1 -- 172.21.15.87:0/3050421907 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094060800 con 0x56012c07ab60 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.740+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 msgr2=0x56012c080e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1094004320 tx=0x7f10940040f0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 shutdown_connections 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c0840e0 0x56012c083590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c07b8d0 0x56012c08fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3050421907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c080e00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 >> 172.21.15.87:0/3050421907 conn(0x56012c041c90 msgr2=0x56012c0806f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 shutdown_connections 2024-04-22T13:35:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.741+0000 7f10a7dab740 1 -- 172.21.15.87:0/3050421907 wait complete. 2024-04-22T13:35:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.742+0000 7f10a7dab740 1 -- start start 2024-04-22T13:35:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.743+0000 7f10a7dab740 1 Processor -- start 2024-04-22T13:35:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.743+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c08c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.744+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.744+0000 7f10a7dab740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c07d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.744+0000 7f10a7dab740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f108c00cd30 con 0x56012c07b8d0 2024-04-22T13:35:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.744+0000 7f10a7dab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f108c00e1a0 con 0x56012c07ab60 2024-04-22T13:35:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.744+0000 7f10a7dab740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f108c00d400 con 0x56012c0840e0 2024-04-22T13:35:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.745+0000 7f10a7dab740 10 client.0 _ll_register_callbacks cb 0x7f109000b4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.746+0000 7f109e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c08c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.746+0000 7f109e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c08c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37634/0 (socket says 172.21.15.87:37634) 2024-04-22T13:35:58.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.746+0000 7f109e59c640 1 -- 172.21.15.87:0/3727278424 learned_addr learned my addr 172.21.15.87:0/3727278424 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.746+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.746 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.747+0000 7f109ed9d640 1 -- 172.21.15.87:0/3727278424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 msgr2=0x56012c07d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.747+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c07d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.747+0000 7f109ed9d640 1 -- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 msgr2=0x56012c08c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.747+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c08c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.747 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.747+0000 7f109ed9d640 1 -- 172.21.15.87:0/3727278424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56012c07db50 con 0x56012c07b8d0 2024-04-22T13:35:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.750+0000 7f109ed9d640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07d0d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1090014ed0 tx=0x7f10900126f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f6f614c150f1e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.750+0000 7f1082ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.751+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109001a040 con 0x56012c07b8d0 2024-04-22T13:35:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.751+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f1090012b70 con 0x56012c07b8d0 2024-04-22T13:35:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.751+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109405a040 con 0x56012c07b8d0 2024-04-22T13:35:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.752+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f10900102b0 con 0x56012c07b8d0 2024-04-22T13:35:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.753+0000 7f10a7dab740 10 client.16836 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.754+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56012c07ee60 con 0x56012c07b8d0 2024-04-22T13:35:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.758+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1090005e40 con 0x56012c07b8d0 2024-04-22T13:35:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f10900036b0 con 0x56012c07b8d0 2024-04-22T13:35:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 10 client.16836 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:35:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 20 client.16836 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c0fbd90 0x56012c0fe1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 4 client.16836 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:35:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.760+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f109405a040 con 0x56012c0fbd90 2024-04-22T13:35:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.765+0000 7f109e59c640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c0fbd90 0x56012c0fe1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.775+0000 7f109e59c640 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c0fbd90 0x56012c0fe1b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea660d947a76acff server_cookie=647985a0ba837133 in_seq=0 out_seq=0 2024-04-22T13:35:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:35:58.776+0000 7f1082ff5640 10 client.16836 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 -- 172.21.15.87:0/495521668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f380810b210 msgr2=0x7f3808113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 --2- 172.21.15.87:0/495521668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f380810b210 0x7f3808113960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3800009ca0 tx=0x7f380002c1c0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 -- 172.21.15.87:0/495521668 shutdown_connections 2024-04-22T13:35:58.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 --2- 172.21.15.87:0/495521668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3808113ea0 0x7f3808116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 --2- 172.21.15.87:0/495521668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f380810b210 0x7f3808113960 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 --2- 172.21.15.87:0/495521668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810a8f0 0x7f380810acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.791+0000 7f380d3e6640 1 -- 172.21.15.87:0/495521668 >> 172.21.15.87:0/495521668 conn(0x7f380806ce60 msgr2=0x7f380806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.797+0000 7f380d3e6640 1 -- 172.21.15.87:0/495521668 shutdown_connections 2024-04-22T13:35:58.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.800+0000 7f380d3e6640 1 -- 172.21.15.87:0/495521668 wait complete. 2024-04-22T13:35:58.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.801+0000 7f380d3e6640 1 Processor -- start 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.801+0000 7f380d3e6640 1 -- start start 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380810a8f0 0x7f38081a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 0x7f38081a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 0x7f3808112d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f380811a870 con 0x7f3808113ea0 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f380811a6f0 con 0x7f380810a8f0 2024-04-22T13:35:58.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.802+0000 7f380d3e6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f380811a9f0 con 0x7f380810b210 2024-04-22T13:35:58.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.804+0000 7f380cbe5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 0x7f3808112d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.804+0000 7f380cbe5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 0x7f3808112d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57802/0 (socket says 172.21.15.87:57802) 2024-04-22T13:35:58.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.804+0000 7f380cbe5640 1 -- 172.21.15.87:0/2726618020 learned_addr learned my addr 172.21.15.87:0/2726618020 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.804+0000 7f38077fe640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 0x7f38081a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.805+0000 7f38077fe640 1 -- 172.21.15.87:0/2726618020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380810a8f0 msgr2=0x7f38081a7a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.805+0000 7f38077fe640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380810a8f0 0x7f38081a7a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.805+0000 7f38077fe640 1 -- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 msgr2=0x7f3808112d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.805+0000 7f38077fe640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 0x7f3808112d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.806+0000 7f38077fe640 1 -- 172.21.15.87:0/2726618020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3808113610 con 0x7f380810b210 2024-04-22T13:35:58.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.806+0000 7f38077fe640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 0x7f38081a7f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3800009bf0 tx=0x7f3800003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.817+0000 7f38057fa640 1 -- 172.21.15.87:0/2726618020 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800039040 con 0x7f380810b210 2024-04-22T13:35:58.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.817+0000 7f38057fa640 1 -- 172.21.15.87:0/2726618020 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3800042070 con 0x7f380810b210 2024-04-22T13:35:58.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.817+0000 7f38057fa640 1 -- 172.21.15.87:0/2726618020 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800034af0 con 0x7f380810b210 2024-04-22T13:35:58.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.820+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38081138a0 con 0x7f380810b210 2024-04-22T13:35:58.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.820+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081a36d0 con 0x7f380810b210 2024-04-22T13:35:58.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.827+0000 7f3811230740 1 Processor -- start 2024-04-22T13:35:58.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.829+0000 7f38057fa640 1 -- 172.21.15.87:0/2726618020 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3800034c90 con 0x7f380810b210 2024-04-22T13:35:58.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.830+0000 7f38057fa640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 0x7f37f00a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.831+0000 7f38057fa640 1 -- 172.21.15.87:0/2726618020 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f38000f1dd0 con 0x7f380810b210 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 -- start start 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e4270ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e427104b0 0x555e42718b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555e426700b0 con 0x555e4270fb90 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555e42621680 con 0x555e42719090 2024-04-22T13:35:58.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.835+0000 7f3811230740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555e4262dc60 con 0x555e427104b0 2024-04-22T13:35:58.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.836+0000 7f380cbe5640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 0x7f37f00a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.836+0000 7f380cbe5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.836+0000 7f380cbe5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:37674/0 (socket says 172.21.15.87:37674) 2024-04-22T13:35:58.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.836+0000 7f380cbe5640 1 -- 172.21.15.87:0/607212004 learned_addr learned my addr 172.21.15.87:0/607212004 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f38077fe640 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e427104b0 0x555e42718b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f380cbe5640 1 -- 172.21.15.87:0/607212004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e427104b0 msgr2=0x555e42718b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f380cbe5640 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e427104b0 0x555e42718b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f380cbe5640 1 -- 172.21.15.87:0/607212004 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 msgr2=0x555e4270ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f380cbe5640 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e4270ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.837+0000 7f380cbe5640 1 -- 172.21.15.87:0/607212004 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e42452c90 con 0x555e42719090 2024-04-22T13:35:58.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.838+0000 7f380cbe5640 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f37f8013750 tx=0x7f37f8013c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72e71085747211c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.839+0000 7f37ee7fc640 1 -- 172.21.15.87:0/607212004 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8022070 con 0x555e42719090 2024-04-22T13:35:58.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.839+0000 7f37ee7fc640 1 -- 172.21.15.87:0/607212004 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f8013e00 con 0x555e42719090 2024-04-22T13:35:58.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.839+0000 7f37ee7fc640 1 -- 172.21.15.87:0/607212004 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f800d340 con 0x555e42719090 2024-04-22T13:35:58.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.843+0000 7f380cbe5640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 0x7f37f00a6370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f37f8010fd0 tx=0x7f37f8011f10 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.843+0000 7f3811230740 1 -- 172.21.15.87:0/607212004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 msgr2=0x555e4271b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:35:58.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.843+0000 7f3811230740 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f37f8013750 tx=0x7f37f8013c10 comp rx=0 tx=0).stop 2024-04-22T13:35:58.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.844+0000 7f3811230740 1 -- 172.21.15.87:0/607212004 shutdown_connections 2024-04-22T13:35:58.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.844+0000 7f3811230740 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e4271b480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.844+0000 7f3811230740 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e427104b0 0x555e42718b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.844+0000 7f3811230740 1 --2- 172.21.15.87:0/607212004 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e4270ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.844+0000 7f3811230740 1 -- 172.21.15.87:0/607212004 >> 172.21.15.87:0/607212004 conn(0x555e4270a2c0 msgr2=0x555e4270a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:35:58.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.847+0000 7f3811230740 1 -- 172.21.15.87:0/607212004 shutdown_connections 2024-04-22T13:35:58.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.847+0000 7f3811230740 1 -- 172.21.15.87:0/607212004 wait complete. 2024-04-22T13:35:58.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.848+0000 7f3811230740 1 -- start start 2024-04-22T13:35:58.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.848+0000 7f3811230740 1 Processor -- start 2024-04-22T13:35:58.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e4270fb90 0x555e42716110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e42716b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f38077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f38077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57820/0 (socket says 172.21.15.87:57820) 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f38077fe640 1 -- 172.21.15.87:0/3508718891 learned_addr learned my addr 172.21.15.87:0/3508718891 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555e42673400 con 0x555e427104b0 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555e4262f840 con 0x555e4270fb90 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.849+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555e42678720 con 0x555e42719090 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f3811230740 10 client.0 _ll_register_callbacks cb 0x555e42680ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f38077fe640 1 -- 172.21.15.87:0/3508718891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 msgr2=0x555e42716b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f38077fe640 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e42716b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f38077fe640 1 -- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e4270fb90 msgr2=0x555e42716110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f38077fe640 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e4270fb90 0x555e42716110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:35:58.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.850+0000 7f38077fe640 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e424b3420 con 0x555e427104b0 2024-04-22T13:35:58.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.851+0000 7f38077fe640 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f38000ba350 tx=0x7f38000bdaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ff730c4bb9bcf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:35:58.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.857+0000 7f37e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:35:58.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.857+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000d6020 con 0x555e427104b0 2024-04-22T13:35:58.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.858+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f8013e00 con 0x555e427104b0 2024-04-22T13:35:58.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.858+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f38000d7220 con 0x555e427104b0 2024-04-22T13:35:58.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.873+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f38000be5f0 con 0x555e427104b0 2024-04-22T13:35:58.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.873+0000 7f3811230740 10 client.16845 fetch_fsmap learned FSMap version 23 2024-04-22T13:35:58.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.873+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555e42346580 con 0x555e427104b0 2024-04-22T13:35:58.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f8013e00 con 0x555e427104b0 2024-04-22T13:35:58.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f38000efce0 con 0x555e427104b0 2024-04-22T13:35:58.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 10 client.16845 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:35:58.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 10 client.16845 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:35:58.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 20 client.16845 populate_metadata read hostname 'smithi087' 2024-04-22T13:35:58.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e42728c50 0x555e4272b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:35:58.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 4 client.16845 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:35:58.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.875+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e42681b10 con 0x555e42728c50 2024-04-22T13:35:58.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.876+0000 7f3807fff640 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e42728c50 0x555e4272b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:35:58.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.880+0000 7f37e77fe640 10 client.16845 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:35:58.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:35:58.880+0000 7f3807fff640 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e42728c50 0x555e4272b0c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49613ba7f7a3886e server_cookie=fce12fb2d4843845 in_seq=0 out_seq=0 2024-04-22T13:35:58.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/302805476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:58.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/302805476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:35:59.631 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:35:59.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:35:59.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:35:59.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:35:59 smithi120 ceph-mon[31101]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 960 MiB/s wr, 2.03k op/s 2024-04-22T13:35:59.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:35:59 smithi132 ceph-mon[31531]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 960 MiB/s wr, 2.03k op/s 2024-04-22T13:35:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:35:59 smithi087 ceph-mon[27999]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 960 MiB/s wr, 2.03k op/s 2024-04-22T13:36:00.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.073+0000 7f496beba640 1 -- 172.21.15.87:0/2753368524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4964110550 msgr2=0x7f4964110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:00.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.073+0000 7f4968c54640 1 -- 172.21.15.87:0/2753368524 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c02cb60 con 0x7f4964110550 2024-04-22T13:36:00.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.073+0000 7f496beba640 1 --2- 172.21.15.87:0/2753368524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4964110550 0x7f4964110930 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f495c009be0 tx=0x7f495c02bfc0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.081+0000 7f496beba640 1 -- 172.21.15.87:0/2753368524 shutdown_connections 2024-04-22T13:36:00.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.081+0000 7f496beba640 1 --2- 172.21.15.87:0/2753368524 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f496410eb40 0x7f496411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.081+0000 7f496beba640 1 --2- 172.21.15.87:0/2753368524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410e0c0 0x7f496410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.081+0000 7f496beba640 1 --2- 172.21.15.87:0/2753368524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4964110550 0x7f4964110930 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.081+0000 7f496beba640 1 -- 172.21.15.87:0/2753368524 >> 172.21.15.87:0/2753368524 conn(0x7f496406ce60 msgr2=0x7f496406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:00.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.084+0000 7f496beba640 1 -- 172.21.15.87:0/2753368524 shutdown_connections 2024-04-22T13:36:00.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.085+0000 7f496beba640 1 -- 172.21.15.87:0/2753368524 wait complete. 2024-04-22T13:36:00.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.086+0000 7f496beba640 1 Processor -- start 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.087+0000 7f496beba640 1 -- start start 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 0x7f49641a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4964110550 0x7f49641abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4964120e30 con 0x7f496410e0c0 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4964120cb0 con 0x7f496410eb40 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.088+0000 7f496beba640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4964120fb0 con 0x7f4964110550 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.089+0000 7f4969455640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.089+0000 7f4969455640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:44984/0 (socket says 172.21.15.87:44984) 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.089+0000 7f4969455640 1 -- 172.21.15.87:0/2501928293 learned_addr learned my addr 172.21.15.87:0/2501928293 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.089+0000 7f496a457640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4964110550 0x7f49641abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.091+0000 7f4969455640 1 -- 172.21.15.87:0/2501928293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4964110550 msgr2=0x7f49641abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.092+0000 7f4969455640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4964110550 0x7f49641abc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.092+0000 7f4969455640 1 -- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 msgr2=0x7f49641a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.092+0000 7f4969c56640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 0x7f49641a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.092+0000 7f4969455640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 0x7f49641a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.092+0000 7f4969455640 1 -- 172.21.15.87:0/2501928293 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49641ac350 con 0x7f496410eb40 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.093+0000 7f4969c56640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 0x7f49641a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.093+0000 7f4969455640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f495400e9f0 tx=0x7f495400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.093+0000 7f495affd640 1 -- 172.21.15.87:0/2501928293 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495400ccc0 con 0x7f496410eb40 2024-04-22T13:36:00.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.093+0000 7f495affd640 1 -- 172.21.15.87:0/2501928293 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495400ce60 con 0x7f496410eb40 2024-04-22T13:36:00.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.094+0000 7f495affd640 1 -- 172.21.15.87:0/2501928293 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954010430 con 0x7f496410eb40 2024-04-22T13:36:00.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.094+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49641ac640 con 0x7f496410eb40 2024-04-22T13:36:00.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.094+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4964110f10 con 0x7f496410eb40 2024-04-22T13:36:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.100+0000 7f496eb84740 1 Processor -- start 2024-04-22T13:36:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.100+0000 7f496eb84740 1 -- start start 2024-04-22T13:36:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8feae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564b8fe09f00 con 0x564b8fea5a80 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b8fda7590 con 0x564b8fea6430 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.101+0000 7f496eb84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564b8fe05ec0 con 0x564b8feaefb0 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.104+0000 7f4969455640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.104+0000 7f4969455640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:47282/0 (socket says 172.21.15.87:47282) 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.104+0000 7f4969455640 1 -- 172.21.15.87:0/894323930 learned_addr learned my addr 172.21.15.87:0/894323930 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.105+0000 7f4969455640 1 -- 172.21.15.87:0/894323930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 msgr2=0x564b8feae9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.105+0000 7f4969455640 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8feae9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.105+0000 7f4969455640 1 -- 172.21.15.87:0/894323930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 msgr2=0x564b8fea5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.105+0000 7f4969455640 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.105+0000 7f4969455640 1 -- 172.21.15.87:0/894323930 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8fbe47e0 con 0x564b8feaefb0 2024-04-22T13:36:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.106+0000 7f4969455640 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f49540ce0d0 tx=0x7f49540d4a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38e806ac2e7cf109 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.106+0000 7f495affd640 1 -- 172.21.15.87:0/2501928293 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f49540046c0 con 0x7f496410eb40 2024-04-22T13:36:00.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.108+0000 7f495affd640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 0x7f49500a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.109+0000 7f495affd640 1 -- 172.21.15.87:0/2501928293 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f49540d0a30 con 0x7f496410eb40 2024-04-22T13:36:00.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.109+0000 7f496a457640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 0x7f49500a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.111+0000 7f496a457640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 0x7f49500a63e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4960009c20 tx=0x7f49600097c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:00.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.111+0000 7f494ffff640 1 -- 172.21.15.87:0/894323930 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540e4070 con 0x564b8feaefb0 2024-04-22T13:36:00.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.111+0000 7f494ffff640 1 -- 172.21.15.87:0/894323930 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49540d9b20 con 0x564b8feaefb0 2024-04-22T13:36:00.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.113+0000 7f496eb84740 1 -- 172.21.15.87:0/894323930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 msgr2=0x564b8feb1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:00.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.113+0000 7f496eb84740 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f49540ce0d0 tx=0x7f49540d4a90 comp rx=0 tx=0).stop 2024-04-22T13:36:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.117+0000 7f496eb84740 1 -- 172.21.15.87:0/894323930 shutdown_connections 2024-04-22T13:36:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.117+0000 7f496eb84740 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8feb1400 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.117+0000 7f496eb84740 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8feae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.117+0000 7f496eb84740 1 --2- 172.21.15.87:0/894323930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.117+0000 7f496eb84740 1 -- 172.21.15.87:0/894323930 >> 172.21.15.87:0/894323930 conn(0x564b8fea0150 msgr2=0x564b8fea0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.118+0000 7f496eb84740 1 -- 172.21.15.87:0/894323930 shutdown_connections 2024-04-22T13:36:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.118+0000 7f496eb84740 1 -- 172.21.15.87:0/894323930 wait complete. 2024-04-22T13:36:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.119+0000 7f496eb84740 1 -- start start 2024-04-22T13:36:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.119+0000 7f496eb84740 1 Processor -- start 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea5a80 0x564b8feabfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea6430 0x564b8feac510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564b8feb1f70 con 0x564b8feaefb0 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b8fdc3580 con 0x564b8fea5a80 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564b8feb20f0 con 0x564b8fea6430 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.120+0000 7f496eb84740 10 client.0 _ll_register_callbacks cb 0x564b8fdc9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51144/0 (socket says 172.21.15.87:51144) 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 -- 172.21.15.87:0/2067425033 learned_addr learned my addr 172.21.15.87:0/2067425033 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 -- 172.21.15.87:0/2067425033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea6430 msgr2=0x564b8feac510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea6430 0x564b8feac510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.121+0000 7f496a457640 1 -- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea5a80 msgr2=0x564b8feabfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.122+0000 7f496a457640 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea5a80 0x564b8feabfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:00.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.122+0000 7f496a457640 1 -- 172.21.15.87:0/2067425033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8fc44f70 con 0x564b8feaefb0 2024-04-22T13:36:00.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.122+0000 7f496a457640 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4960009670 tx=0x7f4960005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9d7fa8e7defd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:00.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.122+0000 7f494d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:00.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.123+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496000fb40 con 0x564b8feaefb0 2024-04-22T13:36:00.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.123+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564b8fe17920 con 0x564b8feaefb0 2024-04-22T13:36:00.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.123+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f496000fce0 con 0x564b8feaefb0 2024-04-22T13:36:00.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.124+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f496001ce50 con 0x564b8feaefb0 2024-04-22T13:36:00.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.126+0000 7f496eb84740 10 client.16851 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:00.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.126+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564b8fadc170 con 0x564b8feaefb0 2024-04-22T13:36:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.127+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564b8fe17920 con 0x564b8feaefb0 2024-04-22T13:36:00.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.128+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4960030050 con 0x564b8feaefb0 2024-04-22T13:36:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.129+0000 7f496eb84740 10 client.16851 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f496eb84740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f496eb84740 20 client.16851 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:00.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8febf720 0x564b8fec1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f496eb84740 4 client.16851 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b8fe11790 con 0x564b8febf720 2024-04-22T13:36:00.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.130+0000 7f4969c56640 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8febf720 0x564b8fec1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.138+0000 7f4969c56640 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8febf720 0x564b8fec1bb0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=563c97f9ef4a3b52 server_cookie=304d469301745456 in_seq=0 out_seq=0 2024-04-22T13:36:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:00.143+0000 7f494d7fa640 10 client.16851 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:01.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:01 smithi120 ceph-mon[31101]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 861 MiB/s wr, 1.94k op/s 2024-04-22T13:36:01.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:01 smithi132 ceph-mon[31531]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 861 MiB/s wr, 1.94k op/s 2024-04-22T13:36:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:01 smithi087 ceph-mon[27999]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 861 MiB/s wr, 1.94k op/s 2024-04-22T13:36:03.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:03 smithi120 ceph-mon[31101]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.59k op/s 2024-04-22T13:36:03.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:03 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:36:03.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:03 smithi132 ceph-mon[31531]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.59k op/s 2024-04-22T13:36:03.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:03 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:36:03.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:03 smithi087 ceph-mon[27999]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.59k op/s 2024-04-22T13:36:03.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:03 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:36:05.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:05 smithi120 ceph-mon[31101]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 993 MiB/s wr, 2.05k op/s 2024-04-22T13:36:05.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:05 smithi132 ceph-mon[31531]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 993 MiB/s wr, 2.05k op/s 2024-04-22T13:36:05.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:05 smithi087 ceph-mon[27999]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 993 MiB/s wr, 2.05k op/s 2024-04-22T13:36:06.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:36:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:36:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:36:06.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:36:06.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:36:06.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:36:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:36:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:36:06.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:36:07.198 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.200+0000 7f1082ff5640 1 -- 172.21.15.87:0/3727278424 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f109405a040 con 0x56012c0fbd90 2024-04-22T13:36:07.198 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.200+0000 7f1082ff5640 10 client.16836 handle_command_reply: tid=0 2024-04-22T13:36:07.198 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.200+0000 7f10a7dab740 1 client.16836 shutdown 2024-04-22T13:36:07.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f494d7fa640 1 -- 172.21.15.87:0/2067425033 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b8fe11790 con 0x564b8febf720 2024-04-22T13:36:07.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f494d7fa640 10 client.16851 handle_command_reply: tid=0 2024-04-22T13:36:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 msgr2=0x56012c07d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07d0d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1090014ed0 tx=0x7f10900126f0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f496eb84740 1 client.16851 shutdown 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 shutdown_connections 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56012c0fbd90 0x56012c0fe1b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56012c0840e0 0x56012c07d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56012c07b8d0 0x56012c07d0d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 --2- 172.21.15.87:0/3727278424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56012c07ab60 0x56012c08c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 >> 172.21.15.87:0/3727278424 conn(0x56012c041c90 msgr2=0x56012c074040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f1082ff5640 0 client.16836 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:07.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f37e77fe640 1 -- 172.21.15.87:0/3508718891 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e42681b10 con 0x555e42728c50 2024-04-22T13:36:07.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.201+0000 7f37e77fe640 10 client.16845 handle_command_reply: tid=0 2024-04-22T13:36:07.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 client.16845 shutdown 2024-04-22T13:36:07.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 shutdown_connections 2024-04-22T13:36:07.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f10a7dab740 1 -- 172.21.15.87:0/3727278424 wait complete. 2024-04-22T13:36:07.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f10a7dab740 20 client.16836 trim_cache size 0 max 16384 2024-04-22T13:36:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 msgr2=0x564b8feaca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4960009670 tx=0x7f4960005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 shutdown_connections 2024-04-22T13:36:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8febf720 0x564b8fec1bb0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8feaefb0 0x564b8feaca50 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea6430 0x564b8feac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 --2- 172.21.15.87:0/2067425033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea5a80 0x564b8feabfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 >> 172.21.15.87:0/2067425033 conn(0x564b8fea0150 msgr2=0x564b8fea1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f494d7fa640 0 client.16851 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 shutdown_connections 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f496eb84740 1 -- 172.21.15.87:0/2067425033 wait complete. 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 msgr2=0x555e42716650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f38000ba350 tx=0x7f38000bdaf0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 shutdown_connections 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e42728c50 0x555e4272b0c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e42716b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42716650 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 --2- 172.21.15.87:0/3508718891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e4270fb90 0x555e42716110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.202+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 >> 172.21.15.87:0/3508718891 conn(0x555e4270a2c0 msgr2=0x555e4270c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:36:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:36:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:36:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:36:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f496eb84740 20 client.16851 trim_cache size 0 max 16384 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 msgr2=0x7f10880a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 0x7f10880a63d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f10980dfb80 tx=0x7f10980b1630 comp rx=0 tx=0).stop 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 msgr2=0x7f10a01178b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f109804f9e0 tx=0x7f109804fea0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 shutdown_connections 2024-04-22T13:36:07.211 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f10880a3f10 0x7f10880a63d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f10a0117400 0x7f10a01178b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10a011b730 0x7f10a0116ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 --2- 172.21.15.87:0/3990822275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10a0110740 0x7f10a011ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 >> 172.21.15.87:0/3990822275 conn(0x7f10a006d020 msgr2=0x7f10a010d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.204+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 shutdown_connections 2024-04-22T13:36:07.212 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:07.204+0000 7f10a51b6640 1 -- 172.21.15.87:0/3990822275 wait complete. 2024-04-22T13:36:07.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f37e77fe640 0 client.16845 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:07.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.203+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 shutdown_connections 2024-04-22T13:36:07.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.204+0000 7f3811230740 1 -- 172.21.15.87:0/3508718891 wait complete. 2024-04-22T13:36:07.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.204+0000 7f3811230740 20 client.16845 trim_cache size 0 max 16384 2024-04-22T13:36:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 Processor -- start 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 -- start start 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea5a80 0x564b8feae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8feaefb0 0x564b8fe9efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f49641c7940 con 0x564b8fea6430 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49641c5210 con 0x564b8feaefb0 2024-04-22T13:36:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.252+0000 7f496eb84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4964068cd0 con 0x564b8fea5a80 2024-04-22T13:36:07.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.256+0000 7f4969455640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.256+0000 7f4969455640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51152/0 (socket says 172.21.15.87:51152) 2024-04-22T13:36:07.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.256+0000 7f4969455640 1 -- 172.21.15.87:0/129923658 learned_addr learned my addr 172.21.15.87:0/129923658 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:07.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.257+0000 7f4969c56640 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea5a80 0x564b8feae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f496a457640 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8feaefb0 0x564b8fe9efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f4969455640 1 -- 172.21.15.87:0/129923658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea5a80 msgr2=0x564b8feae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f4969455640 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea5a80 0x564b8feae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f4969455640 1 -- 172.21.15.87:0/129923658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8feaefb0 msgr2=0x564b8fe9efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f4969455640 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8feaefb0 0x564b8fe9efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.258+0000 7f4969455640 1 -- 172.21.15.87:0/129923658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8febe2c0 con 0x564b8fea6430 2024-04-22T13:36:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.261+0000 7f4969455640 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4954098e40 tx=0x7f49540d44a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c747f1a86b78711c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.265+0000 7f494f7fe640 1 -- 172.21.15.87:0/129923658 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954098c80 con 0x564b8fea6430 2024-04-22T13:36:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.266+0000 7f494f7fe640 1 -- 172.21.15.87:0/129923658 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495409d0e0 con 0x564b8fea6430 2024-04-22T13:36:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.267+0000 7f496eb84740 1 -- 172.21.15.87:0/129923658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 msgr2=0x564b8febae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.267+0000 7f496eb84740 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4954098e40 tx=0x7f49540d44a0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.268+0000 7f496eb84740 1 -- 172.21.15.87:0/129923658 shutdown_connections 2024-04-22T13:36:07.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.268+0000 7f496eb84740 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8feaefb0 0x564b8fe9efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.268+0000 7f496eb84740 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea6430 0x564b8febae80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.268+0000 7f496eb84740 1 --2- 172.21.15.87:0/129923658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8fea5a80 0x564b8feae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.268+0000 7f496eb84740 1 -- 172.21.15.87:0/129923658 >> 172.21.15.87:0/129923658 conn(0x564b8fea0150 msgr2=0x564b8fe9e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.269+0000 7f3811230740 1 Processor -- start 2024-04-22T13:36:07.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.269+0000 7f3811230740 1 -- start start 2024-04-22T13:36:07.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.270+0000 7f3811230740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e4270fb90 0x555e42718840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.270+0000 7f3811230740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f38077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f3811230740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e42724a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f3811230740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c5280 con 0x555e427104b0 2024-04-22T13:36:07.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f3811230740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c30d0 con 0x555e42719090 2024-04-22T13:36:07.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f3811230740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c4bb0 con 0x555e4270fb90 2024-04-22T13:36:07.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f38077fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51156/0 (socket says 172.21.15.87:51156) 2024-04-22T13:36:07.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.271+0000 7f38077fe640 1 -- 172.21.15.87:0/3847807171 learned_addr learned my addr 172.21.15.87:0/3847807171 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f496eb84740 1 -- 172.21.15.87:0/129923658 shutdown_connections 2024-04-22T13:36:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f496eb84740 1 -- 172.21.15.87:0/129923658 wait complete. 2024-04-22T13:36:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f496eb84740 1 -- start start 2024-04-22T13:36:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f496eb84740 1 Processor -- start 2024-04-22T13:36:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f38077fe640 1 -- 172.21.15.87:0/3847807171 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e4270fb90 msgr2=0x555e42718840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f38077fe640 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e4270fb90 0x555e42718840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f38077fe640 1 -- 172.21.15.87:0/3847807171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 msgr2=0x555e42724a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f38077fe640 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e42724a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.272+0000 7f38077fe640 1 -- 172.21.15.87:0/3847807171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e427150c0 con 0x555e427104b0 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8fea91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8fea9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4964117ee0 con 0x564b8fea5a80 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4964119350 con 0x564b8fea6430 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f49641185b0 con 0x564b8feaefb0 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.273+0000 7f496eb84740 10 client.0 _ll_register_callbacks cb 0x7f4960009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.274+0000 7f4969c56640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.274+0000 7f4969c56640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51170/0 (socket says 172.21.15.87:51170) 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.274+0000 7f4969c56640 1 -- 172.21.15.87:0/1901323611 learned_addr learned my addr 172.21.15.87:0/1901323611 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:07.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.274+0000 7f38077fe640 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f38000bdba0 tx=0x7f380003f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506ea400dde94464 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.275+0000 7f496a457640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8fea9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.274+0000 7f37edffb640 1 -- 172.21.15.87:0/3847807171 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000d6070 con 0x555e427104b0 2024-04-22T13:36:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.275+0000 7f37edffb640 1 -- 172.21.15.87:0/3847807171 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3800032930 con 0x555e427104b0 2024-04-22T13:36:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.275+0000 7f37edffb640 1 -- 172.21.15.87:0/3847807171 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000bdef0 con 0x555e427104b0 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f4969455640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8fea91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f496a457640 1 -- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 msgr2=0x564b8fea91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f496a457640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8fea91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f496a457640 1 -- 172.21.15.87:0/1901323611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 msgr2=0x564b8fea8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f496a457640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f496a457640 1 -- 172.21.15.87:0/1901323611 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8ff24cf0 con 0x564b8feaefb0 2024-04-22T13:36:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.276+0000 7f3811230740 1 -- 172.21.15.87:0/3847807171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 msgr2=0x555e42705230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.277+0000 7f3811230740 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f38000bdba0 tx=0x7f380003f070 comp rx=0 tx=0).stop 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f3811230740 1 -- 172.21.15.87:0/3847807171 shutdown_connections 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f3811230740 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e42719090 0x555e42724a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f3811230740 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e427104b0 0x555e42705230 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f3811230740 1 --2- 172.21.15.87:0/3847807171 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e4270fb90 0x555e42718840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f3811230740 1 -- 172.21.15.87:0/3847807171 >> 172.21.15.87:0/3847807171 conn(0x555e4270a2c0 msgr2=0x555e4270bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f496a457640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8fea9700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4960009ae0 tx=0x7f4960005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b76faa172a6cda9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.278+0000 7f494d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:36:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.279+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49600316b0 con 0x564b8feaefb0 2024-04-22T13:36:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.279+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564b8fd8dc50 con 0x564b8feaefb0 2024-04-22T13:36:07.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.279+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f4960031850 con 0x564b8feaefb0 2024-04-22T13:36:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.281+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4960031b50 con 0x564b8feaefb0 2024-04-22T13:36:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.282+0000 7f3811230740 1 -- 172.21.15.87:0/3847807171 shutdown_connections 2024-04-22T13:36:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.282+0000 7f3811230740 1 -- 172.21.15.87:0/3847807171 wait complete. 2024-04-22T13:36:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.282+0000 7f496eb84740 10 client.26035 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.282+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564b8ff26070 con 0x564b8feaefb0 2024-04-22T13:36:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.283+0000 7f3811230740 1 -- start start 2024-04-22T13:36:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.283+0000 7f3811230740 1 Processor -- start 2024-04-22T13:36:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.284+0000 7f3811230740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e427134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.284+0000 7f3811230740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.284+0000 7f3811230740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e4278e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f3811230740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c3e10 con 0x555e4270fb90 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f3811230740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c3740 con 0x555e427104b0 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f3811230740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38081c44e0 con 0x555e42719090 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f38077fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f38077fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:45040/0 (socket says 172.21.15.87:45040) 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f38077fe640 1 -- 172.21.15.87:0/307029464 learned_addr learned my addr 172.21.15.87:0/307029464 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f380cbe5640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e4278e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.285+0000 7f3811230740 10 client.0 _ll_register_callbacks cb 0x7f3800009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f38077fe640 1 -- 172.21.15.87:0/307029464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 msgr2=0x555e4278e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f38077fe640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e4278e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f38077fe640 1 -- 172.21.15.87:0/307029464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 msgr2=0x555e427134f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f38077fe640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e427134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f38077fe640 1 -- 172.21.15.87:0/307029464 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4278e780 con 0x555e427104b0 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564b8fd8dc50 con 0x564b8feaefb0 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4960030040 con 0x564b8feaefb0 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 10 client.26035 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 10 client.26035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 20 client.26035 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:07.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8ff26740 0x564b8ff28b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 4 client.26035 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:36:07.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.286+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564b8fdda720 con 0x564b8ff26740 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.287+0000 7f38077fe640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38000efb90 tx=0x7f38000bdaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc932e1dd32a912 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.287+0000 7f37e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.291+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000d6070 con 0x555e427104b0 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.291+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f38000c6900 con 0x555e427104b0 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.291+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3800032930 con 0x555e427104b0 2024-04-22T13:36:07.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.292+0000 7f4969c56640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8ff26740 0x564b8ff28b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.294+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f38000a4050 con 0x555e427104b0 2024-04-22T13:36:07.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.294+0000 7f3811230740 10 client.25994 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:07.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.294+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555e4278e780 con 0x555e427104b0 2024-04-22T13:36:07.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.297+0000 7f4969c56640 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8ff26740 0x564b8ff28b30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b8d3458d89c634b server_cookie=443c1ccdfc1f0a23 in_seq=0 out_seq=0 2024-04-22T13:36:07.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.297+0000 7f494d7fa640 10 client.26035 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:07.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38000bdef0 con 0x555e427104b0 2024-04-22T13:36:07.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f38000cd070 con 0x555e427104b0 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 10 client.25994 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 10 client.25994 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 20 client.25994 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e4278fce0 0x555e427920d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 4 client.25994 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}] 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.298+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}) -- 0x7f3800032930 con 0x555e4278fce0 2024-04-22T13:36:07.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.301+0000 7f3807fff640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e4278fce0 0x555e427920d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:07.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.307+0000 7f3807fff640 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e4278fce0 0x555e427920d0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a57a03ef454b2703 server_cookie=b2e8ac984796b24a in_seq=0 out_seq=0 2024-04-22T13:36:07.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:07.308+0000 7f37e77fe640 10 client.25994 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:07.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:07 smithi120 ceph-mon[31101]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 773 MiB/s wr, 1.47k op/s 2024-04-22T13:36:07.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:07 smithi132 ceph-mon[31531]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 773 MiB/s wr, 1.47k op/s 2024-04-22T13:36:07.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:07 smithi087 ceph-mon[27999]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 773 MiB/s wr, 1.47k op/s 2024-04-22T13:36:08.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f494d7fa640 1 -- 172.21.15.87:0/1901323611 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564b8fdda720 con 0x564b8ff26740 2024-04-22T13:36:08.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f494d7fa640 10 client.26035 handle_command_reply: tid=0 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 client.26035 shutdown 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 msgr2=0x564b8fea9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8fea9700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4960009ae0 tx=0x7f4960005700 comp rx=0 tx=0).stop 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 shutdown_connections 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564b8ff26740 0x564b8ff28b30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564b8feaefb0 0x564b8fea9700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8fea6430 0x564b8fea91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 --2- 172.21.15.87:0/1901323611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564b8fea5a80 0x564b8fea8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 >> 172.21.15.87:0/1901323611 conn(0x564b8fea0150 msgr2=0x564b8feac820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f494d7fa640 0 client.26035 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 shutdown_connections 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 1 -- 172.21.15.87:0/1901323611 wait complete. 2024-04-22T13:36:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.140+0000 7f496eb84740 20 client.26035 trim_cache size 0 max 16384 2024-04-22T13:36:08.138 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:08.138 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:36:08.138 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "ca0ba2a7-562e-4372-9a9a-fb7271f29f8b", 2024-04-22T13:36:08.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:36:08.138 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 msgr2=0x7f49500a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 0x7f49500a63e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4960009c20 tx=0x7f49600097c0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 msgr2=0x7f49641a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f495400e9f0 tx=0x7f495400eeb0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 shutdown_connections 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f49500a3f20 0x7f49500a63e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4964110550 0x7f49641abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f496410eb40 0x7f49641a78d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 --2- 172.21.15.87:0/2501928293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f496410e0c0 0x7f49641a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 >> 172.21.15.87:0/2501928293 conn(0x7f496406ce60 msgr2=0x7f4964071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 shutdown_connections 2024-04-22T13:36:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.141+0000 7f496beba640 1 -- 172.21.15.87:0/2501928293 wait complete. 2024-04-22T13:36:08.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:36:08.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:36:08.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 -- 172.21.15.87:0/3302215143 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9681010a0 msgr2=0x7fd968101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/3302215143 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9681010a0 0x7fd968101500 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd950009900 tx=0x7fd95002bc30 comp rx=0 tx=0).stop 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 -- 172.21.15.87:0/3302215143 shutdown_connections 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/3302215143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd968101a40 0x7fd96810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/3302215143 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9681010a0 0x7fd968101500 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/3302215143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 0x7fd9681061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 -- 172.21.15.87:0/3302215143 >> 172.21.15.87:0/3302215143 conn(0x7fd968077c40 msgr2=0x7fd968078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 -- 172.21.15.87:0/3302215143 shutdown_connections 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.259+0000 7fd96c9d5640 1 -- 172.21.15.87:0/3302215143 wait complete. 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 Processor -- start 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 -- start start 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 0x7fd9681976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 0x7fd96819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd968111170 con 0x7fd968101a40 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd968110ff0 con 0x7fd9681010a0 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96c9d5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681112f0 con 0x7fd968105dd0 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51188/0 (socket says 172.21.15.87:51188) 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 0x7fd9681976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 -- 172.21.15.87:0/464504419 learned_addr learned my addr 172.21.15.87:0/464504419 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd966d9d640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 0x7fd96819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 -- 172.21.15.87:0/464504419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 msgr2=0x7fd96819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 0x7fd96819bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 -- 172.21.15.87:0/464504419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 msgr2=0x7fd9681976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 0x7fd9681976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd95ffff640 1 -- 172.21.15.87:0/464504419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96819c6e0 con 0x7fd968101a40 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.260+0000 7fd96659c640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 0x7fd9681976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:08.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd95ffff640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd950005940 tx=0x7fd95002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd95f7fe640 1 -- 172.21.15.87:0/464504419 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950043020 con 0x7fd968101a40 2024-04-22T13:36:08.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd95f7fe640 1 -- 172.21.15.87:0/464504419 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd95002ca70 con 0x7fd968101a40 2024-04-22T13:36:08.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd95f7fe640 1 -- 172.21.15.87:0/464504419 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9500039f0 con 0x7fd968101a40 2024-04-22T13:36:08.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96819c970 con 0x7fd968101a40 2024-04-22T13:36:08.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.261+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9681a4260 con 0x7fd968101a40 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.263+0000 7fd95f7fe640 1 -- 172.21.15.87:0/464504419 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd950046020 con 0x7fd968101a40 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd95f7fe640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 Processor -- start 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96659c640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd95f7fe640 1 -- 172.21.15.87:0/464504419 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd9500f2680 con 0x7fd968101a40 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 -- start start 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96659c640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 0x7fd9540a6260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd958009fc0 tx=0x7fd958005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b621540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b621a80 0x55fe3b623e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b57d1e0 con 0x55fe3b618580 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b51a390 con 0x55fe3b621a80 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b5791a0 con 0x55fe3b618ea0 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51192/0 (socket says 172.21.15.87:51192) 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd96659c640 1 -- 172.21.15.87:0/1610064108 learned_addr learned my addr 172.21.15.87:0/1610064108 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd966d9d640 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b621a80 0x55fe3b623e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.264+0000 7fd95ffff640 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b621540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 -- 172.21.15.87:0/1610064108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 msgr2=0x55fe3b621540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b621540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 -- 172.21.15.87:0/1610064108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b621a80 msgr2=0x55fe3b623e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b621a80 0x55fe3b623e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 -- 172.21.15.87:0/1610064108 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3b3597d0 con 0x55fe3b618580 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd96659c640 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd958009ba0 tx=0x7fd958014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eec3ff355047842 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd95cff9640 1 -- 172.21.15.87:0/1610064108 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958026480 con 0x55fe3b618580 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd95cff9640 1 -- 172.21.15.87:0/1610064108 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd958026ad0 con 0x55fe3b618580 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.265+0000 7fd95cff9640 1 -- 172.21.15.87:0/1610064108 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95800fbe0 con 0x55fe3b618580 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1610064108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 msgr2=0x55fe3b618960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd958009ba0 tx=0x7fd958014000 comp rx=0 tx=0).stop 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1610064108 shutdown_connections 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b621a80 0x55fe3b623e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b621540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1610064108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b618960 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1610064108 >> 172.21.15.87:0/1610064108 conn(0x55fe3b612cb0 msgr2=0x55fe3b613090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1610064108 shutdown_connections 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1610064108 wait complete. 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- start start 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 Processor -- start 2024-04-22T13:36:08.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b61eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b61f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b624870 con 0x55fe3b621a80 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b5362f0 con 0x55fe3b618580 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.266+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe3b6249f0 con 0x55fe3b618ea0 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd96f5ca740 10 client.0 _ll_register_callbacks cb 0x55fe3b543580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b61f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd96659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b61eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51196/0 (socket says 172.21.15.87:51196) 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 -- 172.21.15.87:0/1601848735 learned_addr learned my addr 172.21.15.87:0/1601848735 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 -- 172.21.15.87:0/1601848735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 msgr2=0x55fe3b61f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b61f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 -- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 msgr2=0x55fe3b61eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b61eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 -- 172.21.15.87:0/1601848735 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3b28f3f0 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd966d9d640 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd960011d10 tx=0x7fd96000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291ff355bcc1c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd94dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96000fe60 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd960004b20 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd958026ad0 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd960004d60 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd96f5ca740 10 client.16872 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe3b24f0f0 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.267+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd958026ad0 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd960019030 con 0x55fe3b621a80 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 10 client.16872 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:08.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 20 client.16872 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b631b80 0x55fe3b634000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 4 client.16872 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe3b58ac00 con 0x55fe3b631b80 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96659c640 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b631b80 0x55fe3b634000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd96659c640 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b631b80 0x55fe3b634000 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bee9181ef6bba6fa server_cookie=8749c1c0d979d975 in_seq=0 out_seq=0 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.268+0000 7fd94dffb640 10 client.16872 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.269+0000 7fd94dffb640 1 -- 172.21.15.87:0/1601848735 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe3b58ac00 con 0x55fe3b631b80 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.269+0000 7fd94dffb640 10 client.16872 handle_command_reply: tid=0 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 client.16872 shutdown 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 msgr2=0x55fe3b61f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd960011d10 tx=0x7fd96000f790 comp rx=0 tx=0).stop 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 shutdown_connections 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b631b80 0x55fe3b634000 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b621a80 0x55fe3b61f590 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b618ea0 0x55fe3b61f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd94dffb640 0 client.16872 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1601848735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b61eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 >> 172.21.15.87:0/1601848735 conn(0x55fe3b612cb0 msgr2=0x55fe3b614a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 shutdown_connections 2024-04-22T13:36:08.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1601848735 wait complete. 2024-04-22T13:36:08.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.270+0000 7fd96f5ca740 20 client.16872 trim_cache size 0 max 16384 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 Processor -- start 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 -- start start 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 0x55fe3b62d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681af190 con 0x55fe3b618580 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681aca60 con 0x55fe3b618ea0 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681aa8b0 con 0x55fe3b621a80 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51202/0 (socket says 172.21.15.87:51202) 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 -- 172.21.15.87:0/1587752326 learned_addr learned my addr 172.21.15.87:0/1587752326 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd95ffff640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 0x55fe3b62d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd966d9d640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 -- 172.21.15.87:0/1587752326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 msgr2=0x55fe3b61dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 -- 172.21.15.87:0/1587752326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 msgr2=0x55fe3b62d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 0x55fe3b62d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 -- 172.21.15.87:0/1587752326 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3b618cd0 con 0x55fe3b618580 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd95ffff640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 0x55fe3b62d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.288+0000 7fd96659c640 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd958003e50 tx=0x7fd95800f800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5a2402267f8743 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd94ffff640 1 -- 172.21.15.87:0/1587752326 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958037030 con 0x55fe3b618580 2024-04-22T13:36:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd94ffff640 1 -- 172.21.15.87:0/1587752326 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9580054f0 con 0x55fe3b618580 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd94ffff640 1 -- 172.21.15.87:0/1587752326 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9580266e0 con 0x55fe3b618580 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1587752326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 msgr2=0x55fe3b621240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd958003e50 tx=0x7fd95800f800 comp rx=0 tx=0).stop 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1587752326 shutdown_connections 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618ea0 0x55fe3b62d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/1587752326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618580 0x55fe3b621240 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1587752326 >> 172.21.15.87:0/1587752326 conn(0x55fe3b612cb0 msgr2=0x55fe3b61ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1587752326 shutdown_connections 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.289+0000 7fd96f5ca740 1 -- 172.21.15.87:0/1587752326 wait complete. 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 -- start start 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 Processor -- start 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b629fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681aaf20 con 0x55fe3b618ea0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681ac390 con 0x55fe3b618580 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681ab5f0 con 0x55fe3b621a80 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96f5ca740 10 client.0 _ll_register_callbacks cb 0x7fd960009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51208/0 (socket says 172.21.15.87:51208) 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 -- 172.21.15.87:0/3310313603 learned_addr learned my addr 172.21.15.87:0/3310313603 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd96659c640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b629fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 -- 172.21.15.87:0/3310313603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 msgr2=0x55fe3b61b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 -- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 msgr2=0x55fe3b629fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b629fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 -- 172.21.15.87:0/3310313603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3b61c1b0 con 0x55fe3b618ea0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd95ffff640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd950000c00 tx=0x7fd950009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bce762b502f0e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.290+0000 7fd94dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9500be740 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fd9500c6f20 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9580054f0 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd9500a5050 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 10 client.16878 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe3b69c550 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9580054f0 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd950003cb0 con 0x55fe3b618ea0 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 10 client.16878 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 20 client.16878 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b69cd50 0x55fe3b69f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 4 client.16878 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe3b584a70 con 0x55fe3b69cd50 2024-04-22T13:36:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.291+0000 7fd96659c640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b69cd50 0x55fe3b69f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96659c640 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b69cd50 0x55fe3b69f170 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8fc4d1374630342 server_cookie=a50a62960575f7d7 in_seq=0 out_seq=0 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd94dffb640 10 client.16878 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd94dffb640 1 -- 172.21.15.87:0/3310313603 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55fe3b584a70 con 0x55fe3b69cd50 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd94dffb640 10 client.16878 handle_command_reply: tid=0 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 client.16878 shutdown 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 msgr2=0x55fe3b61b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd950000c00 tx=0x7fd950009560 comp rx=0 tx=0).stop 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 shutdown_connections 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fe3b69cd50 0x55fe3b69f170 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe3b621a80 0x55fe3b61b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe3b618ea0 0x55fe3b61b440 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 --2- 172.21.15.87:0/3310313603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe3b618580 0x55fe3b629fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd94dffb640 0 client.16878 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 >> 172.21.15.87:0/3310313603 conn(0x55fe3b612cb0 msgr2=0x55fe3b6114e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 shutdown_connections 2024-04-22T13:36:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.292+0000 7fd96f5ca740 1 -- 172.21.15.87:0/3310313603 wait complete. 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96f5ca740 20 client.16878 trim_cache size 0 max 16384 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "scrub active (3 inodes in the stack)", 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "ca0ba2a7-562e-4372-9a9a-fb7271f29f8b": { 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "ca0ba2a7-562e-4372-9a9a-fb7271f29f8b", 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.294 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.295 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:08.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 msgr2=0x7fd9540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 0x7fd9540a6260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd958009fc0 tx=0x7fd958005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:08.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 msgr2=0x7fd968197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd950005940 tx=0x7fd95002c5f0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 shutdown_connections 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd968105dd0 0x7fd96819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd968101a40 0x7fd968197c30 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 --2- 172.21.15.87:0/464504419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9681010a0 0x7fd9681976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 >> 172.21.15.87:0/464504419 conn(0x7fd968077c40 msgr2=0x7fd9680fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 shutdown_connections 2024-04-22T13:36:08.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.293+0000 7fd96c9d5640 1 -- 172.21.15.87:0/464504419 wait complete. 2024-04-22T13:36:08.300 INFO:tasks.cephfs.filesystem:scrub status for tag:ca0ba2a7-562e-4372-9a9a-fb7271f29f8b - {'path': '/', 'tag': 'ca0ba2a7-562e-4372-9a9a-fb7271f29f8b', 'options': 'recursive,force'} 2024-04-22T13:36:08.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f37e77fe640 1 -- 172.21.15.87:0/307029464 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f3800032930 con 0x555e4278fce0 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f37e77fe640 10 client.25994 handle_command_reply: tid=0 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 client.25994 shutdown 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 msgr2=0x555e42713a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38000efb90 tx=0x7f38000bdaf0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 shutdown_connections 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555e4278fce0 0x555e427920d0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555e42719090 0x555e4278e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f37e77fe640 0 client.25994 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e427104b0 0x555e42713a30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 --2- 172.21.15.87:0/307029464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555e4270fb90 0x555e427134f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.404+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 >> 172.21.15.87:0/307029464 conn(0x555e4270a2c0 msgr2=0x555e42708b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 shutdown_connections 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f3811230740 1 -- 172.21.15.87:0/307029464 wait complete. 2024-04-22T13:36:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f3811230740 20 client.25994 trim_cache size 0 max 16384 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 18, 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "7503ae6d": { 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 18, 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:08.402 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 90.0, 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.403 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.404 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:08.404 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:08.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 msgr2=0x7f37f00a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 0x7f37f00a6370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f37f8010fd0 tx=0x7f37f8011f10 comp rx=0 tx=0).stop 2024-04-22T13:36:08.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 msgr2=0x7f38081a7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:08.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 0x7f38081a7f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3800009bf0 tx=0x7f3800003f10 comp rx=0 tx=0).stop 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 shutdown_connections 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37f00a3eb0 0x7f37f00a6370 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3808113ea0 0x7f3808112d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f380810b210 0x7f38081a7f90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 --2- 172.21.15.87:0/2726618020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380810a8f0 0x7f38081a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 >> 172.21.15.87:0/2726618020 conn(0x7f380806ce60 msgr2=0x7f38080708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 shutdown_connections 2024-04-22T13:36:08.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:08.405+0000 7f380d3e6640 1 -- 172.21.15.87:0/2726618020 wait complete. 2024-04-22T13:36:08.412 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 7503ae6d, quiesce took 11.8 seconds. Will release after: 30.0 2024-04-22T13:36:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:09 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:36:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:09 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:09 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:36:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:09 smithi120 ceph-mon[31101]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 945 MiB/s wr, 2.12k op/s 2024-04-22T13:36:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:09 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:09 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:36:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:09 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:09 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:36:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:09 smithi132 ceph-mon[31531]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 945 MiB/s wr, 2.12k op/s 2024-04-22T13:36:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:09 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:09.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:09 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:36:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:09 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:09 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:36:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:09 smithi087 ceph-mon[27999]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 945 MiB/s wr, 2.12k op/s 2024-04-22T13:36:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:09 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:11.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:11 smithi120 ceph-mon[31101]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 814 MiB/s wr, 1.97k op/s 2024-04-22T13:36:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:11 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:36:11.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:11 smithi132 ceph-mon[31531]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 814 MiB/s wr, 1.97k op/s 2024-04-22T13:36:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:11 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:36:11.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:11 smithi087 ceph-mon[27999]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 814 MiB/s wr, 1.97k op/s 2024-04-22T13:36:11.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:11 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:36:13.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:13 smithi120 ceph-mon[31101]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 684 MiB/s wr, 1.68k op/s 2024-04-22T13:36:13.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:13 smithi132 ceph-mon[31531]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 684 MiB/s wr, 1.68k op/s 2024-04-22T13:36:13.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:13 smithi087 ceph-mon[27999]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 684 MiB/s wr, 1.68k op/s 2024-04-22T13:36:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:15 smithi120 ceph-mon[31101]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 1.71k op/s 2024-04-22T13:36:15.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:15 smithi132 ceph-mon[31531]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 1.71k op/s 2024-04-22T13:36:15.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:15 smithi087 ceph-mon[27999]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 695 MiB/s wr, 1.71k op/s 2024-04-22T13:36:17.919 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:17 smithi120 ceph-mon[31101]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 427 MiB/s wr, 1.26k op/s 2024-04-22T13:36:17.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:17 smithi132 ceph-mon[31531]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 427 MiB/s wr, 1.26k op/s 2024-04-22T13:36:17.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:17 smithi087 ceph-mon[27999]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 427 MiB/s wr, 1.26k op/s 2024-04-22T13:36:19.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:19 smithi132 ceph-mon[31531]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 353 MiB/s wr, 1.13k op/s 2024-04-22T13:36:19.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:19 smithi087 ceph-mon[27999]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 353 MiB/s wr, 1.13k op/s 2024-04-22T13:36:20.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:19 smithi120 ceph-mon[31101]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 353 MiB/s wr, 1.13k op/s 2024-04-22T13:36:20.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 -- 172.21.15.87:0/981062911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9490101aa0 msgr2=0x7f9490105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 --2- 172.21.15.87:0/981062911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9490101aa0 0x7f9490105a90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9480009900 tx=0x7f948002bc30 comp rx=0 tx=0).stop 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 -- 172.21.15.87:0/981062911 shutdown_connections 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 --2- 172.21.15.87:0/981062911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9490106170 0x7f949010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 --2- 172.21.15.87:0/981062911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9490101aa0 0x7f9490105a90 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 --2- 172.21.15.87:0/981062911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94901010f0 0x7f94901014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.340+0000 7f9497fe6640 1 -- 172.21.15.87:0/981062911 >> 172.21.15.87:0/981062911 conn(0x7f9490077fd0 msgr2=0x7f94900783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- 172.21.15.87:0/981062911 shutdown_connections 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- 172.21.15.87:0/981062911 wait complete. 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 Processor -- start 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- start start 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 0x7f94901975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 0x7f9490197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9490111090 con 0x7f949019be90 2024-04-22T13:36:20.339 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9490110f10 con 0x7f9490106170 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.341+0000 7f9497fe6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9490111210 con 0x7f94901010f0 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50200/0 (socket says 172.21.15.87:50200) 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 -- 172.21.15.87:0/438402105 learned_addr learned my addr 172.21.15.87:0/438402105 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9495581640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 0x7f9490197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9495d82640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 0x7f94901975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 -- 172.21.15.87:0/438402105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 msgr2=0x7f94901975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 0x7f94901975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 -- 172.21.15.87:0/438402105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 msgr2=0x7f9490197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 0x7f9490197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 -- 172.21.15.87:0/438402105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949019c980 con 0x7f949019be90 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9495581640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 0x7f9490197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9495d82640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 0x7f94901975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9496583640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f948800ebb0 tx=0x7f9488007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.340 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9486ffd640 1 -- 172.21.15.87:0/438402105 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488007ca0 con 0x7f949019be90 2024-04-22T13:36:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9486ffd640 1 -- 172.21.15.87:0/438402105 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9488007e40 con 0x7f949019be90 2024-04-22T13:36:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9486ffd640 1 -- 172.21.15.87:0/438402105 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94880057d0 con 0x7f949019be90 2024-04-22T13:36:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949019cc10 con 0x7f949019be90 2024-04-22T13:36:20.341 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.342+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9490069840 con 0x7f949019be90 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.344+0000 7f9486ffd640 1 -- 172.21.15.87:0/438402105 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9488004a40 con 0x7f949019be90 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 Processor -- start 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 -- start start 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfb9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfc54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f9486ffd640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 0x7f947c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5611bcf1e8d0 con 0x5611bcfb9b60 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5611bcebbe80 con 0x5611bcfba8d0 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.345+0000 7f949abd5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5611bcf1a890 con 0x5611bcfc30e0 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9496583640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 0x7f947c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50210/0 (socket says 172.21.15.87:50210) 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 -- 172.21.15.87:0/2352359036 learned_addr learned my addr 172.21.15.87:0/2352359036 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495581640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfb9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9486ffd640 1 -- 172.21.15.87:0/438402105 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f94880d1e10 con 0x7f949019be90 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495581640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfc54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 -- 172.21.15.87:0/2352359036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 msgr2=0x5611bcfc54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfc54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 -- 172.21.15.87:0/2352359036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 msgr2=0x5611bcfb9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfb9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 -- 172.21.15.87:0/2352359036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bcedd320 con 0x5611bcfb9b60 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495581640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfb9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9496583640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 0x7f947c0a6280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f948809dac0 tx=0x7f948806f800 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f9495d82640 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f948c00bd40 tx=0x7f948c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69923cacd95383e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f947bfff640 1 -- 172.21.15.87:0/2352359036 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c015070 con 0x5611bcfb9b60 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f947bfff640 1 -- 172.21.15.87:0/2352359036 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c00be30 con 0x5611bcfb9b60 2024-04-22T13:36:20.347 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.346+0000 7f947bfff640 1 -- 172.21.15.87:0/2352359036 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0119a0 con 0x5611bcfb9b60 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 -- 172.21.15.87:0/2352359036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 msgr2=0x5611bcfb9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f948c00bd40 tx=0x7f948c00f730 comp rx=0 tx=0).stop 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 -- 172.21.15.87:0/2352359036 shutdown_connections 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfc54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfb9b60 0x5611bcfb9fe0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 --2- 172.21.15.87:0/2352359036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfb9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 -- 172.21.15.87:0/2352359036 >> 172.21.15.87:0/2352359036 conn(0x5611bcf80c90 msgr2=0x5611bcfb4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 -- 172.21.15.87:0/2352359036 shutdown_connections 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.347+0000 7f949abd5740 1 -- 172.21.15.87:0/2352359036 wait complete. 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 -- start start 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 Processor -- start 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfc2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfbfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5611bcfc5e10 con 0x5611bcfc30e0 2024-04-22T13:36:20.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5611bcea6eb0 con 0x5611bcfba8d0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5611bcfc5f90 con 0x5611bcfb9b60 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f949abd5740 10 client.0 _ll_register_callbacks cb 0x5611bced3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50218/0 (socket says 172.21.15.87:50218) 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 -- 172.21.15.87:0/2203190484 learned_addr learned my addr 172.21.15.87:0/2203190484 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495d82640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfbfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 -- 172.21.15.87:0/2203190484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 msgr2=0x5611bcfc2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfc2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.348+0000 7f9495581640 1 -- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 msgr2=0x5611bcfbfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f9495581640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfbfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f9495581640 1 -- 172.21.15.87:0/2203190484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bcc31210 con 0x5611bcfc30e0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f9495581640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f949006b730 tx=0x7f948002c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9e38e286dac631 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480047020 con 0x5611bcfc30e0 2024-04-22T13:36:20.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f948002caf0 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948c00be30 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9480052050 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 10 client.16890 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611bccfd620 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c00be30 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9480040040 con 0x5611bcfc30e0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 10 client.16890 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 10 client.16890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.349+0000 7f949abd5740 20 client.16890 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bcfd31c0 0x5611bcfd55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f949abd5740 4 client.16890 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611bcf32880 con 0x5611bcfd31c0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f9495d82640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bcfd31c0 0x5611bcfd55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f9495d82640 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bcfd31c0 0x5611bcfd55b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5b837f32be43c6b server_cookie=98d7aa308a706bd7 in_seq=0 out_seq=0 2024-04-22T13:36:20.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.350+0000 7f94797fa640 10 client.16890 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:20.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f94797fa640 1 -- 172.21.15.87:0/2203190484 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611bcf32880 con 0x5611bcfd31c0 2024-04-22T13:36:20.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f94797fa640 10 client.16890 handle_command_reply: tid=0 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 client.16890 shutdown 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 msgr2=0x5611bcfc0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f949006b730 tx=0x7f948002c6d0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 shutdown_connections 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bcfd31c0 0x5611bcfd55b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc0490 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfbfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 --2- 172.21.15.87:0/2203190484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfc2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f94797fa640 0 client.16890 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 >> 172.21.15.87:0/2203190484 conn(0x5611bcf80c90 msgr2=0x5611bcfb65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 shutdown_connections 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 1 -- 172.21.15.87:0/2203190484 wait complete. 2024-04-22T13:36:20.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.351+0000 7f949abd5740 20 client.16890 trim_cache size 0 max 16384 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 Processor -- start 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 -- start start 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfc25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f94901adfe0 con 0x5611bcfc30e0 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94901ab8b0 con 0x5611bcfba8d0 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f949abd5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94901a9700 con 0x5611bcfb9b60 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f9495581640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f9495581640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50230/0 (socket says 172.21.15.87:50230) 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f9495581640 1 -- 172.21.15.87:0/4169074705 learned_addr learned my addr 172.21.15.87:0/4169074705 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f9495d82640 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfc25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.366+0000 7f9495581640 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 -- 172.21.15.87:0/4169074705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 msgr2=0x5611bcfced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 -- 172.21.15.87:0/4169074705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 msgr2=0x5611bcfc25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfc25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 -- 172.21.15.87:0/4169074705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611bcfb8e30 con 0x5611bcfc30e0 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f9495581640 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9480005c40 tx=0x7f94800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bae1cedd923ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f947b7fe640 1 -- 172.21.15.87:0/4169074705 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480047070 con 0x5611bcfc30e0 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f947b7fe640 1 -- 172.21.15.87:0/4169074705 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9480003d70 con 0x5611bcfc30e0 2024-04-22T13:36:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.367+0000 7f947b7fe640 1 -- 172.21.15.87:0/4169074705 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480005070 con 0x5611bcfc30e0 2024-04-22T13:36:20.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- 172.21.15.87:0/4169074705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 msgr2=0x5611bcfc2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9480005c40 tx=0x7f94800046c0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- 172.21.15.87:0/4169074705 shutdown_connections 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfc30e0 0x5611bcfc2af0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfba8d0 0x5611bcfc25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- 172.21.15.87:0/4169074705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfb9b60 0x5611bcfced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- 172.21.15.87:0/4169074705 >> 172.21.15.87:0/4169074705 conn(0x5611bcf80c90 msgr2=0x5611bcfbb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- 172.21.15.87:0/4169074705 shutdown_connections 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- 172.21.15.87:0/4169074705 wait complete. 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- start start 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 Processor -- start 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfb9b60 0x5611bcfcb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfbd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f94901a9d70 con 0x5611bcfba8d0 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94901ab1e0 con 0x5611bcfb9b60 2024-04-22T13:36:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94901aa440 con 0x5611bcfc30e0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.368+0000 7f949abd5740 10 client.0 _ll_register_callbacks cb 0x7f9480009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50246/0 (socket says 172.21.15.87:50246) 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 -- 172.21.15.87:0/1367953418 learned_addr learned my addr 172.21.15.87:0/1367953418 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495581640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfb9b60 0x5611bcfcb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495581640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfbd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 -- 172.21.15.87:0/1367953418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 msgr2=0x5611bcfbd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfbd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 -- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfb9b60 msgr2=0x5611bcfcb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfb9b60 0x5611bcfcb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 -- 172.21.15.87:0/1367953418 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611bcfbda20 con 0x5611bcfba8d0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495581640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfbd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f9495d82640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f948c00bd40 tx=0x7f948c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb849d2510c2293b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f94797fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c015040 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f948c005550 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9480003d70 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f948c004020 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f949abd5740 10 client.16896 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611bcfc41a0 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.369+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9480003d70 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f948c0270d0 con 0x5611bcfba8d0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 10 client.16896 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 10 client.16896 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 20 client.16896 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bd0397f0 0x5611bd03bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 4 client.16896 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5611bcebe3d0 con 0x5611bd0397f0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f9495581640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bd0397f0 0x5611bd03bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f94797fa640 10 client.16896 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.370+0000 7f9495581640 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bd0397f0 0x5611bd03bbe0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac6b7054d1bd6ee server_cookie=93d348715927687c in_seq=0 out_seq=0 2024-04-22T13:36:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.382+0000 7f94797fa640 1 -- 172.21.15.87:0/1367953418 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5611bcebe3d0 con 0x5611bd0397f0 2024-04-22T13:36:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.382+0000 7f94797fa640 10 client.16896 handle_command_reply: tid=0 2024-04-22T13:36:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.382+0000 7f949abd5740 1 client.16896 shutdown 2024-04-22T13:36:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.383+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 msgr2=0x5611bcfcb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.383+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f948c00bd40 tx=0x7f948c00bf20 comp rx=0 tx=0).stop 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 shutdown_connections 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5611bd0397f0 0x5611bd03bbe0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611bcfc30e0 0x5611bcfbd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5611bcfba8d0 0x5611bcfcb760 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 --2- 172.21.15.87:0/1367953418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5611bcfb9b60 0x5611bcfcb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 >> 172.21.15.87:0/1367953418 conn(0x5611bcf80c90 msgr2=0x5611bcfbe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.384+0000 7f94797fa640 0 client.16896 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.385+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 shutdown_connections 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.385+0000 7f949abd5740 1 -- 172.21.15.87:0/1367953418 wait complete. 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.385+0000 7f949abd5740 20 client.16896 trim_cache size 0 max 16384 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:36:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 msgr2=0x7f947c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 0x7f947c0a6280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f948809dac0 tx=0x7f948806f800 comp rx=0 tx=0).stop 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 msgr2=0x7f949019c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f948800ebb0 tx=0x7f9488007590 comp rx=0 tx=0).stop 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 shutdown_connections 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f947c0a3dc0 0x7f947c0a6280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f949019be90 0x7f949019c340 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9490106170 0x7f9490197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 --2- 172.21.15.87:0/438402105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94901010f0 0x7f94901975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 >> 172.21.15.87:0/438402105 conn(0x7f9490077fd0 msgr2=0x7f94900ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 shutdown_connections 2024-04-22T13:36:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:20.386+0000 7f9497fe6640 1 -- 172.21.15.87:0/438402105 wait complete. 2024-04-22T13:36:21.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:21 smithi132 ceph-mon[31531]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 146 MiB/s wr, 464 op/s 2024-04-22T13:36:21.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:21 smithi087 ceph-mon[27999]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 146 MiB/s wr, 464 op/s 2024-04-22T13:36:22.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:21 smithi120 ceph-mon[31101]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 146 MiB/s wr, 464 op/s 2024-04-22T13:36:23.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:23 smithi132 ceph-mon[31531]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 49 MiB/s wr, 144 op/s 2024-04-22T13:36:23.971 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:23 smithi120 ceph-mon[31101]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 49 MiB/s wr, 144 op/s 2024-04-22T13:36:23.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:23 smithi087 ceph-mon[27999]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 49 MiB/s wr, 144 op/s 2024-04-22T13:36:25.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:25 smithi132 ceph-mon[31531]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 200 MiB/s wr, 1.09k op/s 2024-04-22T13:36:25.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:25 smithi087 ceph-mon[27999]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 200 MiB/s wr, 1.09k op/s 2024-04-22T13:36:26.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:25 smithi120 ceph-mon[31101]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 200 MiB/s wr, 1.09k op/s 2024-04-22T13:36:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:27 smithi132 ceph-mon[31531]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 212 MiB/s wr, 1.22k op/s 2024-04-22T13:36:27.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:27 smithi087 ceph-mon[27999]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 212 MiB/s wr, 1.22k op/s 2024-04-22T13:36:28.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:27 smithi120 ceph-mon[31101]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 212 MiB/s wr, 1.22k op/s 2024-04-22T13:36:29.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:29 smithi132 ceph-mon[31531]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 248 MiB/s wr, 1.48k op/s 2024-04-22T13:36:29.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:29 smithi087 ceph-mon[27999]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 248 MiB/s wr, 1.48k op/s 2024-04-22T13:36:30.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:29 smithi120 ceph-mon[31101]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 248 MiB/s wr, 1.48k op/s 2024-04-22T13:36:31.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:31 smithi132 ceph-mon[31531]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 1.42k op/s 2024-04-22T13:36:31.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:31 smithi087 ceph-mon[27999]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 1.42k op/s 2024-04-22T13:36:32.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:31 smithi120 ceph-mon[31101]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 1.42k op/s 2024-04-22T13:36:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 -- 172.21.15.87:0/4182708314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01061b0 msgr2=0x7f5db010dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 --2- 172.21.15.87:0/4182708314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01061b0 0x7f5db010dd80 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009ca0 tx=0x7f5da802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 -- 172.21.15.87:0/4182708314 shutdown_connections 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 --2- 172.21.15.87:0/4182708314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01061b0 0x7f5db010dd80 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 --2- 172.21.15.87:0/4182708314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db0101a90 0x7f5db0105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 --2- 172.21.15.87:0/4182708314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db01010e0 0x7f5db01014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 -- 172.21.15.87:0/4182708314 >> 172.21.15.87:0/4182708314 conn(0x7f5db00771d0 msgr2=0x7f5db00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 -- 172.21.15.87:0/4182708314 shutdown_connections 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.483+0000 7f5db5396640 1 -- 172.21.15.87:0/4182708314 wait complete. 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 Processor -- start 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 -- start start 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db01061b0 0x7f5db0199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db0199f60 0x7f5db01a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0111120 con 0x7f5db01010e0 2024-04-22T13:36:33.482 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0110fa0 con 0x7f5db01061b0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5db5396640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01112a0 con 0x7f5db0199f60 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5daeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5daeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55718/0 (socket says 172.21.15.87:55718) 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5daeffd640 1 -- 172.21.15.87:0/1962229637 learned_addr learned my addr 172.21.15.87:0/1962229637 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.484+0000 7f5da7fff640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db01061b0 0x7f5db0199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db0199f60 0x7f5db01a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 -- 172.21.15.87:0/1962229637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db0199f60 msgr2=0x7f5db01a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db0199f60 0x7f5db01a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 -- 172.21.15.87:0/1962229637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db01061b0 msgr2=0x7f5db0199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db01061b0 0x7f5db0199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 -- 172.21.15.87:0/1962229637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db01a4d00 con 0x7f5db01010e0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5da000d770 tx=0x7f5da000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5dacff9640 1 -- 172.21.15.87:0/1962229637 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0017070 con 0x7f5db01010e0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5dacff9640 1 -- 172.21.15.87:0/1962229637 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da00054d0 con 0x7f5db01010e0 2024-04-22T13:36:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db01a4ff0 con 0x7f5db01010e0 2024-04-22T13:36:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5dacff9640 1 -- 172.21.15.87:0/1962229637 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0004c30 con 0x7f5db01010e0 2024-04-22T13:36:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.485+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db01a5460 con 0x7f5db01010e0 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.487+0000 7f5dacff9640 1 -- 172.21.15.87:0/1962229637 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5da0004dd0 con 0x7f5db01010e0 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 Processor -- start 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 -- start start 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084e8d0 0x56009084d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5dacff9640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 0x7f5d9c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5600907b28d0 con 0x5600908570e0 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56009074fe80 con 0x56009084e8d0 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5db7f84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5600907ae890 con 0x56009084db60 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5dacff9640 1 -- 172.21.15.87:0/1962229637 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f5da00d0220 con 0x7f5db01010e0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5da7fff640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 0x7f5d9c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.488+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55726/0 (socket says 172.21.15.87:55726) 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 -- 172.21.15.87:0/376456333 learned_addr learned my addr 172.21.15.87:0/376456333 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daeffd640 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084e8d0 0x56009084d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 -- 172.21.15.87:0/376456333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 msgr2=0x56009084dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 -- 172.21.15.87:0/376456333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084e8d0 msgr2=0x56009084d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084e8d0 0x56009084d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 -- 172.21.15.87:0/376456333 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560090771320 con 0x5600908570e0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5da7fff640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 0x7f5d9c0a62b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5db006b2a0 tx=0x7f5d9800b410 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009ca0 tx=0x7f5da802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ae86fc3189051e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5da57fa640 1 -- 172.21.15.87:0/376456333 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8048070 con 0x5600908570e0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5da57fa640 1 -- 172.21.15.87:0/376456333 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da8032970 con 0x5600908570e0 2024-04-22T13:36:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.489+0000 7f5da57fa640 1 -- 172.21.15.87:0/376456333 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8039070 con 0x5600908570e0 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- 172.21.15.87:0/376456333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 msgr2=0x5600908594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009ca0 tx=0x7f5da802e040 comp rx=0 tx=0).stop 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- 172.21.15.87:0/376456333 shutdown_connections 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x5600908594d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 --2- 172.21.15.87:0/376456333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084e8d0 0x56009084d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- 172.21.15.87:0/376456333 >> 172.21.15.87:0/376456333 conn(0x560090814c90 msgr2=0x560090848850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- 172.21.15.87:0/376456333 shutdown_connections 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- 172.21.15.87:0/376456333 wait complete. 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 -- start start 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.490+0000 7f5db7f84740 1 Processor -- start 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x560090853e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x560090854490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560090859db0 con 0x56009084db60 2024-04-22T13:36:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56009073aeb0 con 0x5600908570e0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560090859f30 con 0x56009084e8d0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 10 client.0 _ll_register_callbacks cb 0x560090767b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55734/0 (socket says 172.21.15.87:55734) 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 -- 172.21.15.87:0/23190910 learned_addr learned my addr 172.21.15.87:0/23190910 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5da7fff640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x560090853e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x560090854490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 -- 172.21.15.87:0/23190910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 msgr2=0x560090853e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x560090853e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 -- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 msgr2=0x560090854490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x560090854490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 -- 172.21.15.87:0/23190910 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600904c5210 con 0x56009084db60 2024-04-22T13:36:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x560090854490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5daeffd640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5da009c1c0 tx=0x7f5da009bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0e8c3721f862cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5d967fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0020040 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f5da00cdc60 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.491+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da8032970 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5da00a05e0 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 10 client.16908 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560090591620 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da8032970 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5da00b80d0 con 0x56009084db60 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 10 client.16908 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 10 client.16908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:33.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 20 client.16908 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560090867110 0x560090869500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 4 client.16908 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600907c6880 con 0x560090867110 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.492+0000 7f5da7fff640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560090867110 0x560090869500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.493+0000 7f5da7fff640 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560090867110 0x560090869500 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dba5afc746fefd2 server_cookie=db3d17b1a2cc60d4 in_seq=0 out_seq=0 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.493+0000 7f5d967fc640 10 client.16908 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.493+0000 7f5d967fc640 1 -- 172.21.15.87:0/23190910 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600907c6880 con 0x560090867110 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.493+0000 7f5d967fc640 10 client.16908 handle_command_reply: tid=0 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 client.16908 shutdown 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 msgr2=0x560090856af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5da009c1c0 tx=0x7f5da009bc80 comp rx=0 tx=0).stop 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 shutdown_connections 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560090867110 0x560090869500 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.495 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x560090854490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x560090853e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 --2- 172.21.15.87:0/23190910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084db60 0x560090856af0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 >> 172.21.15.87:0/23190910 conn(0x560090814c90 msgr2=0x56009084a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5d967fc640 0 client.16908 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 shutdown_connections 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 1 -- 172.21.15.87:0/23190910 wait complete. 2024-04-22T13:36:33.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.494+0000 7f5db7f84740 20 client.16908 trim_cache size 0 max 16384 2024-04-22T13:36:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 Processor -- start 2024-04-22T13:36:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 -- start start 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x5600908565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01b0230 con 0x56009084e8d0 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01adb00 con 0x5600908570e0 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5db7f84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01ab970 con 0x56009084db60 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55742/0 (socket says 172.21.15.87:55742) 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 -- 172.21.15.87:0/573250792 learned_addr learned my addr 172.21.15.87:0/573250792 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5daeffd640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x5600908565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 -- 172.21.15.87:0/573250792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 msgr2=0x56009084f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 -- 172.21.15.87:0/573250792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 msgr2=0x5600908565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x5600908565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5da7fff640 1 -- 172.21.15.87:0/573250792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600908523d0 con 0x56009084e8d0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5daeffd640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.509+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x5600908565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5da7fff640 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5d98005fd0 tx=0x7f5d98013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9e4e38c5cb8237 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5da4ff9640 1 -- 172.21.15.87:0/573250792 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9802c070 con 0x56009084e8d0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5da4ff9640 1 -- 172.21.15.87:0/573250792 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d9801e420 con 0x56009084e8d0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5da4ff9640 1 -- 172.21.15.87:0/573250792 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9801d610 con 0x56009084e8d0 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 -- 172.21.15.87:0/573250792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 msgr2=0x560090862cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5d98005fd0 tx=0x7f5d98013000 comp rx=0 tx=0).stop 2024-04-22T13:36:33.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 -- 172.21.15.87:0/573250792 shutdown_connections 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600908570e0 0x5600908565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56009084e8d0 0x560090862cb0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.510+0000 7f5db7f84740 1 --2- 172.21.15.87:0/573250792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084db60 0x56009084f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- 172.21.15.87:0/573250792 >> 172.21.15.87:0/573250792 conn(0x560090814c90 msgr2=0x56009084ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- 172.21.15.87:0/573250792 shutdown_connections 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- 172.21.15.87:0/573250792 wait complete. 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- start start 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 Processor -- start 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084db60 0x5600908584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x5600908589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01abfc0 con 0x5600908570e0 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01ad430 con 0x56009084db60 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01ac690 con 0x56009084e8d0 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5db7f84740 10 client.0 _ll_register_callbacks cb 0x7f5da0009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:33.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5daf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55754/0 (socket says 172.21.15.87:55754) 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5daf7fe640 1 -- 172.21.15.87:0/1213003539 learned_addr learned my addr 172.21.15.87:0/1213003539 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5da7fff640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x5600908589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.511+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084db60 0x5600908584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 -- 172.21.15.87:0/1213003539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 msgr2=0x5600908589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x5600908589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 -- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084db60 msgr2=0x5600908584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084db60 0x5600908584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 -- 172.21.15.87:0/1213003539 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56009085de00 con 0x5600908570e0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5daf7fe640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009ca0 tx=0x7f5da80326f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c2f6be02a1ea51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8039070 con 0x5600908570e0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f5da8003970 con 0x5600908570e0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d9801e420 con 0x5600908570e0 2024-04-22T13:36:33.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5da8048040 con 0x5600908570e0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 10 client.16914 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5600908cde70 con 0x5600908570e0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d9801e420 con 0x5600908570e0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5da8055020 con 0x5600908570e0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 10 client.16914 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 10 client.16914 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.512+0000 7f5db7f84740 20 client.16914 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5600908ce150 0x5600908d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5db7f84740 4 client.16914 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600907523d0 con 0x5600908ce150 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5600908ce150 0x5600908d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5daeffd640 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5600908ce150 0x5600908d0540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b13ae217dcba65ec server_cookie=9c31794a77609acb in_seq=0 out_seq=0 2024-04-22T13:36:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.513+0000 7f5d967fc640 10 client.16914 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:33.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5d967fc640 1 -- 172.21.15.87:0/1213003539 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600907523d0 con 0x5600908ce150 2024-04-22T13:36:33.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5d967fc640 10 client.16914 handle_command_reply: tid=0 2024-04-22T13:36:33.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 client.16914 shutdown 2024-04-22T13:36:33.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 msgr2=0x56009085d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.519 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009ca0 tx=0x7f5da80326f0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 shutdown_connections 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5600908ce150 0x5600908d0540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5600908570e0 0x56009085d8c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56009084e8d0 0x5600908589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 --2- 172.21.15.87:0/1213003539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56009084db60 0x5600908584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5d967fc640 0 client.16914 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 >> 172.21.15.87:0/1213003539 conn(0x560090814c90 msgr2=0x560090847040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 shutdown_connections 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 1 -- 172.21.15.87:0/1213003539 wait complete. 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.523+0000 7f5db7f84740 20 client.16914 trim_cache size 0 max 16384 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:36:33.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:36:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:36:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 msgr2=0x7f5d9c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.521 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 0x7f5d9c0a62b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5db006b2a0 tx=0x7f5d9800b410 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 msgr2=0x7f5db01994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5da000d770 tx=0x7f5da000dc30 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 shutdown_connections 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5d9c0a3df0 0x7f5d9c0a62b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5db0199f60 0x7f5db01a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db01061b0 0x7f5db0199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 --2- 172.21.15.87:0/1962229637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5db01010e0 0x7f5db01994e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 >> 172.21.15.87:0/1962229637 conn(0x7f5db00771d0 msgr2=0x7f5db00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 shutdown_connections 2024-04-22T13:36:33.522 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:33.524+0000 7f5db5396640 1 -- 172.21.15.87:0/1962229637 wait complete. 2024-04-22T13:36:33.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:33 smithi132 ceph-mon[31531]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:33.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:33 smithi087 ceph-mon[27999]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:34.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:33 smithi120 ceph-mon[31101]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:35.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:35 smithi132 ceph-mon[31531]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:35.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:35 smithi087 ceph-mon[27999]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:36.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:35 smithi120 ceph-mon[31101]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 214 MiB/s wr, 1.38k op/s 2024-04-22T13:36:37.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:37 smithi132 ceph-mon[31531]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 441 op/s 2024-04-22T13:36:37.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:37 smithi087 ceph-mon[27999]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 441 op/s 2024-04-22T13:36:38.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:37 smithi120 ceph-mon[31101]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 441 op/s 2024-04-22T13:36:38.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:36:38.301 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:36:38.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:36:38.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/482838294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24104a90 msgr2=0x7f2a24104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/482838294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24104a90 0x7f2a24104ef0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c009cd0 tx=0x7f2a1c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/482838294 shutdown_connections 2024-04-22T13:36:38.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/482838294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24104a90 0x7f2a24104ef0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/482838294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2406abb0 0x7f2a2406b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/482838294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a24106f20 0x7f2a2406a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/482838294 >> 172.21.15.87:0/482838294 conn(0x7f2a240771d0 msgr2=0x7f2a240775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/482838294 shutdown_connections 2024-04-22T13:36:38.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.417+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/482838294 wait complete. 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 Processor -- start 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 -- start start 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a24104a90 0x7f2a241a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 0x7f2a2419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241187d0 con 0x7f2a24106f20 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a24118650 con 0x7f2a24104a90 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2b2c0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a24118950 con 0x7f2a2406abb0 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2985d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 0x7f2a2419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:48818/0 (socket says 172.21.15.87:48818) 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2905c640 1 -- 172.21.15.87:0/321944226 learned_addr learned my addr 172.21.15.87:0/321944226 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.418+0000 7f2a2885b640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a24104a90 0x7f2a241a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 -- 172.21.15.87:0/321944226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a24104a90 msgr2=0x7f2a241a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a24104a90 0x7f2a241a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 -- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 msgr2=0x7f2a2419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 0x7f2a2419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 -- 172.21.15.87:0/321944226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2419f270 con 0x7f2a2406abb0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2985d640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 0x7f2a2419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2905c640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e9f0 tx=0x7f2a1400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a1a7fc640 1 -- 172.21.15.87:0/321944226 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400cd80 con 0x7f2a2406abb0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2419f560 con 0x7f2a2406abb0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a1a7fc640 1 -- 172.21.15.87:0/321944226 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a140054d0 con 0x7f2a2406abb0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a1a7fc640 1 -- 172.21.15.87:0/321944226 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a14010690 con 0x7f2a2406abb0 2024-04-22T13:36:38.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.419+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a241ab800 con 0x7f2a2406abb0 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 Processor -- start 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 -- start start 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57580 0x55d9f8c57960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c60540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8bbc1e0 con 0x55d9f8c60a80 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8b59390 con 0x55d9f8c57ea0 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2df82740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8bb81a0 con 0x55d9f8c57580 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55774/0 (socket says 172.21.15.87:55774) 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 -- 172.21.15.87:0/3480841079 learned_addr learned my addr 172.21.15.87:0/3480841079 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2985d640 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57580 0x55d9f8c57960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a1a7fc640 1 -- 172.21.15.87:0/321944226 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a14004e90 con 0x7f2a2406abb0 2024-04-22T13:36:38.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 -- 172.21.15.87:0/3480841079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57580 msgr2=0x55d9f8c57960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57580 0x55d9f8c57960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 -- 172.21.15.87:0/3480841079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 msgr2=0x55d9f8c60540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c60540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.422+0000 7f2a2885b640 1 -- 172.21.15.87:0/3480841079 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f89987d0 con 0x55d9f8c60a80 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009c60 tx=0x7f2a20011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d403e29146135c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a0f7fe640 1 -- 172.21.15.87:0/3480841079 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20004230 con 0x55d9f8c60a80 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a1a7fc640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 0x7f2a100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a0f7fe640 1 -- 172.21.15.87:0/3480841079 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200043d0 con 0x55d9f8c60a80 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a0f7fe640 1 -- 172.21.15.87:0/3480841079 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20004cd0 con 0x55d9f8c60a80 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a2985d640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 0x7f2a100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.423+0000 7f2a1a7fc640 1 -- 172.21.15.87:0/321944226 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2a140d0b80 con 0x7f2a2406abb0 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- 172.21.15.87:0/3480841079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 msgr2=0x55d9f8c62e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2985d640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 0x7f2a100a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c005cc0 tx=0x7f2a1c005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009c60 tx=0x7f2a20011810 comp rx=0 tx=0).stop 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- 172.21.15.87:0/3480841079 shutdown_connections 2024-04-22T13:36:38.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c60a80 0x55d9f8c62e70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c60540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3480841079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57580 0x55d9f8c57960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- 172.21.15.87:0/3480841079 >> 172.21.15.87:0/3480841079 conn(0x55d9f8c51cb0 msgr2=0x55d9f8c52090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- 172.21.15.87:0/3480841079 shutdown_connections 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- 172.21.15.87:0/3480841079 wait complete. 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 -- start start 2024-04-22T13:36:38.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.424+0000 7f2a2df82740 1 Processor -- start 2024-04-22T13:36:38.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 0x55d9f8c5e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8c63870 con 0x55d9f8c57580 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8b752f0 con 0x55d9f8c60a80 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8c639f0 con 0x55d9f8c57ea0 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 10 client.0 _ll_register_callbacks cb 0x55d9f8b82580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55790/0 (socket says 172.21.15.87:55790) 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 -- 172.21.15.87:0/3738095033 learned_addr learned my addr 172.21.15.87:0/3738095033 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2985d640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 0x55d9f8c5e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 -- 172.21.15.87:0/3738095033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 msgr2=0x55d9f8c5dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 -- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 msgr2=0x55d9f8c5e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 0x55d9f8c5e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 -- 172.21.15.87:0/3738095033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f88ce3f0 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2985d640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 0x55d9f8c5e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2905c640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2a140ce3f0 tx=0x7f2a1409c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e93065f92c45f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140b6060 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2a14099200 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.425+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a200043d0 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2a1409cb90 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 10 client.16926 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d9f888e0f0 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200043d0 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2a140988e0 con 0x55d9f8c57580 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 10 client.16926 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:38.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 20 client.16926 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8c70a90 0x55d9f8c72e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 4 client.16926 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9f8bc9c00 con 0x55d9f8c70a90 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.426+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8c70a90 0x55d9f8c72e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.427+0000 7f2a2885b640 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8c70a90 0x55d9f8c72e80 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd50d558568d6896 server_cookie=903777bb924fee2c in_seq=0 out_seq=0 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.427+0000 7f2a0cff9640 10 client.16926 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a0cff9640 1 -- 172.21.15.87:0/3738095033 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9f8bc9c00 con 0x55d9f8c70a90 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a0cff9640 10 client.16926 handle_command_reply: tid=0 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 client.16926 shutdown 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 msgr2=0x55d9f8c5da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2a140ce3f0 tx=0x7f2a1409c4f0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 shutdown_connections 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8c70a90 0x55d9f8c72e80 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c60a80 0x55d9f8c5e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a0cff9640 0 client.16926 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:38.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 --2- 172.21.15.87:0/3738095033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c5da80 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 >> 172.21.15.87:0/3738095033 conn(0x55d9f8c51cb0 msgr2=0x55d9f8c53a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 shutdown_connections 2024-04-22T13:36:38.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 1 -- 172.21.15.87:0/3738095033 wait complete. 2024-04-22T13:36:38.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.428+0000 7f2a2df82740 20 client.16926 trim_cache size 0 max 16384 2024-04-22T13:36:38.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 Processor -- start 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 -- start start 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c50bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b6880 con 0x55d9f8c57580 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b4150 con 0x55d9f8c57ea0 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2df82740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b1fa0 con 0x55d9f8c60a80 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55804/0 (socket says 172.21.15.87:55804) 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 -- 172.21.15.87:0/1366285918 learned_addr learned my addr 172.21.15.87:0/1366285918 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2885b640 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 -- 172.21.15.87:0/1366285918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 msgr2=0x55d9f8c50bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c50bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 -- 172.21.15.87:0/1366285918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 msgr2=0x55d9f8c5d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 -- 172.21.15.87:0/1366285918 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f8c734b0 con 0x55d9f8c57580 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.447+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400c6a0 tx=0x7f2a14004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494daa7d6eb2d291 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a0effd640 1 -- 172.21.15.87:0/1366285918 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1409c960 con 0x55d9f8c57580 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a0effd640 1 -- 172.21.15.87:0/1366285918 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a140a0f40 con 0x55d9f8c57580 2024-04-22T13:36:38.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a0effd640 1 -- 172.21.15.87:0/1366285918 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a14005180 con 0x55d9f8c57580 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 -- 172.21.15.87:0/1366285918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 msgr2=0x55d9f8c6c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400c6a0 tx=0x7f2a14004a00 comp rx=0 tx=0).stop 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 -- 172.21.15.87:0/1366285918 shutdown_connections 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c50bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c5d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1366285918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c6c730 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 -- 172.21.15.87:0/1366285918 >> 172.21.15.87:0/1366285918 conn(0x55d9f8c51cb0 msgr2=0x55d9f8c504e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 -- 172.21.15.87:0/1366285918 shutdown_connections 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.448+0000 7f2a2df82740 1 -- 172.21.15.87:0/1366285918 wait complete. 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 -- start start 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 Processor -- start 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c643c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c64900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b2610 con 0x55d9f8c57580 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b3a80 con 0x55d9f8c57ea0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a241b2ce0 con 0x55d9f8c60a80 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2df82740 10 client.0 _ll_register_callbacks cb 0x7f2a1400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2905c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55820/0 (socket says 172.21.15.87:55820) 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2905c640 1 -- 172.21.15.87:0/1031820479 learned_addr learned my addr 172.21.15.87:0/1031820479 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.449+0000 7f2a2885b640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2985d640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c64900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 -- 172.21.15.87:0/1031820479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 msgr2=0x55d9f8c64900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c64900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 -- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 msgr2=0x55d9f8c643c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c643c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 -- 172.21.15.87:0/1031820479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f8c62e70 con 0x55d9f8c57580 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2905c640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2a140d0f30 tx=0x7f2a14009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e7b0d162770b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140a5800 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2a1409bbf0 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a140a0f40 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2a140a5370 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2df82740 10 client.16932 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d9f8c62e70 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a140a0f40 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.450+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2a140cea90 con 0x55d9f8c57580 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 10 client.16932 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 20 client.16932 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8cd7060 0x55d9f8cd9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 4 client.16932 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9f8bc3a70 con 0x55d9f8cd7060 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2885b640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8cd7060 0x55d9f8cd9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a0cff9640 10 client.16932 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2885b640 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8cd7060 0x55d9f8cd9450 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e8607ec68cebe1 server_cookie=e9fa60bc41e3ec7b in_seq=0 out_seq=0 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a0cff9640 1 -- 172.21.15.87:0/1031820479 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d9f8bc3a70 con 0x55d9f8cd7060 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a0cff9640 10 client.16932 handle_command_reply: tid=0 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.451+0000 7f2a2df82740 1 client.16932 shutdown 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 msgr2=0x55d9f8c69050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2a140d0f30 tx=0x7f2a14009310 comp rx=0 tx=0).stop 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 shutdown_connections 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d9f8cd7060 0x55d9f8cd9450 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f8c60a80 0x55d9f8c64900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d9f8c57ea0 0x55d9f8c643c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a0cff9640 0 client.16932 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 --2- 172.21.15.87:0/1031820479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d9f8c57580 0x55d9f8c69050 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 >> 172.21.15.87:0/1031820479 conn(0x55d9f8c51cb0 msgr2=0x55d9f8c5e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 shutdown_connections 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 1 -- 172.21.15.87:0/1031820479 wait complete. 2024-04-22T13:36:38.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2df82740 20 client.16932 trim_cache size 0 max 16384 2024-04-22T13:36:38.454 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:38.454 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:36:38.454 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:36:38.454 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 msgr2=0x7f2a100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 0x7f2a100a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c005cc0 tx=0x7f2a1c005c50 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 msgr2=0x7f2a241a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e9f0 tx=0x7f2a1400edf0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 shutdown_connections 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2a100a3d20 0x7f2a100a61e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2a24106f20 0x7f2a2419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a24104a90 0x7f2a241a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 --2- 172.21.15.87:0/321944226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a2406abb0 0x7f2a241a4940 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 >> 172.21.15.87:0/321944226 conn(0x7f2a240771d0 msgr2=0x7f2a240fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 shutdown_connections 2024-04-22T13:36:38.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.452+0000 7f2a2b2c0640 1 -- 172.21.15.87:0/321944226 wait complete. 2024-04-22T13:36:38.461 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:36:38.461 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:36:38.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.519+0000 7f93ac643640 1 -- 172.21.15.87:0/1367830545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a4101b60 msgr2=0x7f93a410dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.519+0000 7f93ac643640 1 --2- 172.21.15.87:0/1367830545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a4101b60 0x7f93a410dc90 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f939c009cd0 tx=0x7f939c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 -- 172.21.15.87:0/1367830545 shutdown_connections 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 --2- 172.21.15.87:0/1367830545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a4101b60 0x7f93a410dc90 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 --2- 172.21.15.87:0/1367830545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a41011a0 0x7f93a4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 --2- 172.21.15.87:0/1367830545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a4106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 -- 172.21.15.87:0/1367830545 >> 172.21.15.87:0/1367830545 conn(0x7f93a4077e30 msgr2=0x7f93a4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 -- 172.21.15.87:0/1367830545 shutdown_connections 2024-04-22T13:36:38.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 -- 172.21.15.87:0/1367830545 wait complete. 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 Processor -- start 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 -- start start 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.520+0000 7f93ac643640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 0x7f93a4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4112240 con 0x7f93a41011a0 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41120c0 con 0x7f93a4101b60 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41123c0 con 0x7f93a4105e30 2024-04-22T13:36:38.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55840/0 (socket says 172.21.15.87:55840) 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 -- 172.21.15.87:0/2963440348 learned_addr learned my addr 172.21.15.87:0/2963440348 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aabe0640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93a9bde640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 0x7f93a4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 -- 172.21.15.87:0/2963440348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 msgr2=0x7f93a419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 -- 172.21.15.87:0/2963440348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 msgr2=0x7f93a4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 0x7f93a4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 -- 172.21.15.87:0/2963440348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a419d6d0 con 0x7f93a41011a0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aabe0640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93a9bde640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 0x7f93a4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93aa3df640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f93a000d770 tx=0x7f93a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0017070 con 0x7f93a41011a0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 -- 172.21.15.87:0/2963440348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a419d9c0 con 0x7f93a41011a0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93a00054d0 con 0x7f93a41011a0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0004c30 con 0x7f93a41011a0 2024-04-22T13:36:38.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.521+0000 7f93ac643640 1 -- 172.21.15.87:0/2963440348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a41a5250 con 0x7f93a41011a0 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.522+0000 7f93ac643640 1 -- 172.21.15.87:0/2963440348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a4068de0 con 0x7f93a41011a0 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.524+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f93a0004dd0 con 0x7f93a41011a0 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.524+0000 7f939b7fe640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 0x7f93900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.524+0000 7f93a9bde640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 0x7f93900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.524+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f93a00d0700 con 0x7f93a41011a0 2024-04-22T13:36:38.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.525+0000 7f93a9bde640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 0x7f93900a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93a4100cf0 tx=0x7f9394005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.528+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93a009ce10 con 0x7f93a41011a0 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 -- 172.21.15.87:0/1143826668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0102030 msgr2=0x7f48d010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 --2- 172.21.15.87:0/1143826668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0102030 0x7f48d010dd50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009ca0 tx=0x7f48c802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 -- 172.21.15.87:0/1143826668 shutdown_connections 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 --2- 172.21.15.87:0/1143826668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0102030 0x7f48d010dd50 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 --2- 172.21.15.87:0/1143826668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0101670 0x7f48d0101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 --2- 172.21.15.87:0/1143826668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0105ec0 0x7f48d01062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 -- 172.21.15.87:0/1143826668 >> 172.21.15.87:0/1143826668 conn(0x7f48d0077480 msgr2=0x7f48d00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 -- 172.21.15.87:0/1143826668 shutdown_connections 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 -- 172.21.15.87:0/1143826668 wait complete. 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.606+0000 7f48d58e4640 1 Processor -- start 2024-04-22T13:36:38.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 -- start start 2024-04-22T13:36:38.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 0x7f48d01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 0x7f48d0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f48d01110f0 con 0x7f48d0105ec0 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48d0110f70 con 0x7f48d0102030 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48d58e4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f48d0111270 con 0x7f48d0101670 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55862/0 (socket says 172.21.15.87:55862) 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 -- 172.21.15.87:0/3169212065 learned_addr learned my addr 172.21.15.87:0/3169212065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48ce7fc640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 0x7f48d0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48ceffd640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 0x7f48d01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 -- 172.21.15.87:0/3169212065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 msgr2=0x7f48d01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 0x7f48d01986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 -- 172.21.15.87:0/3169212065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 msgr2=0x7f48d0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 0x7f48d0198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48cf7fe640 1 -- 172.21.15.87:0/3169212065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d019d690 con 0x7f48d0105ec0 2024-04-22T13:36:38.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48ceffd640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 0x7f48d01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.607+0000 7f48ce7fc640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 0x7f48d0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48cf7fe640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f48c802c700 tx=0x7f48c80346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8041020 con 0x7f48d0105ec0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d019d920 con 0x7f48d0105ec0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48c8032960 con 0x7f48d0105ec0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8039040 con 0x7f48d0105ec0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.608+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d01a5210 con 0x7f48d0105ec0 2024-04-22T13:36:38.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.609+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d0069230 con 0x7f48d0105ec0 2024-04-22T13:36:38.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.610+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48c8044020 con 0x7f48d0105ec0 2024-04-22T13:36:38.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.611+0000 7f48d48e2640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 0x7f48b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:38.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.611+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f48c80f2930 con 0x7f48d0105ec0 2024-04-22T13:36:38.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.611+0000 7f48ceffd640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 0x7f48b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:38.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.611+0000 7f48ceffd640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 0x7f48b80a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009fc0 tx=0x7f48c0005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:38.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.614+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48c804b020 con 0x7f48d0105ec0 2024-04-22T13:36:38.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.886+0000 7f93ac643640 1 -- 172.21.15.87:0/2963440348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93a4199a30 con 0x7f93a41011a0 2024-04-22T13:36:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.888+0000 7f939b7fe640 1 -- 172.21.15.87:0/2963440348 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f93a00b4e70 con 0x7f93a41011a0 2024-04-22T13:36:38.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:36:38.886 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:36:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:36:38.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.897+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 msgr2=0x7f93900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.897+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 0x7f93900a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f93a4100cf0 tx=0x7f9394005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:38.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.897+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 msgr2=0x7f93a41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:38.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.897+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f93a000d770 tx=0x7f93a000dc30 comp rx=0 tx=0).stop 2024-04-22T13:36:38.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 shutdown_connections 2024-04-22T13:36:38.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f93900a3d20 0x7f93900a61e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93a4105e30 0x7f93a419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a4101b60 0x7f93a4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 --2- 172.21.15.87:0/2963440348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93a41011a0 0x7f93a41986e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.898+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 >> 172.21.15.87:0/2963440348 conn(0x7f93a4077e30 msgr2=0x7f93a40fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.900+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 shutdown_connections 2024-04-22T13:36:38.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:38.902+0000 7f93997fa640 1 -- 172.21.15.87:0/2963440348 wait complete. 2024-04-22T13:36:38.927 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 7503ae6d --release --await' 2024-04-22T13:36:38.927 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 7503ae6d --release --await 2024-04-22T13:36:39.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.029+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d010a400 con 0x7f48d0105ec0 2024-04-22T13:36:39.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.034+0000 7f48d48e2640 1 -- 172.21.15.87:0/3169212065 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f48c80bf270 con 0x7f48d0105ec0 2024-04-22T13:36:39.031 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:36:39.031 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:36:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:36:39.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.039+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 msgr2=0x7f48b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.039+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 0x7f48b80a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009fc0 tx=0x7f48c0005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 msgr2=0x7f48d019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f48c802c700 tx=0x7f48c80346d0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 shutdown_connections 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f48b80a3d20 0x7f48b80a61e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48d0105ec0 0x7f48d019cf60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48d0102030 0x7f48d0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 --2- 172.21.15.87:0/3169212065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48d0101670 0x7f48d01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.040+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 >> 172.21.15.87:0/3169212065 conn(0x7f48d0077480 msgr2=0x7f48d01007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.042+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 shutdown_connections 2024-04-22T13:36:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.043+0000 7f48d58e4640 1 -- 172.21.15.87:0/3169212065 wait complete. 2024-04-22T13:36:39.067 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:36:39.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.239+0000 7fdda5881640 1 -- 172.21.15.87:0/3355502181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd980a5b80 msgr2=0x7fdd980a9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.239+0000 7fdda5881640 1 --2- 172.21.15.87:0/3355502181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd980a5b80 0x7fdd980a9bc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009860 tx=0x7fdd9402bbb0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.240+0000 7fdda5881640 1 -- 172.21.15.87:0/3355502181 shutdown_connections 2024-04-22T13:36:39.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.240+0000 7fdda5881640 1 --2- 172.21.15.87:0/3355502181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd980b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.240+0000 7fdda5881640 1 --2- 172.21.15.87:0/3355502181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd980a5b80 0x7fdd980a9bc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.240+0000 7fdda5881640 1 --2- 172.21.15.87:0/3355502181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd980a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.240+0000 7fdda5881640 1 -- 172.21.15.87:0/3355502181 >> 172.21.15.87:0/3355502181 conn(0x7fdd9801a670 msgr2=0x7fdd9801aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.247+0000 7fdda5881640 1 -- 172.21.15.87:0/3355502181 shutdown_connections 2024-04-22T13:36:39.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.247+0000 7fdda5881640 1 -- 172.21.15.87:0/3355502181 wait complete. 2024-04-22T13:36:39.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.248+0000 7fdda5881640 1 Processor -- start 2024-04-22T13:36:39.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.249+0000 7fdda5881640 1 -- start start 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.249+0000 7fdda5881640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd9813d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.249+0000 7fdda5881640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdda5881640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 0x7fdd981423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd9813d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:48926/0 (socket says 172.21.15.87:48926) 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd9813d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48220/0 (socket says 172.21.15.87:48220) 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/2816219617 learned_addr learned my addr 172.21.15.87:0/2816219617 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980b4e10 con 0x7fdd98141ef0 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980b4c90 con 0x7fdd980a51d0 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980b4f90 con 0x7fdd980aa2a0 2024-04-22T13:36:39.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.250+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 0x7fdd981423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/2816219617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 msgr2=0x7fdd9813d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd9813d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 msgr2=0x7fdd981423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 0x7fdd981423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd981429e0 con 0x7fdd980aa2a0 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd94037040 tx=0x7fdd94004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 0x7fdd981423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdda487f640 1 -- 172.21.15.87:0/2816219617 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94041020 con 0x7fdd980aa2a0 2024-04-22T13:36:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdda487f640 1 -- 172.21.15.87:0/2816219617 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd9402ca00 con 0x7fdd980aa2a0 2024-04-22T13:36:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.251+0000 7fdda487f640 1 -- 172.21.15.87:0/2816219617 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94039040 con 0x7fdd980aa2a0 2024-04-22T13:36:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.253+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd98142c70 con 0x7fdd980aa2a0 2024-04-22T13:36:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.253+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd980ac020 con 0x7fdd980aa2a0 2024-04-22T13:36:39.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.257+0000 7fdda487f640 1 -- 172.21.15.87:0/2816219617 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd94044020 con 0x7fdd980aa2a0 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.258+0000 7fdda487f640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 0x7fdd880aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.258+0000 7fdda487f640 1 -- 172.21.15.87:0/2816219617 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fdd940f1630 con 0x7fdd980aa2a0 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 Processor -- start 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 -- start start 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12b7d0 0x55ff2d12bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d13dc50 0x55ff2d140040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d088500 con 0x55ff2d12b7d0 2024-04-22T13:36:39.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d02e6a0 con 0x55ff2d12c180 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.259+0000 7fdda8479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d0844c0 con 0x55ff2d13dc50 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d13dc50 0x55ff2d140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48230/0 (socket says 172.21.15.87:48230) 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1043618437 learned_addr learned my addr 172.21.15.87:0/1043618437 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9effd640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 0x7fdd880aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1043618437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d13dc50 msgr2=0x55ff2d140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d13dc50 0x55ff2d140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1043618437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12b7d0 msgr2=0x55ff2d12bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12b7d0 0x55ff2d12bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1043618437 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2d04d5e0 con 0x55ff2d12c180 2024-04-22T13:36:39.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.260+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd700 tx=0x7fdd94004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34f3e7709df031a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.261+0000 7fdd9effd640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 0x7fdd880aa7f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdda0065470 tx=0x7fdda0050bd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.261+0000 7fdd85ffb640 1 -- 172.21.15.87:0/1043618437 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940bd0d0 con 0x55ff2d12c180 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.261+0000 7fdd85ffb640 1 -- 172.21.15.87:0/1043618437 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd940bde70 con 0x55ff2d12c180 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 -- 172.21.15.87:0/1043618437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 msgr2=0x55ff2d1254d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd700 tx=0x7fdd94004610 comp rx=0 tx=0).stop 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 -- 172.21.15.87:0/1043618437 shutdown_connections 2024-04-22T13:36:39.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d13dc50 0x55ff2d140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d1254d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 --2- 172.21.15.87:0/1043618437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12b7d0 0x55ff2d12bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.262+0000 7fdda8479740 1 -- 172.21.15.87:0/1043618437 >> 172.21.15.87:0/1043618437 conn(0x55ff2d126410 msgr2=0x55ff2d1267f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.263+0000 7fdda8479740 1 -- 172.21.15.87:0/1043618437 shutdown_connections 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.263+0000 7fdda8479740 1 -- 172.21.15.87:0/1043618437 wait complete. 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 -- start start 2024-04-22T13:36:39.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 Processor -- start 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d1253a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d122cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d12cc40 con 0x55ff2d12c180 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d041a60 con 0x55ff2d13dc50 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ff2d12cdc0 con 0x55ff2d12b7d0 2024-04-22T13:36:39.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.264+0000 7fdda8479740 10 client.0 _ll_register_callbacks cb 0x55ff2cd9a110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55916/0 (socket says 172.21.15.87:55916) 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/851031761 learned_addr learned my addr 172.21.15.87:0/851031761 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d122cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9effd640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d1253a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/851031761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 msgr2=0x55ff2d1253a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d1253a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 msgr2=0x55ff2d122cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d122cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/851031761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff2cd9a920 con 0x55ff2d12c180 2024-04-22T13:36:39.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.265+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d122cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.267+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd470 tx=0x7fdd94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9cc014879dc637 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.268+0000 7fdd837f6640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.268+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940da020 con 0x55ff2d12c180 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.268+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fdd940c5f50 con 0x55ff2d12c180 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.268+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd940bde70 con 0x55ff2d12c180 2024-04-22T13:36:39.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.269+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdd940ce050 con 0x55ff2d12c180 2024-04-22T13:36:39.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.270+0000 7fdda8479740 10 client.16950 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.270+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ff2ce66d30 con 0x55ff2d12c180 2024-04-22T13:36:39.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.272+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd940bde70 con 0x55ff2d12c180 2024-04-22T13:36:39.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdd9403c030 con 0x55ff2d12c180 2024-04-22T13:36:39.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 10 client.16950 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 10 client.16950 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:36:39.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 20 client.16950 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d139ef0 0x55ff2d13c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 4 client.16950 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:39.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.273+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff2d05ba10 con 0x55ff2d139ef0 2024-04-22T13:36:39.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.274+0000 7fdd9effd640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d139ef0 0x55ff2d13c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.278+0000 7fdd9effd640 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d139ef0 0x55ff2d13c310 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7148e9310ff56b server_cookie=795c0e7fb00975bb in_seq=0 out_seq=0 2024-04-22T13:36:39.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.278+0000 7fdd837f6640 10 client.16950 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:39.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.279+0000 7fdd837f6640 1 -- 172.21.15.87:0/851031761 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff2d05ba10 con 0x55ff2d139ef0 2024-04-22T13:36:39.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.279+0000 7fdd837f6640 10 client.16950 handle_command_reply: tid=0 2024-04-22T13:36:39.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.279+0000 7fdda8479740 1 client.16950 shutdown 2024-04-22T13:36:39.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.282+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 msgr2=0x55ff2d122730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.282+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd470 tx=0x7fdd94005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 shutdown_connections 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d139ef0 0x55ff2d13c310 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d122cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d122730 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 --2- 172.21.15.87:0/851031761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d1253a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 >> 172.21.15.87:0/851031761 conn(0x55ff2d126410 msgr2=0x55ff2d128160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdd837f6640 0 client.16950 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 shutdown_connections 2024-04-22T13:36:39.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.283+0000 7fdda8479740 1 -- 172.21.15.87:0/851031761 wait complete. 2024-04-22T13:36:39.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.284+0000 7fdda8479740 20 client.16950 trim_cache size 0 max 16384 2024-04-22T13:36:39.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f0033fff640 1 -- 172.21.15.87:0/2780800245 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380356a0 con 0x7f003c106560 2024-04-22T13:36:39.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 -- 172.21.15.87:0/2780800245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 msgr2=0x7f003c106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 --2- 172.21.15.87:0/2780800245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c106940 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f003800aa10 tx=0x7f003802cf70 comp rx=0 tx=0).stop 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 -- 172.21.15.87:0/2780800245 shutdown_connections 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 --2- 172.21.15.87:0/2780800245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c100e60 0x7f003c10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 --2- 172.21.15.87:0/2780800245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c1004a0 0x7f003c100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 --2- 172.21.15.87:0/2780800245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c106940 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.292+0000 7f004251c640 1 -- 172.21.15.87:0/2780800245 >> 172.21.15.87:0/2780800245 conn(0x7f003c0772e0 msgr2=0x7f003c0fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.293+0000 7f004251c640 1 -- 172.21.15.87:0/2780800245 shutdown_connections 2024-04-22T13:36:39.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.293+0000 7f004251c640 1 -- 172.21.15.87:0/2780800245 wait complete. 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 Processor -- start 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 -- start start 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c100e60 0x7f003c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f003c10fee0 con 0x7f003c106560 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f003c10fd60 con 0x7f003c1004a0 2024-04-22T13:36:39.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004251c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f003c110060 con 0x7f003c100e60 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004151a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004151a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48252/0 (socket says 172.21.15.87:48252) 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f0041d1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f004151a640 1 -- 172.21.15.87:0/2579309894 learned_addr learned my addr 172.21.15.87:0/2579309894 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.294+0000 7f0040d19640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c100e60 0x7f003c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 -- 172.21.15.87:0/2579309894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c100e60 msgr2=0x7f003c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c100e60 0x7f003c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 -- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 msgr2=0x7f003c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 -- 172.21.15.87:0/2579309894 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c19f720 con 0x7f003c1004a0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f004151a640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0038000c00 tx=0x7f00380048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f00327fc640 1 -- 172.21.15.87:0/2579309894 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038042020 con 0x7f003c1004a0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.295+0000 7f00327fc640 1 -- 172.21.15.87:0/2579309894 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0038004240 con 0x7f003c1004a0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.296+0000 7f00327fc640 1 -- 172.21.15.87:0/2579309894 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003803e6a0 con 0x7f003c1004a0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.296+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003c19f9b0 con 0x7f003c1004a0 2024-04-22T13:36:39.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.296+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003c1a72a0 con 0x7f003c1004a0 2024-04-22T13:36:39.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.298+0000 7f00327fc640 1 -- 172.21.15.87:0/2579309894 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0038045020 con 0x7f003c1004a0 2024-04-22T13:36:39.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 Processor -- start 2024-04-22T13:36:39.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 -- start start 2024-04-22T13:36:39.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff9551ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f00327fc640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 0x7f00280a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559ff94b61f0 con 0x559ff955ac00 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559ff94534b0 con 0x559ff9552080 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004636a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559ff94b21b0 con 0x559ff95516d0 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f0041d1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f004151a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f0041d1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55948/0 (socket says 172.21.15.87:55948) 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.299+0000 7f0041d1b640 1 -- 172.21.15.87:0/2440018569 learned_addr learned my addr 172.21.15.87:0/2440018569 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f0041d1b640 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff9551ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f0040d19640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 0x7f00280a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 -- 172.21.15.87:0/2440018569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 msgr2=0x559ff9551ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff9551ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 -- 172.21.15.87:0/2440018569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 msgr2=0x559ff955d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 -- 172.21.15.87:0/2440018569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff94451b0 con 0x559ff9552080 2024-04-22T13:36:39.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f00327fc640 1 -- 172.21.15.87:0/2579309894 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f00380f2b10 con 0x7f003c1004a0 2024-04-22T13:36:39.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f004151a640 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00380f03e0 tx=0x7f003802d490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea11602b0d3f8fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f00277fe640 1 -- 172.21.15.87:0/2440018569 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380a9040 con 0x559ff9552080 2024-04-22T13:36:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f00277fe640 1 -- 172.21.15.87:0/2440018569 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00380bf330 con 0x559ff9552080 2024-04-22T13:36:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.300+0000 7f00277fe640 1 -- 172.21.15.87:0/2440018569 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380f0550 con 0x559ff9552080 2024-04-22T13:36:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 -- 172.21.15.87:0/2440018569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 msgr2=0x559ff955a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f0040d19640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 0x7f00280a6290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f003c105bc0 tx=0x7f002c00b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00380f03e0 tx=0x7f003802d490 comp rx=0 tx=0).stop 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 -- 172.21.15.87:0/2440018569 shutdown_connections 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 --2- 172.21.15.87:0/2440018569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff9551ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 -- 172.21.15.87:0/2440018569 >> 172.21.15.87:0/2440018569 conn(0x559ff954bda0 msgr2=0x559ff954c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 -- 172.21.15.87:0/2440018569 shutdown_connections 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.301+0000 7f004636a740 1 -- 172.21.15.87:0/2440018569 wait complete. 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 -- start start 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 Processor -- start 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff9552080 0x559ff9557900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff955ac00 0x559ff9557f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559ff955d930 con 0x559ff9552080 2024-04-22T13:36:39.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559ff946f500 con 0x559ff95516d0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.302+0000 7f004636a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559ff955dab0 con 0x559ff955ac00 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48274/0 (socket says 172.21.15.87:48274) 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 -- 172.21.15.87:0/4107818467 learned_addr learned my addr 172.21.15.87:0/4107818467 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0040d19640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff9552080 0x559ff9557900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004636a740 10 client.0 _ll_register_callbacks cb 0x559ff943db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0041d1b640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff955ac00 0x559ff9557f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 -- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff955ac00 msgr2=0x559ff9557f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff955ac00 0x559ff9557f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 -- 172.21.15.87:0/4107818467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff9552080 msgr2=0x559ff9557900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff9552080 0x559ff9557900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 -- 172.21.15.87:0/4107818467 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff91c83f0 con 0x559ff95516d0 2024-04-22T13:36:39.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004151a640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00380bebc0 tx=0x7f0038003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bdfc8cfb273f160 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0024ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380a9040 con 0x559ff95516d0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f00380df0c0 con 0x559ff95516d0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00380bf330 con 0x559ff95516d0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.303+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f00380c2fe0 con 0x559ff95516d0 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.304+0000 7f004636a740 10 client.26090 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.304+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559ff92927d0 con 0x559ff95516d0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.304+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00380bf330 con 0x559ff95516d0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f003800b040 con 0x559ff95516d0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 10 client.26090 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 10 client.26090 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 20 client.26090 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff9573180 0x559ff9575570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 4 client.26090 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.305+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ff9455930 con 0x559ff9573180 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.306+0000 7f0040d19640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff9573180 0x559ff9575570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.306+0000 7f0024ff9640 10 client.26090 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.306+0000 7f0040d19640 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff9573180 0x559ff9575570 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23ba0ce488614f7f server_cookie=fdf66ec803f03473 in_seq=0 out_seq=0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f0024ff9640 1 -- 172.21.15.87:0/4107818467 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ff9455930 con 0x559ff9573180 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f0024ff9640 10 client.26090 handle_command_reply: tid=0 2024-04-22T13:36:39.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 client.26090 shutdown 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 msgr2=0x559ff955a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00380bebc0 tx=0x7f0038003c80 comp rx=0 tx=0).stop 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 shutdown_connections 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff9573180 0x559ff9575570 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff955ac00 0x559ff9557f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff9552080 0x559ff9557900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 --2- 172.21.15.87:0/4107818467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff95516d0 0x559ff955a570 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 >> 172.21.15.87:0/4107818467 conn(0x559ff954bda0 msgr2=0x559ff954daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f0024ff9640 0 client.26090 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.307+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 shutdown_connections 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.308+0000 7f004636a740 1 -- 172.21.15.87:0/4107818467 wait complete. 2024-04-22T13:36:39.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.308+0000 7f004636a740 20 client.26090 trim_cache size 0 max 16384 2024-04-22T13:36:39.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.312+0000 7fdda8479740 1 Processor -- start 2024-04-22T13:36:39.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.312+0000 7fdda8479740 1 -- start start 2024-04-22T13:36:39.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.312+0000 7fdda8479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d135b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.312+0000 7fdda8479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdda8479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 0x55ff2d124e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d135b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d135b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:49000/0 (socket says 172.21.15.87:49000) 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:48284/0 (socket says 172.21.15.87:48284) 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9effd640 1 -- 172.21.15.87:0/1458945123 learned_addr learned my addr 172.21.15.87:0/1458945123 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 0x55ff2d124e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98152eb0 con 0x55ff2d13dc50 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98150780 con 0x55ff2d12c180 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9814e6a0 con 0x55ff2d12b7d0 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1458945123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 msgr2=0x55ff2d135b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d135b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1458945123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 msgr2=0x55ff2d124e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 0x55ff2d124e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1458945123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2d123650 con 0x55ff2d12c180 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.313+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 0x55ff2d124e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.314+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdd940ba0f0 tx=0x7fdd94004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86188b1fd24baab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.314+0000 7fdd857fa640 1 -- 172.21.15.87:0/1458945123 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940da040 con 0x55ff2d12c180 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.314+0000 7fdd857fa640 1 -- 172.21.15.87:0/1458945123 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd940bde70 con 0x55ff2d12c180 2024-04-22T13:36:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.314+0000 7fdd857fa640 1 -- 172.21.15.87:0/1458945123 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940c1b20 con 0x55ff2d12c180 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 msgr2=0x55ff2d121670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdd940ba0f0 tx=0x7fdd94004610 comp rx=0 tx=0).stop 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 shutdown_connections 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d13dc50 0x55ff2d124e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d12c180 0x55ff2d121670 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 --2- 172.21.15.87:0/1458945123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d135b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 >> 172.21.15.87:0/1458945123 conn(0x55ff2d126410 msgr2=0x55ff2d1280f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 shutdown_connections 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- 172.21.15.87:0/1458945123 wait complete. 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 -- start start 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.315+0000 7fdda8479740 1 Processor -- start 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d132450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55980/0 (socket says 172.21.15.87:55980) 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d12d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d132450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:49012/0 (socket says 172.21.15.87:49012) 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1557266804 learned_addr learned my addr 172.21.15.87:0/1557266804 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9814ec60 con 0x55ff2d12c180 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981500b0 con 0x55ff2d13dc50 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9814f310 con 0x55ff2d12b7d0 2024-04-22T13:36:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9f7fe640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d12d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1557266804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 msgr2=0x55ff2d132450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d132450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 msgr2=0x55ff2d12d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d12d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdd9e7fc640 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff2d12c730 con 0x55ff2d12c180 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 10 client.0 _ll_register_callbacks cb 0x7fdd94009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.316+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd940bde70 con 0x55ff2d12c180 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.317+0000 7fdd9e7fc640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd700 tx=0x7fdd94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd70e1eac6558d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.317+0000 7fdd837f6640 10 client.16968 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.317+0000 7fdd837f6640 1 -- 172.21.15.87:0/1557266804 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940a1060 con 0x55ff2d12c180 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.317+0000 7fdd837f6640 1 -- 172.21.15.87:0/1557266804 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fdd9402c550 con 0x55ff2d12c180 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.318+0000 7fdd837f6640 1 -- 172.21.15.87:0/1557266804 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdd940da070 con 0x55ff2d12c180 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.318+0000 7fdda8479740 10 client.16968 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.318+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=23}) -- 0x55ff2d12c730 con 0x55ff2d12c180 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.318+0000 7fdd837f6640 1 -- 172.21.15.87:0/1557266804 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdd940f10f0 con 0x55ff2d12c180 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 10 client.16968 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 10 client.16968 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:36:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 20 client.16968 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d1a2da0 0x55ff2d1a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 4 client.16968 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7503ae6d", "release": true, "await": true}] 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7503ae6d", "release": true, "await": true}) -- 0x55ff2d095f20 con 0x55ff2d1a2da0 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.319+0000 7fdd9effd640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d1a2da0 0x55ff2d1a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.321+0000 7fdd9effd640 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d1a2da0 0x55ff2d1a51c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=115f9052ec7a8477 server_cookie=20c132b454cb68ed in_seq=0 out_seq=0 2024-04-22T13:36:39.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.321+0000 7fdd837f6640 10 client.16968 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdd837f6640 1 -- 172.21.15.87:0/1557266804 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ff2d095f20 con 0x55ff2d1a2da0 2024-04-22T13:36:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdd837f6640 10 client.16968 handle_command_reply: tid=0 2024-04-22T13:36:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdda8479740 1 client.16968 shutdown 2024-04-22T13:36:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 msgr2=0x55ff2d12d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdd940bd700 tx=0x7fdd94005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.324+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 shutdown_connections 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55ff2d1a2da0 0x55ff2d1a51c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ff2d13dc50 0x55ff2d12d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ff2d12c180 0x55ff2d12d060 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 --2- 172.21.15.87:0/1557266804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff2d12b7d0 0x55ff2d132450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 >> 172.21.15.87:0/1557266804 conn(0x55ff2d126410 msgr2=0x55ff2d13cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdd837f6640 0 client.16968 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 shutdown_connections 2024-04-22T13:36:39.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 1 -- 172.21.15.87:0/1557266804 wait complete. 2024-04-22T13:36:39.324 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:39.324 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:36:39.324 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 20, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "7503ae6d": { 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 20, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 90.0, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:36:39.325 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:39.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.325+0000 7fdda8479740 20 client.16968 trim_cache size 0 max 16384 2024-04-22T13:36:39.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.328+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 msgr2=0x7fdd880aa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.328+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 0x7fdd880aa7f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdda0065470 tx=0x7fdda0050bd0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.328+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 msgr2=0x7fdd9813db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.328+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd94037040 tx=0x7fdd94004270 comp rx=0 tx=0).stop 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 shutdown_connections 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdd880a8330 0x7fdd880aa7f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd98141ef0 0x7fdd981423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd980aa2a0 0x7fdd9813db70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 --2- 172.21.15.87:0/2816219617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd980a51d0 0x7fdd9813d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 >> 172.21.15.87:0/2816219617 conn(0x7fdd9801a670 msgr2=0x7fdd980a3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.331+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 shutdown_connections 2024-04-22T13:36:39.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.332+0000 7fdda5881640 1 -- 172.21.15.87:0/2816219617 wait complete. 2024-04-22T13:36:39.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.332+0000 7f004636a740 1 Processor -- start 2024-04-22T13:36:39.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.332+0000 7f004636a740 1 -- start start 2024-04-22T13:36:39.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.332+0000 7f004636a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004636a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004636a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004636a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1ade00 con 0x559ff955ac00 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004636a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1ab6d0 con 0x559ff9552080 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004636a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1a9560 con 0x559ff95516d0 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:49016/0 (socket says 172.21.15.87:49016) 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 -- 172.21.15.87:0/2596261787 learned_addr learned my addr 172.21.15.87:0/2596261787 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f0040d19640 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 -- 172.21.15.87:0/2596261787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 msgr2=0x559ff955a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 -- 172.21.15.87:0/2596261787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 msgr2=0x559ff955a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.333+0000 7f004151a640 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.334+0000 7f004151a640 1 -- 172.21.15.87:0/2596261787 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff9547670 con 0x559ff95516d0 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.334+0000 7f004151a640 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f00380c7370 tx=0x7f0038005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc6272d6153c680e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.334+0000 7f0026ffd640 1 -- 172.21.15.87:0/2596261787 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380a9040 con 0x559ff95516d0 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.334+0000 7f0026ffd640 1 -- 172.21.15.87:0/2596261787 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00380bf330 con 0x559ff95516d0 2024-04-22T13:36:39.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.334+0000 7f0026ffd640 1 -- 172.21.15.87:0/2596261787 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003800c0c0 con 0x559ff95516d0 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 -- 172.21.15.87:0/2596261787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 msgr2=0x559ff954aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f00380c7370 tx=0x7f0038005fd0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 -- 172.21.15.87:0/2596261787 shutdown_connections 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff955a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff955a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 --2- 172.21.15.87:0/2596261787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff954aca0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.336+0000 7f004636a740 1 -- 172.21.15.87:0/2596261787 >> 172.21.15.87:0/2596261787 conn(0x559ff954bda0 msgr2=0x559ff954a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.337+0000 7f004636a740 1 -- 172.21.15.87:0/2596261787 shutdown_connections 2024-04-22T13:36:39.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.337+0000 7f004636a740 1 -- 172.21.15.87:0/2596261787 wait complete. 2024-04-22T13:36:39.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.337+0000 7f004636a740 1 -- start start 2024-04-22T13:36:39.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.337+0000 7f004636a740 1 Processor -- start 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.339+0000 7f004636a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.339+0000 7f004636a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff9566ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004636a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff9567000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004636a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1a9bb0 con 0x559ff955ac00 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004636a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1ab000 con 0x559ff9552080 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004636a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f003c1aa260 con 0x559ff95516d0 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004636a740 10 client.0 _ll_register_callbacks cb 0x7f00380bea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004151a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff9566ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004151a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:49026/0 (socket says 172.21.15.87:49026) 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004151a640 1 -- 172.21.15.87:0/1052939624 learned_addr learned my addr 172.21.15.87:0/1052939624 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 -- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 msgr2=0x559ff956b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 -- 172.21.15.87:0/1052939624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 msgr2=0x559ff9567000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff9567000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 -- 172.21.15.87:0/1052939624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff9567740 con 0x559ff9552080 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f004151a640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.340+0000 7f0040d19640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff9566ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x559ff9575cd0 tx=0x7f002c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0c9fd2a5709310 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.341+0000 7f0024ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.342+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c005220 con 0x559ff9552080 2024-04-22T13:36:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.342+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f002c0053c0 con 0x559ff9552080 2024-04-22T13:36:39.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.342+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00380bf330 con 0x559ff9552080 2024-04-22T13:36:39.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.343+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f002c0055e0 con 0x559ff9552080 2024-04-22T13:36:39.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.343+0000 7f004636a740 10 client.26102 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.343+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559ff95d9500 con 0x559ff9552080 2024-04-22T13:36:39.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.343+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c02b070 con 0x559ff9552080 2024-04-22T13:36:39.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.343+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f002c027440 con 0x559ff9552080 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 10 client.26102 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 10 client.26102 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 20 client.26102 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff95d9c20 0x559ff95dc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 4 client.26102 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:36:39.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.344+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f00380bf330 con 0x559ff95d9c20 2024-04-22T13:36:39.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.345+0000 7f004151a640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff95d9c20 0x559ff95dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.349+0000 7f004151a640 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff95d9c20 0x559ff95dc010 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=261efce8e060bf1b server_cookie=1d21be90304566ce in_seq=0 out_seq=0 2024-04-22T13:36:39.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.349+0000 7f0024ff9640 10 client.26102 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:36:39.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.350+0000 7f0024ff9640 1 -- 172.21.15.87:0/1052939624 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f00380bf330 con 0x559ff95d9c20 2024-04-22T13:36:39.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.350+0000 7f0024ff9640 10 client.26102 handle_command_reply: tid=0 2024-04-22T13:36:39.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.350+0000 7f004636a740 1 client.26102 shutdown 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 msgr2=0x559ff9566ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff9566ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x559ff9575cd0 tx=0x7f002c013000 comp rx=0 tx=0).stop 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 shutdown_connections 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x559ff95d9c20 0x559ff95dc010 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559ff955ac00 0x559ff9567000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ff9552080 0x559ff9566ac0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 --2- 172.21.15.87:0/1052939624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ff95516d0 0x559ff956b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f0024ff9640 0 client.26102 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:36:39.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 >> 172.21.15.87:0/1052939624 conn(0x559ff954bda0 msgr2=0x559ff954ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 shutdown_connections 2024-04-22T13:36:39.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.351+0000 7f004636a740 1 -- 172.21.15.87:0/1052939624 wait complete. 2024-04-22T13:36:39.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.352+0000 7f004636a740 20 client.26102 trim_cache size 0 max 16384 2024-04-22T13:36:39.350 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:36:39.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.357+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 msgr2=0x7f00280a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.357+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 0x7f00280a6290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f003c105bc0 tx=0x7f002c00b390 comp rx=0 tx=0).stop 2024-04-22T13:36:39.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.357+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 msgr2=0x7f003c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.357+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0038000c00 tx=0x7f00380048f0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 shutdown_connections 2024-04-22T13:36:39.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f00280a3dd0 0x7f00280a6290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f003c106560 0x7f003c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f003c100e60 0x7f003c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 --2- 172.21.15.87:0/2579309894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f003c1004a0 0x7f003c19a730 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 >> 172.21.15.87:0/2579309894 conn(0x7f003c0772e0 msgr2=0x7f003c0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.361+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 shutdown_connections 2024-04-22T13:36:39.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.362+0000 7f004251c640 1 -- 172.21.15.87:0/2579309894 wait complete. 2024-04-22T13:36:39.359 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 7503ae6d, seconds elapsed: 42.8 2024-04-22T13:36:39.359 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-22T13:36:39.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:36:39.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 -- 172.21.15.87:0/2518097045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 msgr2=0x7f3bc4106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 --2- 172.21.15.87:0/2518097045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc4106210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009890 tx=0x7f3bb402bb90 comp rx=0 tx=0).stop 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 -- 172.21.15.87:0/2518097045 shutdown_connections 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 --2- 172.21.15.87:0/2518097045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc4101b60 0x7f3bc410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 --2- 172.21.15.87:0/2518097045 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc41011a0 0x7f3bc4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 --2- 172.21.15.87:0/2518097045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc4106210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 -- 172.21.15.87:0/2518097045 >> 172.21.15.87:0/2518097045 conn(0x7f3bc4077e30 msgr2=0x7f3bc4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.606+0000 7f3bc8b84640 1 -- 172.21.15.87:0/2518097045 shutdown_connections 2024-04-22T13:36:39.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.607+0000 7f3bc8b84640 1 -- 172.21.15.87:0/2518097045 wait complete. 2024-04-22T13:36:39.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.608+0000 7f3bc8b84640 1 Processor -- start 2024-04-22T13:36:39.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.608+0000 7f3bc8b84640 1 -- start start 2024-04-22T13:36:39.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.608+0000 7f3bc8b84640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.608+0000 7f3bc8b84640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 0x7f3bc4196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.608+0000 7f3bc8b84640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc41968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc1d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 0x7f3bc4196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50238/0 (socket says 172.21.15.87:50238) 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc259c640 1 -- 172.21.15.87:0/1538347398 learned_addr learned my addr 172.21.15.87:0/1538347398 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc2d9d640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc41968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc410ffe0 con 0x7f3bc4101b60 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc410fe60 con 0x7f3bc41011a0 2024-04-22T13:36:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4110160 con 0x7f3bc4105e30 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc2d9d640 1 -- 172.21.15.87:0/1538347398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 msgr2=0x7f3bc419d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc2d9d640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc2d9d640 1 -- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 msgr2=0x7f3bc4196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.609+0000 7f3bc2d9d640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 0x7f3bc4196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc2d9d640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc4196ee0 con 0x7f3bc4105e30 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc259c640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc1d9b640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 0x7f3bc4196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc2d9d640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc41968a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00da10 tx=0x7f3bbc00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc01b020 con 0x7f3bc4105e30 2024-04-22T13:36:39.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bbc00bb70 con 0x7f3bc4105e30 2024-04-22T13:36:39.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc419ae00 con 0x7f3bc4105e30 2024-04-22T13:36:39.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc419b350 con 0x7f3bc4105e30 2024-04-22T13:36:39.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.610+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0058b0 con 0x7f3bc4105e30 2024-04-22T13:36:39.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.612+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba4005290 con 0x7f3bc4105e30 2024-04-22T13:36:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.614+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3bbc004b20 con 0x7f3bc4105e30 2024-04-22T13:36:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.615+0000 7f3bb37fe640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 0x7f3bac0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.615+0000 7f3bc259c640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 0x7f3bac0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.615+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3bbc0d0300 con 0x7f3bc4105e30 2024-04-22T13:36:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.616+0000 7f3bc259c640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 0x7f3bac0a62e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4005e00 tx=0x7f3bb4009390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.620+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bbc025020 con 0x7f3bc4105e30 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 -- 172.21.15.87:0/2310289357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa250101090 msgr2=0x7fa250101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 --2- 172.21.15.87:0/2310289357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa250101090 0x7fa250101470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa238009900 tx=0x7fa23802bc30 comp rx=0 tx=0).stop 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 -- 172.21.15.87:0/2310289357 shutdown_connections 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 --2- 172.21.15.87:0/2310289357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250106160 0x7fa25010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 --2- 172.21.15.87:0/2310289357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250101a40 0x7fa250105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 --2- 172.21.15.87:0/2310289357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa250101090 0x7fa250101470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 -- 172.21.15.87:0/2310289357 >> 172.21.15.87:0/2310289357 conn(0x7fa250077f50 msgr2=0x7fa250078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 -- 172.21.15.87:0/2310289357 shutdown_connections 2024-04-22T13:36:39.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.647+0000 7fa25487b640 1 -- 172.21.15.87:0/2310289357 wait complete. 2024-04-22T13:36:39.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.648+0000 7fa25487b640 1 Processor -- start 2024-04-22T13:36:39.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 -- start start 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 0x7fa25019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2501a3640 0x7fa2501a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa250110010 con 0x7fa250101090 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010fe90 con 0x7fa250106160 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa25487b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa250110190 con 0x7fa2501a3640 2024-04-22T13:36:39.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50258/0 (socket says 172.21.15.87:50258) 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 0x7fa25019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.649+0000 7fa24dd9b640 1 -- 172.21.15.87:0/1343597434 learned_addr learned my addr 172.21.15.87:0/1343597434 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2501a3640 0x7fa2501a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 -- 172.21.15.87:0/1343597434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2501a3640 msgr2=0x7fa2501a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2501a3640 0x7fa2501a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 -- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 msgr2=0x7fa25019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 0x7fa25019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 -- 172.21.15.87:0/1343597434 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2501a4130 con 0x7fa250106160 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24e59c640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 0x7fa25019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e9f0 tx=0x7fa24000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.650+0000 7fa2477fe640 1 -- 172.21.15.87:0/1343597434 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000ccc0 con 0x7fa250106160 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.651+0000 7fa2477fe640 1 -- 172.21.15.87:0/1343597434 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2400054d0 con 0x7fa250106160 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.651+0000 7fa2477fe640 1 -- 172.21.15.87:0/1343597434 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240010430 con 0x7fa250106160 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.652+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2501a4420 con 0x7fa250106160 2024-04-22T13:36:39.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.652+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2501abbc0 con 0x7fa250106160 2024-04-22T13:36:39.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.655+0000 7fa2477fe640 1 -- 172.21.15.87:0/1343597434 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa2400046c0 con 0x7fa250106160 2024-04-22T13:36:39.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.656+0000 7fa257472740 1 Processor -- start 2024-04-22T13:36:39.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.656+0000 7fa257472740 1 -- start start 2024-04-22T13:36:39.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.657+0000 7fa2477fe640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 0x7fa23c0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.657+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 0x7fa23c0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.657+0000 7fa2477fe640 1 -- 172.21.15.87:0/1343597434 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fa2400d0c10 con 0x7fa250106160 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa257472740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa257472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa60a2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa257472740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 0x7fa23c0a62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa248005fd0 tx=0x7fa248009500 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51486/0 (socket says 172.21.15.87:51486) 2024-04-22T13:36:39.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24e59c640 1 -- 172.21.15.87:0/247727852 learned_addr learned my addr 172.21.15.87:0/247727852 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa60a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55afa5ffe1f0 con 0x55afa60a2a00 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.658+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55afa5f9b4b0 con 0x55afa6099e80 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55afa5ffa1b0 con 0x55afa60994d0 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24dd9b640 1 -- 172.21.15.87:0/247727852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 msgr2=0x55afa60998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24dd9b640 1 -- 172.21.15.87:0/247727852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 msgr2=0x55afa60a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24dd9b640 1 -- 172.21.15.87:0/247727852 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afa5f8d1b0 con 0x55afa6099e80 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.659+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.660+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa60a2430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa2400ce370 tx=0x7fa24009c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf776e90161d0aec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.660+0000 7fa244ff9640 1 -- 172.21.15.87:0/247727852 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2400a0f70 con 0x55afa6099e80 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.660+0000 7fa244ff9640 1 -- 172.21.15.87:0/247727852 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2400a5830 con 0x55afa6099e80 2024-04-22T13:36:39.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.660+0000 7fa244ff9640 1 -- 172.21.15.87:0/247727852 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240005000 con 0x55afa6099e80 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 msgr2=0x55afa60a2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa60a2430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa2400ce370 tx=0x7fa24009c610 comp rx=0 tx=0).stop 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 shutdown_connections 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa60a2430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 --2- 172.21.15.87:0/247727852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.661+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 >> 172.21.15.87:0/247727852 conn(0x55afa6093ba0 msgr2=0x55afa6093f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.662+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 shutdown_connections 2024-04-22T13:36:39.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.662+0000 7fa257472740 1 -- 172.21.15.87:0/247727852 wait complete. 2024-04-22T13:36:39.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.662+0000 7fa257472740 1 -- start start 2024-04-22T13:36:39.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.663+0000 7fa257472740 1 Processor -- start 2024-04-22T13:36:39.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.663+0000 7fa257472740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa257472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa609f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa257472740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51494/0 (socket says 172.21.15.87:51494) 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa609f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa609f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50280/0 (socket says 172.21.15.87:50280) 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35370/0 (socket says 172.21.15.87:35370) 2024-04-22T13:36:39.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa24e59c640 1 -- 172.21.15.87:0/1345818580 learned_addr learned my addr 172.21.15.87:0/1345818580 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.664+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55afa60a58b0 con 0x55afa60a2a00 2024-04-22T13:36:39.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55afa5fb7500 con 0x55afa6099e80 2024-04-22T13:36:39.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55afa60a5a30 con 0x55afa60994d0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24e59c640 1 -- 172.21.15.87:0/1345818580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 msgr2=0x55afa609f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24e59c640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa609f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24e59c640 1 -- 172.21.15.87:0/1345818580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 msgr2=0x55afa609fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24e59c640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24e59c640 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afa5d103f0 con 0x55afa60994d0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.665+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa24e59c640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa23802c170 tx=0x7fa23802ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61770fb819775f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa257472740 10 client.0 _ll_register_callbacks cb 0x55afa5f85b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa235ffb640 1 -- 172.21.15.87:0/1345818580 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23802c5f0 con 0x55afa60994d0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa235ffb640 1 -- 172.21.15.87:0/1345818580 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fa2380035f0 con 0x55afa60994d0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2400a5830 con 0x55afa60994d0 2024-04-22T13:36:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa235ffb640 1 -- 172.21.15.87:0/1345818580 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa2380037f0 con 0x55afa60994d0 2024-04-22T13:36:39.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa257472740 10 client.26167 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.666+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({fsmap=23}) -- 0x55afa5dda7d0 con 0x55afa60994d0 2024-04-22T13:36:39.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa235ffb640 1 -- 172.21.15.87:0/1345818580 <== mon.1 v2:172.21.15.132:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa238003990 con 0x55afa60994d0 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 10 client.26167 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 10 client.26167 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 20 client.26167 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa60b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 4 client.26167 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa2400a5830 con 0x55afa60b27f0 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.667+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa60b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.668+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa60b4c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2105779e57cb12d7 server_cookie=39c4a821d4e2be62 in_seq=0 out_seq=0 2024-04-22T13:36:39.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.668+0000 7fa235ffb640 10 client.26167 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.668+0000 7fa235ffb640 1 -- 172.21.15.87:0/1345818580 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa2400a5830 con 0x55afa60b27f0 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.668+0000 7fa235ffb640 10 client.26167 handle_command_reply: tid=0 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.669+0000 7fa257472740 1 client.26167 shutdown 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.669+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 msgr2=0x55afa60a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.669+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa23802c170 tx=0x7fa23802ca10 comp rx=0 tx=0).stop 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.669+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 shutdown_connections 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.669+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa60b4c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa235ffb640 0 client.26167 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa609fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa6099e80 0x55afa609f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 --2- 172.21.15.87:0/1345818580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60994d0 0x55afa60a22a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 >> 172.21.15.87:0/1345818580 conn(0x55afa6093ba0 msgr2=0x55afa60958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 shutdown_connections 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 1 -- 172.21.15.87:0/1345818580 wait complete. 2024-04-22T13:36:39.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.670+0000 7fa257472740 20 client.26167 trim_cache size 0 max 16384 2024-04-22T13:36:39.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.700+0000 7fa257472740 1 Processor -- start 2024-04-22T13:36:39.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.700+0000 7fa257472740 1 -- start start 2024-04-22T13:36:39.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.701+0000 7fa257472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.701+0000 7fa257472740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.701+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.701+0000 7fa24e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50296/0 (socket says 172.21.15.87:50296) 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35374/0 (socket says 172.21.15.87:35374) 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa24e59c640 1 -- 172.21.15.87:0/4153432905 learned_addr learned my addr 172.21.15.87:0/4153432905 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa257472740 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60a2a00 0x55afa60a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b69c0 con 0x55afa6099e80 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b4290 con 0x55afa60994d0 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b2100 con 0x55afa60a2a00 2024-04-22T13:36:39.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.702+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60a2a00 0x55afa60a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 -- 172.21.15.87:0/4153432905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60a2a00 msgr2=0x55afa60a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60a2a00 0x55afa60a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 -- 172.21.15.87:0/4153432905 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 msgr2=0x55afa60ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 -- 172.21.15.87:0/4153432905 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afa609b200 con 0x55afa60994d0 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:39.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.703+0000 7fa24e59c640 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa238009230 tx=0x7fa238042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21f5e4bea5497cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.704+0000 7fa237fff640 1 -- 172.21.15.87:0/4153432905 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238058070 con 0x55afa60994d0 2024-04-22T13:36:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.704+0000 7fa237fff640 1 -- 172.21.15.87:0/4153432905 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23802c5f0 con 0x55afa60994d0 2024-04-22T13:36:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.704+0000 7fa237fff640 1 -- 172.21.15.87:0/4153432905 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238034440 con 0x55afa60994d0 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 msgr2=0x55afa6092e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa238009230 tx=0x7fa238042000 comp rx=0 tx=0).stop 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 shutdown_connections 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa60a2a00 0x55afa60a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa6099e80 0x55afa60ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 --2- 172.21.15.87:0/4153432905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa6092e40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 >> 172.21.15.87:0/4153432905 conn(0x55afa6093ba0 msgr2=0x55afa609aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 shutdown_connections 2024-04-22T13:36:39.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.705+0000 7fa257472740 1 -- 172.21.15.87:0/4153432905 wait complete. 2024-04-22T13:36:39.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.706+0000 7fa257472740 1 -- start start 2024-04-22T13:36:39.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.706+0000 7fa257472740 1 Processor -- start 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.706+0000 7fa257472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa60aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.706+0000 7fa257472740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.706+0000 7fa257472740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa257472740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b2750 con 0x55afa60a2a00 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa257472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b3bc0 con 0x55afa60994d0 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa257472740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa2501b2e20 con 0x55afa6099e80 2024-04-22T13:36:39.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa257472740 10 client.0 _ll_register_callbacks cb 0x7fa2380043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51502/0 (socket says 172.21.15.87:51502) 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 -- 172.21.15.87:0/3865824366 learned_addr learned my addr 172.21.15.87:0/3865824366 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24e59c640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa60aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 -- 172.21.15.87:0/3865824366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 msgr2=0x55afa60aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa60aacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 -- 172.21.15.87:0/3865824366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 msgr2=0x55afa60a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.707+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.708+0000 7fa24dd9b640 1 -- 172.21.15.87:0/3865824366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afa60a6bc0 con 0x55afa6099e80 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.708+0000 7fa24ed9d640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:39.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.708+0000 7fa235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:36:39.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.708+0000 7fa24dd9b640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa240005820 tx=0x7fa24009cac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd995a7a87a44ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:39.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.712+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2400a5830 con 0x55afa6099e80 2024-04-22T13:36:39.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.712+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fa2400bd0b0 con 0x55afa6099e80 2024-04-22T13:36:39.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.712+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55afa5f9d930 con 0x55afa6099e80 2024-04-22T13:36:39.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.716+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa2400ce5f0 con 0x55afa6099e80 2024-04-22T13:36:39.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.716+0000 7fa257472740 10 client.26173 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:39.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.716+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55afa6118a90 con 0x55afa6099e80 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2400ce790 con 0x55afa6099e80 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa2400989a0 con 0x55afa6099e80 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 10 client.26173 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 10 client.26173 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 20 client.26173 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa611ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:39.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 4 client.26173 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:36:39.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.717+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afa5f9d930 con 0x55afa60b27f0 2024-04-22T13:36:39.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa24e59c640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa611ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa24e59c640 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa611ad30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b54905e045205b8a server_cookie=8388502ec6714b67 in_seq=0 out_seq=0 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa235ffb640 10 client.26173 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa235ffb640 1 -- 172.21.15.87:0/3865824366 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afa5f9d930 con 0x55afa60b27f0 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa235ffb640 10 client.26173 handle_command_reply: tid=0 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.718+0000 7fa257472740 1 client.26173 shutdown 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 msgr2=0x55afa60ab210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa240005820 tx=0x7fa24009cac0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 shutdown_connections 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55afa60b27f0 0x55afa611ad30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55afa60a2a00 0x55afa60a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afa6099e80 0x55afa60ab210 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 --2- 172.21.15.87:0/3865824366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55afa60994d0 0x55afa60aacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 >> 172.21.15.87:0/3865824366 conn(0x55afa6093ba0 msgr2=0x55afa6094d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa235ffb640 0 client.26173 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 shutdown_connections 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 1 -- 172.21.15.87:0/3865824366 wait complete. 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.719+0000 7fa257472740 20 client.26173 trim_cache size 0 max 16384 2024-04-22T13:36:39.717 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:36:39.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.720+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 msgr2=0x7fa23c0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.720+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 0x7fa23c0a62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa248005fd0 tx=0x7fa248009500 comp rx=0 tx=0).stop 2024-04-22T13:36:39.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.720+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 msgr2=0x7fa25019f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:39.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.720+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e9f0 tx=0x7fa24000eeb0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 shutdown_connections 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa23c0a3e00 0x7fa23c0a62c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2501a3640 0x7fa2501a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa250106160 0x7fa25019f2c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 --2- 172.21.15.87:0/1343597434 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa250101090 0x7fa25019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 >> 172.21.15.87:0/1343597434 conn(0x7fa250077f50 msgr2=0x7fa2500ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 shutdown_connections 2024-04-22T13:36:39.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:39.721+0000 7fa25487b640 1 -- 172.21.15.87:0/1343597434 wait complete. 2024-04-22T13:36:39.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:36:39.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:39 smithi132 ceph-mon[31531]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 279 op/s 2024-04-22T13:36:39.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:39 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2963440348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:39.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:39 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3169212065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:39.992 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:39 smithi087 ceph-mon[27999]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 279 op/s 2024-04-22T13:36:39.992 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:39 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2963440348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:39.992 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:39 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3169212065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:40.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.149+0000 7f0424715640 1 -- 172.21.15.87:0/1487038871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c11b730 msgr2=0x7f041c11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.149+0000 7f0424715640 1 --2- 172.21.15.87:0/1487038871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c11b730 0x7f041c11db20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0418009900 tx=0x7f041802bc60 comp rx=0 tx=0).stop 2024-04-22T13:36:40.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.154+0000 7f0424715640 1 -- 172.21.15.87:0/1487038871 shutdown_connections 2024-04-22T13:36:40.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.154+0000 7f0424715640 1 --2- 172.21.15.87:0/1487038871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c11b730 0x7f041c11db20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.154+0000 7f0424715640 1 --2- 172.21.15.87:0/1487038871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c10e7d0 0x7f041c10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.154+0000 7f0424715640 1 --2- 172.21.15.87:0/1487038871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c1107a0 0x7f041c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.154+0000 7f0424715640 1 -- 172.21.15.87:0/1487038871 >> 172.21.15.87:0/1487038871 conn(0x7f041c06d060 msgr2=0x7f041c06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.155+0000 7f0424715640 1 -- 172.21.15.87:0/1487038871 shutdown_connections 2024-04-22T13:36:40.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.163+0000 7f0424715640 1 -- 172.21.15.87:0/1487038871 wait complete. 2024-04-22T13:36:40.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.164+0000 7f0424715640 1 Processor -- start 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 -- start start 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10e7d0 0x7f041c1a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c11b730 0x7f041c1abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f041c120f40 con 0x7f041c1107a0 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f041c120dc0 con 0x7f041c11b730 2024-04-22T13:36:40.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.168+0000 7f0424715640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f041c1210c0 con 0x7f041c10e7d0 2024-04-22T13:36:40.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.172+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.172+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35394/0 (socket says 172.21.15.87:35394) 2024-04-22T13:36:40.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.172+0000 7f0421cb0640 1 -- 172.21.15.87:0/347133450 learned_addr learned my addr 172.21.15.87:0/347133450 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.173+0000 7f0422cb2640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c11b730 0x7f041c1abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.173+0000 7f04224b1640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10e7d0 0x7f041c1a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:39 smithi120 ceph-mon[31101]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 279 op/s 2024-04-22T13:36:40.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:39 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2963440348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:40.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:39 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3169212065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:40.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.180+0000 7f0421cb0640 1 -- 172.21.15.87:0/347133450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10e7d0 msgr2=0x7f041c1a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.180+0000 7f0421cb0640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10e7d0 0x7f041c1a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.180+0000 7f0421cb0640 1 -- 172.21.15.87:0/347133450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c11b730 msgr2=0x7f041c1abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.180+0000 7f0421cb0640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c11b730 0x7f041c1abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.180+0000 7f0421cb0640 1 -- 172.21.15.87:0/347133450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041c1ac490 con 0x7f041c1107a0 2024-04-22T13:36:40.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.190+0000 7f0421cb0640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d950 tx=0x7f040c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.197+0000 7f04137fe640 1 -- 172.21.15.87:0/347133450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c017070 con 0x7f041c1107a0 2024-04-22T13:36:40.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.198+0000 7f04137fe640 1 -- 172.21.15.87:0/347133450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f040c009c30 con 0x7f041c1107a0 2024-04-22T13:36:40.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.198+0000 7f04137fe640 1 -- 172.21.15.87:0/347133450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c00f620 con 0x7f041c1107a0 2024-04-22T13:36:40.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.203+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041c1ac780 con 0x7f041c1107a0 2024-04-22T13:36:40.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.203+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041c1b4010 con 0x7f041c1107a0 2024-04-22T13:36:40.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.204+0000 7f3bc8b84640 1 -- 172.21.15.87:0/1538347398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ba4005850 con 0x7f3bc4105e30 2024-04-22T13:36:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:36:40.217 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:36:40.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.206+0000 7f3bb37fe640 1 -- 172.21.15.87:0/1538347398 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f3bbc09ca40 con 0x7f3bc4105e30 2024-04-22T13:36:40.217 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.207+0000 7f04273e2740 1 Processor -- start 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.207+0000 7f04273e2740 1 -- start start 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e06430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e08e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561fc6d621f0 con 0x561fc6dfde80 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561fc6cff4b0 con 0x561fc6dfd4d0 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.208+0000 7f04273e2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561fc6d5e1b0 con 0x561fc6e06a00 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50332/0 (socket says 172.21.15.87:50332) 2024-04-22T13:36:40.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 -- 172.21.15.87:0/3823342531 learned_addr learned my addr 172.21.15.87:0/3823342531 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 -- 172.21.15.87:0/3823342531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 msgr2=0x561fc6e08e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e08e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 -- 172.21.15.87:0/3823342531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 msgr2=0x561fc6e06430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e06430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.209+0000 7f0422cb2640 1 -- 172.21.15.87:0/3823342531 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc6cf11b0 con 0x561fc6dfd4d0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.210+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f041802c270 tx=0x7f041802c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfcfddf370ad288a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.210+0000 7f04137fe640 1 -- 172.21.15.87:0/347133450 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f040c00f7c0 con 0x7f041c1107a0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.211+0000 7f04137fe640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 0x7f04080a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.212+0000 7f04224b1640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 0x7f04080a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.212+0000 7f04137fe640 1 -- 172.21.15.87:0/347133450 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f040c0d0060 con 0x7f041c1107a0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.214+0000 7f04224b1640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 0x7f04080a6360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x561fc6df8c10 tx=0x7f041400a990 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.215+0000 7f0410ff9640 1 -- 172.21.15.87:0/3823342531 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418045020 con 0x561fc6dfd4d0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.215+0000 7f0410ff9640 1 -- 172.21.15.87:0/3823342531 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f041802ca10 con 0x561fc6dfd4d0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.215+0000 7f0410ff9640 1 -- 172.21.15.87:0/3823342531 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04180046c0 con 0x561fc6dfd4d0 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 -- 172.21.15.87:0/3823342531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 msgr2=0x561fc6dfd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f041802c270 tx=0x7f041802c7e0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 -- 172.21.15.87:0/3823342531 shutdown_connections 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e08e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e06430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 --2- 172.21.15.87:0/3823342531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfd4d0 0x561fc6dfd8b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.216+0000 7f04273e2740 1 -- 172.21.15.87:0/3823342531 >> 172.21.15.87:0/3823342531 conn(0x561fc6df7ba0 msgr2=0x561fc6df7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.219+0000 7f04273e2740 1 -- 172.21.15.87:0/3823342531 shutdown_connections 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.219+0000 7f04273e2740 1 -- 172.21.15.87:0/3823342531 wait complete. 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.220+0000 7f04273e2740 1 -- start start 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.220+0000 7f04273e2740 1 Processor -- start 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.220+0000 7f04273e2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfd4d0 0x561fc6e06300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6e03c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561fc6e09910 con 0x561fc6dfde80 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561fc6d1b500 con 0x561fc6e06a00 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35402/0 (socket says 172.21.15.87:35402) 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f0421cb0640 1 -- 172.21.15.87:0/4243014362 learned_addr learned my addr 172.21.15.87:0/4243014362 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561fc6e09a90 con 0x561fc6dfd4d0 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f0422cb2640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6e03c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.221+0000 7f04273e2740 10 client.0 _ll_register_callbacks cb 0x561fc6ce9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 msgr2=0x7f3bac0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 0x7f3bac0a62e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4005e00 tx=0x7f3bb4009390 comp rx=0 tx=0).stop 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f04224b1640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfd4d0 0x561fc6e06300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 msgr2=0x7f3bc41968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc41968a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00da10 tx=0x7f3bbc00ded0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0421cb0640 1 -- 172.21.15.87:0/4243014362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfd4d0 msgr2=0x561fc6e06300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfd4d0 0x561fc6e06300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0421cb0640 1 -- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 msgr2=0x561fc6e03c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6e03c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0421cb0640 1 -- 172.21.15.87:0/4243014362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc6a743f0 con 0x561fc6dfde80 2024-04-22T13:36:40.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f0422cb2640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6e03c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:40.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.226+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f040c009610 tx=0x7f040c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12cf4715817d24d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.227+0000 7f0405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 shutdown_connections 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bac0a3e20 0x7f3bac0a62e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bc4105e30 0x7f3bc41968a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bc4101b60 0x7f3bc4196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 --2- 172.21.15.87:0/1538347398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bc41011a0 0x7f3bc419d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.224+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 >> 172.21.15.87:0/1538347398 conn(0x7f3bc4077e30 msgr2=0x7f3bc40feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.227+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 shutdown_connections 2024-04-22T13:36:40.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.227+0000 7f3bb17fa640 1 -- 172.21.15.87:0/1538347398 wait complete. 2024-04-22T13:36:40.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.230+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c020050 con 0x561fc6dfde80 2024-04-22T13:36:40.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.230+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f040c098830 con 0x561fc6dfde80 2024-04-22T13:36:40.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.230+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041802ca10 con 0x561fc6dfde80 2024-04-22T13:36:40.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.232+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f040c0a04c0 con 0x561fc6dfde80 2024-04-22T13:36:40.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.241+0000 7f04273e2740 10 client.17001 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:40.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.241+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561fc6b3e7d0 con 0x561fc6dfde80 2024-04-22T13:36:40.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.242+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f041802ca10 con 0x561fc6dfde80 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.243+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f040c097f10 con 0x561fc6dfde80 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 10 client.17001 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 20 client.17001 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e16bb0 0x561fc6e18fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 4 client.17001 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fc6d01930 con 0x561fc6e16bb0 2024-04-22T13:36:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.246+0000 7f04224b1640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e16bb0 0x561fc6e18fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.253+0000 7f04224b1640 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e16bb0 0x561fc6e18fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a72af8a8cfacd2de server_cookie=49f87de40a416261 in_seq=0 out_seq=0 2024-04-22T13:36:40.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.254+0000 7f0405ffb640 10 client.17001 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:40.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.256+0000 7f0405ffb640 1 -- 172.21.15.87:0/4243014362 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fc6d01930 con 0x561fc6e16bb0 2024-04-22T13:36:40.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.256+0000 7f0405ffb640 10 client.17001 handle_command_reply: tid=0 2024-04-22T13:36:40.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.256+0000 7f04273e2740 1 client.17001 shutdown 2024-04-22T13:36:40.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.258+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 msgr2=0x561fc6e03690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.258+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f040c009610 tx=0x7f040c009de0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 shutdown_connections 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e16bb0 0x561fc6e18fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6e03c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfde80 0x561fc6e03690 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 --2- 172.21.15.87:0/4243014362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfd4d0 0x561fc6e06300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.259+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 >> 172.21.15.87:0/4243014362 conn(0x561fc6df7ba0 msgr2=0x561fc6df98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.260+0000 7f0405ffb640 0 client.17001 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:40.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.260+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 shutdown_connections 2024-04-22T13:36:40.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.262+0000 7f04273e2740 1 -- 172.21.15.87:0/4243014362 wait complete. 2024-04-22T13:36:40.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.263+0000 7f04273e2740 20 client.17001 trim_cache size 0 max 16384 2024-04-22T13:36:40.281 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 90.0 --await' 2024-04-22T13:36:40.281 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 90.0 --await 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 Processor -- start 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 -- start start 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e12860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfde80 0x561fc6e02870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0400010cd0 con 0x561fc6dfd4d0 2024-04-22T13:36:40.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f040000e5a0 con 0x561fc6e06a00 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f04273e2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f040000c4a0 con 0x561fc6dfde80 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f0422cb2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f0422cb2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50350/0 (socket says 172.21.15.87:50350) 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.314+0000 7f0422cb2640 1 -- 172.21.15.87:0/3791392710 learned_addr learned my addr 172.21.15.87:0/3791392710 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.315+0000 7f0421cb0640 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfde80 0x561fc6e02870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.317+0000 7f0422cb2640 1 -- 172.21.15.87:0/3791392710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfde80 msgr2=0x561fc6e02870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.317+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfde80 0x561fc6e02870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.317+0000 7f0422cb2640 1 -- 172.21.15.87:0/3791392710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 msgr2=0x561fc6e12860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.317+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e12860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.317+0000 7f0422cb2640 1 -- 172.21.15.87:0/3791392710 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc6e03f80 con 0x561fc6e06a00 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.318+0000 7f0422cb2640 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0418009900 tx=0x7f041802c600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4833b205d30b96bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.323+0000 7f0407fff640 1 -- 172.21.15.87:0/3791392710 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418045030 con 0x561fc6e06a00 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.323+0000 7f0407fff640 1 -- 172.21.15.87:0/3791392710 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0418004490 con 0x561fc6e06a00 2024-04-22T13:36:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.323+0000 7f0407fff640 1 -- 172.21.15.87:0/3791392710 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04180035f0 con 0x561fc6e06a00 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 -- 172.21.15.87:0/3791392710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 msgr2=0x561fc6df6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0418009900 tx=0x7f041802c600 comp rx=0 tx=0).stop 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 -- 172.21.15.87:0/3791392710 shutdown_connections 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6e06a00 0x561fc6df6a80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6dfde80 0x561fc6e02870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 --2- 172.21.15.87:0/3791392710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e12860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.329+0000 7f04273e2740 1 -- 172.21.15.87:0/3791392710 >> 172.21.15.87:0/3791392710 conn(0x561fc6df7ba0 msgr2=0x561fc6df6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.338+0000 7f04273e2740 1 -- 172.21.15.87:0/3791392710 shutdown_connections 2024-04-22T13:36:40.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.338+0000 7f04273e2740 1 -- 172.21.15.87:0/3791392710 wait complete. 2024-04-22T13:36:40.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.338+0000 7f04273e2740 1 -- start start 2024-04-22T13:36:40.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.338+0000 7f04273e2740 1 Processor -- start 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e0ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e00440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f040000ca60 con 0x561fc6dfd4d0 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f040000ded0 con 0x561fc6dfde80 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f040000d130 con 0x561fc6e06a00 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f04273e2740 10 client.0 _ll_register_callbacks cb 0x7f040c09bd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f0422cb2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e00440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f0421cb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50354/0 (socket says 172.21.15.87:50354) 2024-04-22T13:36:40.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.339+0000 7f0421cb0640 1 -- 172.21.15.87:0/4023032184 learned_addr learned my addr 172.21.15.87:0/4023032184 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 -- 172.21.15.87:0/4023032184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 msgr2=0x561fc6e00440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e00440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 -- 172.21.15.87:0/4023032184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 msgr2=0x561fc6e0ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e0ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 -- 172.21.15.87:0/4023032184 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc6e00a00 con 0x561fc6dfde80 2024-04-22T13:36:40.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.340+0000 7f0421cb0640 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c0cdde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42e0dbcbe579f685 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.350+0000 7f0405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:40.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.350+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c020070 con 0x561fc6dfde80 2024-04-22T13:36:40.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.350+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0418004490 con 0x561fc6dfde80 2024-04-22T13:36:40.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.350+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f040c098830 con 0x561fc6dfde80 2024-04-22T13:36:40.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.351+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f040c09bf90 con 0x561fc6dfde80 2024-04-22T13:36:40.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.353+0000 7f04273e2740 10 client.26138 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.353+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561fc6e00a00 con 0x561fc6dfde80 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0418004490 con 0x561fc6dfde80 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f040c0a1650 con 0x561fc6dfde80 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 10 client.26138 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 10 client.26138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:40.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 20 client.26138 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e8e240 0x561fc6e90660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 4 client.26138 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:36:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.354+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fc6d6fc10 con 0x561fc6e8e240 2024-04-22T13:36:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.355+0000 7f04224b1640 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e8e240 0x561fc6e90660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.356+0000 7f0405ffb640 10 client.26138 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:40.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.356+0000 7f04224b1640 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e8e240 0x561fc6e90660 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84204be3382d6bb0 server_cookie=5f87c387164e4442 in_seq=0 out_seq=0 2024-04-22T13:36:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.358+0000 7f0405ffb640 1 -- 172.21.15.87:0/4023032184 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fc6d6fc10 con 0x561fc6e8e240 2024-04-22T13:36:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.358+0000 7f0405ffb640 10 client.26138 handle_command_reply: tid=0 2024-04-22T13:36:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.358+0000 7f04273e2740 1 client.26138 shutdown 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 msgr2=0x561fc6dfff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c0cdde0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 shutdown_connections 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561fc6e8e240 0x561fc6e90660 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561fc6e06a00 0x561fc6e00440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561fc6dfde80 0x561fc6dfff00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 --2- 172.21.15.87:0/4023032184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561fc6dfd4d0 0x561fc6e0ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.361+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 >> 172.21.15.87:0/4023032184 conn(0x561fc6df7ba0 msgr2=0x561fc6df6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.362+0000 7f0405ffb640 0 client.26138 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.362+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 shutdown_connections 2024-04-22T13:36:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.362+0000 7f04273e2740 1 -- 172.21.15.87:0/4023032184 wait complete. 2024-04-22T13:36:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.362+0000 7f04273e2740 20 client.26138 trim_cache size 0 max 16384 2024-04-22T13:36:40.362 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:36:40.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 msgr2=0x7f04080a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 0x7f04080a6360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x561fc6df8c10 tx=0x7f041400a990 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 msgr2=0x7f041c1a79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f040c00d950 tx=0x7f040c00de10 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 shutdown_connections 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04080a3ea0 0x7f04080a6360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f041c11b730 0x7f041c1abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f041c1107a0 0x7f041c1a79e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 --2- 172.21.15.87:0/347133450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10e7d0 0x7f041c1a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.372+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 >> 172.21.15.87:0/347133450 conn(0x7f041c06d060 msgr2=0x7f041c072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.373+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 shutdown_connections 2024-04-22T13:36:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.373+0000 7f0424715640 1 -- 172.21.15.87:0/347133450 wait complete. 2024-04-22T13:36:40.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.938+0000 7f4fedab3640 1 -- 172.21.15.87:0/4025658232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810f4e0 msgr2=0x7f4fe810d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.938+0000 7f4fedab3640 1 --2- 172.21.15.87:0/4025658232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810f4e0 0x7f4fe810d820 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009ca0 tx=0x7f4fe002c120 comp rx=0 tx=0).stop 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.939+0000 7f4fedab3640 1 -- 172.21.15.87:0/4025658232 shutdown_connections 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.939+0000 7f4fedab3640 1 --2- 172.21.15.87:0/4025658232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe810df00 0x7f4fe8115ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.939+0000 7f4fedab3640 1 --2- 172.21.15.87:0/4025658232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810f4e0 0x7f4fe810d820 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.939+0000 7f4fedab3640 1 --2- 172.21.15.87:0/4025658232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4fe810eb30 0x7f4fe810ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.939+0000 7f4fedab3640 1 -- 172.21.15.87:0/4025658232 >> 172.21.15.87:0/4025658232 conn(0x7f4fe806ce60 msgr2=0x7f4fe806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.944+0000 7f4fedab3640 1 -- 172.21.15.87:0/4025658232 shutdown_connections 2024-04-22T13:36:40.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.949+0000 7f4fedab3640 1 -- 172.21.15.87:0/4025658232 wait complete. 2024-04-22T13:36:40.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.950+0000 7f4fedab3640 1 Processor -- start 2024-04-22T13:36:40.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.950+0000 7f4fedab3640 1 -- start start 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe810eb30 0x7f4fe81a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4fe81a7870 0x7f4fe81a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe811c9f0 con 0x7f4fe81a7870 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe811c870 con 0x7f4fe810eb30 2024-04-22T13:36:40.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.951+0000 7f4fedab3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe811cb70 con 0x7f4fe810df00 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.953+0000 7f4fecab1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.953+0000 7f4fecab1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51562/0 (socket says 172.21.15.87:51562) 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.953+0000 7f4fecab1640 1 -- 172.21.15.87:0/221981065 learned_addr learned my addr 172.21.15.87:0/221981065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.955+0000 7f4fecab1640 1 -- 172.21.15.87:0/221981065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe810eb30 msgr2=0x7f4fe81a34f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.955+0000 7f4fecab1640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe810eb30 0x7f4fe81a34f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.955+0000 7f4fecab1640 1 -- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4fe81a7870 msgr2=0x7f4fe81a7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.955+0000 7f4fecab1640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4fe81a7870 0x7f4fe81a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.955+0000 7f4fecab1640 1 -- 172.21.15.87:0/221981065 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe81a8360 con 0x7f4fe810df00 2024-04-22T13:36:40.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.957+0000 7f4fecab1640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800ddc0 tx=0x7f4fd800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.963+0000 7f4fe5ffb640 1 -- 172.21.15.87:0/221981065 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd801b070 con 0x7f4fe810df00 2024-04-22T13:36:40.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.963+0000 7f4fe5ffb640 1 -- 172.21.15.87:0/221981065 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd800be40 con 0x7f4fe810df00 2024-04-22T13:36:40.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.963+0000 7f4fe5ffb640 1 -- 172.21.15.87:0/221981065 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8016a80 con 0x7f4fe810df00 2024-04-22T13:36:40.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.965+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe81a8650 con 0x7f4fe810df00 2024-04-22T13:36:40.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.965+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe81afdf0 con 0x7f4fe810df00 2024-04-22T13:36:40.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.969+0000 7f4ff19dd740 1 Processor -- start 2024-04-22T13:36:40.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.970+0000 7f4ff19dd740 1 -- start start 2024-04-22T13:36:40.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.970+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.971+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e79360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.971+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.971+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x565239dc80b0 con 0x565239e798a0 2024-04-22T13:36:40.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51572/0 (socket says 172.21.15.87:51572) 2024-04-22T13:36:40.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3783635321 learned_addr learned my addr 172.21.15.87:0/3783635321 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565239d79680 con 0x565239e70cc0 2024-04-22T13:36:40.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565239d85c60 con 0x565239e703a0 2024-04-22T13:36:40.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.973+0000 7f4fecab1640 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e79360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.974+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.975+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3783635321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 msgr2=0x565239e79360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.975+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e79360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.975+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3783635321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 msgr2=0x565239e7bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.975+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.975+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3783635321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565239baac90 con 0x565239e703a0 2024-04-22T13:36:40.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.976+0000 7f4fe5ffb640 1 -- 172.21.15.87:0/221981065 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4fd801e070 con 0x7f4fe810df00 2024-04-22T13:36:40.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.977+0000 7f4fe5ffb640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 0x7f4fcc0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.978+0000 7f4fe5ffb640 1 -- 172.21.15.87:0/221981065 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f4fd80d34d0 con 0x7f4fe810df00 2024-04-22T13:36:40.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.978+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400a960 tx=0x7f4fd400ae20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8bfc71d8a81c85 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.980+0000 7f4fecab1640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 0x7f4fcc0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.981+0000 7f4fdeffd640 1 -- 172.21.15.87:0/3783635321 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400cd80 con 0x565239e703a0 2024-04-22T13:36:40.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.981+0000 7f4fdeffd640 1 -- 172.21.15.87:0/3783635321 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd400f900 con 0x565239e703a0 2024-04-22T13:36:40.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.982+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 msgr2=0x565239e70780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:40.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.982+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400a960 tx=0x7f4fd400ae20 comp rx=0 tx=0).stop 2024-04-22T13:36:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.988+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 shutdown_connections 2024-04-22T13:36:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.988+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.988+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e79360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.988+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3783635321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e70780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.988+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 >> 172.21.15.87:0/3783635321 conn(0x565239e6aad0 msgr2=0x565239e6aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:40.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.989+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 shutdown_connections 2024-04-22T13:36:40.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.989+0000 7f4fecab1640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 0x7f4fcc0a63f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x565239e6fe70 tx=0x7f4fd8014000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:40.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.989+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3783635321 wait complete. 2024-04-22T13:36:40.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.990+0000 7f4ff19dd740 1 -- start start 2024-04-22T13:36:40.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.990+0000 7f4ff19dd740 1 Processor -- start 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e703a0 0x565239e76890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e76dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 0x565239e77310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x565239dcb400 con 0x565239e703a0 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565239d87840 con 0x565239e70cc0 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565239dd0720 con 0x565239e798a0 2024-04-22T13:36:40.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.991+0000 7f4ff19dd740 10 client.0 _ll_register_callbacks cb 0x565239dd8ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:40.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.996+0000 7f4fe7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e76dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.996+0000 7f4fe7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e76dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50372/0 (socket says 172.21.15.87:50372) 2024-04-22T13:36:40.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.996+0000 7f4fe7fff640 1 -- 172.21.15.87:0/3180540173 learned_addr learned my addr 172.21.15.87:0/3180540173 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:40.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.995+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 0x565239e77310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.996+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e703a0 0x565239e76890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:40.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.999+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 msgr2=0x565239e76dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.999+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e76dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.999+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3180540173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e703a0 msgr2=0x565239e76890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:40.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.999+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e703a0 0x565239e76890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:40.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:40.999+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3180540173 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565239c0b420 con 0x565239e798a0 2024-04-22T13:36:40.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.002+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 0x565239e77310 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4004700 tx=0x7f4fd401c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b62e6283e5d4ce38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.006+0000 7f4fcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:36:41.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.006+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400f450 con 0x565239e798a0 2024-04-22T13:36:41.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.006+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f4fd4023040 con 0x565239e798a0 2024-04-22T13:36:41.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.006+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd400f900 con 0x565239e798a0 2024-04-22T13:36:41.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.007+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4fd400f5f0 con 0x565239e798a0 2024-04-22T13:36:41.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.011+0000 7f4ff19dd740 10 client.26197 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:41.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.011+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x565239a9e580 con 0x565239e798a0 2024-04-22T13:36:41.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.011+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd400f900 con 0x565239e798a0 2024-04-22T13:36:41.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4fd402f020 con 0x565239e798a0 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 10 client.26197 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 10 client.26197 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 20 client.26197 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239e89960 0x565239e8bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 4 client.26197 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:41.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.012+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565239dd9b10 con 0x565239e89960 2024-04-22T13:36:41.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.018+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239e89960 0x565239e8bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.021+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239e89960 0x565239e8bd80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e7259ad50f31c21 server_cookie=10ee894e75372f36 in_seq=0 out_seq=0 2024-04-22T13:36:41.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.021+0000 7f4fcbfff640 10 client.26197 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.025+0000 7f4fcbfff640 1 -- 172.21.15.87:0/3180540173 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565239dd9b10 con 0x565239e89960 2024-04-22T13:36:41.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.025+0000 7f4fcbfff640 10 client.26197 handle_command_reply: tid=0 2024-04-22T13:36:41.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.025+0000 7f4ff19dd740 1 client.26197 shutdown 2024-04-22T13:36:41.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.028+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 msgr2=0x565239e77310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.028+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 0x565239e77310 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4004700 tx=0x7f4fd401c040 comp rx=0 tx=0).stop 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 shutdown_connections 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239e89960 0x565239e8bd80 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e798a0 0x565239e77310 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e76dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3180540173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e703a0 0x565239e76890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 >> 172.21.15.87:0/3180540173 conn(0x565239e6aad0 msgr2=0x565239e6c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.029+0000 7f4fcbfff640 0 client.26197 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.037+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 shutdown_connections 2024-04-22T13:36:41.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.040+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3180540173 wait complete. 2024-04-22T13:36:41.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.041+0000 7f4ff19dd740 20 client.26197 trim_cache size 0 max 16384 2024-04-22T13:36:41.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.121+0000 7f4ff19dd740 1 Processor -- start 2024-04-22T13:36:41.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.122+0000 7f4ff19dd740 1 -- start start 2024-04-22T13:36:41.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.122+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.122+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e70cc0 0x565239e6c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e798a0 0x565239e91670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000e880 con 0x565239e70cc0 2024-04-22T13:36:41.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000c6d0 con 0x565239e798a0 2024-04-22T13:36:41.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000e1b0 con 0x565239e703a0 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51578/0 (socket says 172.21.15.87:51578) 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.123+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3951008907 learned_addr learned my addr 172.21.15.87:0/3951008907 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e798a0 0x565239e91670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3951008907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e798a0 msgr2=0x565239e91670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e798a0 0x565239e91670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3951008907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e70cc0 msgr2=0x565239e6c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e70cc0 0x565239e6c5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.125+0000 7f4fed2b2640 1 -- 172.21.15.87:0/3951008907 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565239e6a900 con 0x565239e703a0 2024-04-22T13:36:41.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.130+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4031fd0 tx=0x7f4fd4018400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72d1b255fee14ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.131+0000 7f4fde7fc640 1 -- 172.21.15.87:0/3951008907 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400ce90 con 0x565239e703a0 2024-04-22T13:36:41.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.131+0000 7f4fde7fc640 1 -- 172.21.15.87:0/3951008907 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd4036070 con 0x565239e703a0 2024-04-22T13:36:41.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.131+0000 7f4fde7fc640 1 -- 172.21.15.87:0/3951008907 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400f790 con 0x565239e703a0 2024-04-22T13:36:41.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.133+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3951008907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 msgr2=0x565239e78fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.133+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4031fd0 tx=0x7f4fd4018400 comp rx=0 tx=0).stop 2024-04-22T13:36:41.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.137+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3951008907 shutdown_connections 2024-04-22T13:36:41.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.137+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e798a0 0x565239e91670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.137+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e70cc0 0x565239e6c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.137+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/3951008907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e78fc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.137+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3951008907 >> 172.21.15.87:0/3951008907 conn(0x565239e6aad0 msgr2=0x565239e6bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.139+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3951008907 shutdown_connections 2024-04-22T13:36:41.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.140+0000 7f4ff19dd740 1 -- 172.21.15.87:0/3951008907 wait complete. 2024-04-22T13:36:41.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.141+0000 7f4ff19dd740 1 -- start start 2024-04-22T13:36:41.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.141+0000 7f4ff19dd740 1 Processor -- start 2024-04-22T13:36:41.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e819b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e7cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000d410 con 0x565239e798a0 2024-04-22T13:36:41.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000cd40 con 0x565239e70cc0 2024-04-22T13:36:41.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.143+0000 7f4ff19dd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd000dae0 con 0x565239e703a0 2024-04-22T13:36:41.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.144+0000 7f4ff19dd740 10 client.0 _ll_register_callbacks cb 0x7f4fd80d4d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.149+0000 7f4fe7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e7cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.149+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.149+0000 7f4fed2b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35450/0 (socket says 172.21.15.87:35450) 2024-04-22T13:36:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.149+0000 7f4fed2b2640 1 -- 172.21.15.87:0/2318199831 learned_addr learned my addr 172.21.15.87:0/2318199831 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.149+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e819b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.152+0000 7f4fe7fff640 1 -- 172.21.15.87:0/2318199831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 msgr2=0x565239e819b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.152+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e819b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.152+0000 7f4fe7fff640 1 -- 172.21.15.87:0/2318199831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 msgr2=0x565239e7d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.152+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.152+0000 7f4fe7fff640 1 -- 172.21.15.87:0/2318199831 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565239e7d890 con 0x565239e70cc0 2024-04-22T13:36:41.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.154+0000 7f4fe7fff640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e7cde0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0000f80 tx=0x7f4fe002c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca80967e56319c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.155+0000 7f4fcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:41.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.158+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0039040 con 0x565239e70cc0 2024-04-22T13:36:41.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.158+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f4fe0032e00 con 0x565239e70cc0 2024-04-22T13:36:41.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.158+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd4036070 con 0x565239e70cc0 2024-04-22T13:36:41.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.160+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4fe00324c0 con 0x565239e70cc0 2024-04-22T13:36:41.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.160+0000 7f4ff19dd740 10 client.26150 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:41.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.160+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x565239eefc40 con 0x565239e70cc0 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.163+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd4036070 con 0x565239e70cc0 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.163+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4fe0045350 con 0x565239e70cc0 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.163+0000 7f4ff19dd740 10 client.26150 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4ff19dd740 10 client.26150 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4ff19dd740 20 client.26150 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:41.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239ef03c0 0x565239ef27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4ff19dd740 4 client.26150 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}] 2024-04-22T13:36:41.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}) -- 0x565239dd3980 con 0x565239ef03c0 2024-04-22T13:36:41.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.164+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239ef03c0 0x565239ef27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.172+0000 7f4fed2b2640 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239ef03c0 0x565239ef27e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fa074f6403e6209 server_cookie=6520e0040e247d22 in_seq=0 out_seq=0 2024-04-22T13:36:41.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.172+0000 7f4fcbfff640 10 client.26150 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:40 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1538347398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:41.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:40 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1538347398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:41.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:40 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1538347398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:36:41.406 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:36:41.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:36:41.406 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.617+0000 7f0d08ae4640 1 -- 172.21.15.87:0/2168396353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d04101a30 msgr2=0x7f0d04105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.617+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/2168396353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d04101a30 0x7f0d04105a70 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009860 tx=0x7f0cf802bb70 comp rx=0 tx=0).stop 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.618+0000 7f0d08ae4640 1 -- 172.21.15.87:0/2168396353 shutdown_connections 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.618+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/2168396353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d04106150 0x7f0d0410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.618+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/2168396353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d04101a30 0x7f0d04105a70 secure :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009860 tx=0x7f0cf802bb70 comp rx=0 tx=0).stop 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.618+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/2168396353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04101080 0x7f0d04101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.618+0000 7f0d08ae4640 1 -- 172.21.15.87:0/2168396353 >> 172.21.15.87:0/2168396353 conn(0x7f0d04077ef0 msgr2=0x7f0d04078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.622+0000 7f0d08ae4640 1 -- 172.21.15.87:0/2168396353 shutdown_connections 2024-04-22T13:36:41.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.625+0000 7f0d08ae4640 1 -- 172.21.15.87:0/2168396353 wait complete. 2024-04-22T13:36:41.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.626+0000 7f0d08ae4640 1 Processor -- start 2024-04-22T13:36:41.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.626+0000 7f0d08ae4640 1 -- start start 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.627+0000 7f0d08ae4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d04101080 0x7f0d04073bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d08ae4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d08ae4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d0406e290 0x7f0d0406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d08ae4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0406c270 con 0x7f0d0406e290 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d08ae4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0406c0f0 con 0x7f0d04106150 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d08ae4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0406c3f0 con 0x7f0d04101080 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.631+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.632+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50396/0 (socket says 172.21.15.87:50396) 2024-04-22T13:36:41.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.632+0000 7f0d01d9b640 1 -- 172.21.15.87:0/1424908978 learned_addr learned my addr 172.21.15.87:0/1424908978 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.633+0000 7f0d01d9b640 1 -- 172.21.15.87:0/1424908978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d04101080 msgr2=0x7f0d04073bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.633+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d04101080 0x7f0d04073bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.633+0000 7f0d01d9b640 1 -- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d0406e290 msgr2=0x7f0d0406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.633+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d0406e290 0x7f0d0406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.633+0000 7f0d01d9b640 1 -- 172.21.15.87:0/1424908978 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0406ea60 con 0x7f0d04106150 2024-04-22T13:36:41.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.634+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8037040 tx=0x7f0cf80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.637+0000 7f0cf37fe640 1 -- 172.21.15.87:0/1424908978 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8039040 con 0x7f0d04106150 2024-04-22T13:36:41.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.637+0000 7f0cf37fe640 1 -- 172.21.15.87:0/1424908978 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf80347b0 con 0x7f0d04106150 2024-04-22T13:36:41.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.637+0000 7f0cf37fe640 1 -- 172.21.15.87:0/1424908978 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf803d3f0 con 0x7f0d04106150 2024-04-22T13:36:41.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.639+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0406a8c0 con 0x7f0d04106150 2024-04-22T13:36:41.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.639+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0406ad60 con 0x7f0d04106150 2024-04-22T13:36:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.643+0000 7f0d0b7b5740 1 Processor -- start 2024-04-22T13:36:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.644+0000 7f0d0b7b5740 1 -- start start 2024-04-22T13:36:41.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.644+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.645+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da1c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.645+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da1ce10 0x55e90da1f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.645+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e90d977f00 con 0x55e90da14290 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.645+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e90d915590 con 0x55e90da138e0 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.645+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e90d973ec0 con 0x55e90da1ce10 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.646+0000 7f0d02d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.646+0000 7f0d02d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50412/0 (socket says 172.21.15.87:50412) 2024-04-22T13:36:41.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.646+0000 7f0d02d9d640 1 -- 172.21.15.87:0/1876099347 learned_addr learned my addr 172.21.15.87:0/1876099347 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.646+0000 7f0d0259c640 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da1c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.646+0000 7f0cf37fe640 1 -- 172.21.15.87:0/1424908978 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0cf8034b90 con 0x7f0d04106150 2024-04-22T13:36:41.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.647+0000 7f0d02d9d640 1 -- 172.21.15.87:0/1876099347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da1ce10 msgr2=0x55e90da1f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.647+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da1ce10 0x55e90da1f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.647+0000 7f0d02d9d640 1 -- 172.21.15.87:0/1876099347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 msgr2=0x55e90da1c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.647+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da1c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.648+0000 7f0d02d9d640 1 -- 172.21.15.87:0/1876099347 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90d7527e0 con 0x55e90da138e0 2024-04-22T13:36:41.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.648+0000 7f0cf37fe640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 0x7f0cec0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.648+0000 7f0d0259c640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 0x7f0cec0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.649+0000 7f0cf37fe640 1 -- 172.21.15.87:0/1424908978 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f0cf80f1610 con 0x7f0d04106150 2024-04-22T13:36:41.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.651+0000 7f0d0259c640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 0x7f0cec0a63b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55e90da0f020 tx=0x7f0cfc0095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.654+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400ba00 tx=0x7f0cf400bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c62a9da61700dfbe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.656+0000 7f0cf0ff9640 1 -- 172.21.15.87:0/1876099347 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4016070 con 0x55e90da138e0 2024-04-22T13:36:41.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.656+0000 7f0cf0ff9640 1 -- 172.21.15.87:0/1876099347 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf40054d0 con 0x55e90da138e0 2024-04-22T13:36:41.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.656+0000 7f0cf0ff9640 1 -- 172.21.15.87:0/1876099347 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4004e10 con 0x55e90da138e0 2024-04-22T13:36:41.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/1876099347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 msgr2=0x55e90da13cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400ba00 tx=0x7f0cf400bec0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/1876099347 shutdown_connections 2024-04-22T13:36:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da1ce10 0x55e90da1f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da1c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/1876099347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da138e0 0x55e90da13cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.658+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/1876099347 >> 172.21.15.87:0/1876099347 conn(0x55e90da0dfb0 msgr2=0x55e90da0e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.662+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/1876099347 shutdown_connections 2024-04-22T13:36:41.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.666+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/1876099347 wait complete. 2024-04-22T13:36:41.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.667+0000 7f0d0b7b5740 1 -- start start 2024-04-22T13:36:41.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.667+0000 7f0d0b7b5740 1 Processor -- start 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da138e0 0x55e90da19ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da14290 0x55e90da1a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e90da1fd80 con 0x55e90da138e0 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e90d931580 con 0x55e90da1ce10 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e90da1ff00 con 0x55e90da14290 2024-04-22T13:36:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.668+0000 7f0d0b7b5740 10 client.0 _ll_register_callbacks cb 0x55e90d937650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.670+0000 7f0d02d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.670+0000 7f0d02d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:50416/0 (socket says 172.21.15.87:50416) 2024-04-22T13:36:41.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.670+0000 7f0d02d9d640 1 -- 172.21.15.87:0/2155586595 learned_addr learned my addr 172.21.15.87:0/2155586595 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.672+0000 7f0d02d9d640 1 -- 172.21.15.87:0/2155586595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da14290 msgr2=0x55e90da1a3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.672+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da14290 0x55e90da1a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.672+0000 7f0d02d9d640 1 -- 172.21.15.87:0/2155586595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da138e0 msgr2=0x55e90da19ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.672+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da138e0 0x55e90da19ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.672+0000 7f0d02d9d640 1 -- 172.21.15.87:0/2155586595 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90d7b2f70 con 0x55e90da1ce10 2024-04-22T13:36:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.674+0000 7f0d02d9d640 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40049b0 tx=0x7f0cf4005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf4bac63a540dd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.674+0000 7f0ce9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:36:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.675+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4016070 con 0x55e90da1ce10 2024-04-22T13:36:41.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.675+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf40054d0 con 0x55e90da1ce10 2024-04-22T13:36:41.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.675+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f0cf400d9e0 con 0x55e90da1ce10 2024-04-22T13:36:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.677+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0cf400dc00 con 0x55e90da1ce10 2024-04-22T13:36:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.678+0000 7f0d0b7b5740 10 client.26162 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.678+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e90d64a170 con 0x55e90da1ce10 2024-04-22T13:36:41.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.680+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf402c070 con 0x55e90da1ce10 2024-04-22T13:36:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.681+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0cf4003ad0 con 0x55e90da1ce10 2024-04-22T13:36:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.681+0000 7f0d0b7b5740 10 client.26162 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.681+0000 7f0d0b7b5740 10 client.26162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.682+0000 7f0d0b7b5740 20 client.26162 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.682+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90da3e7d0 0x55e90da40c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.682+0000 7f0d0b7b5740 4 client.26162 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.682+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e90d985920 con 0x55e90da3e7d0 2024-04-22T13:36:41.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.684+0000 7f0d0259c640 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90da3e7d0 0x55e90da40c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.689+0000 7f0d0259c640 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90da3e7d0 0x55e90da40c60 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=572c682f6c27cc3f server_cookie=5642325acee87e62 in_seq=0 out_seq=0 2024-04-22T13:36:41.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.690+0000 7f0ce9ffb640 10 client.26162 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.693+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/2155586595 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e90d985920 con 0x55e90da3e7d0 2024-04-22T13:36:41.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.693+0000 7f0ce9ffb640 10 client.26162 handle_command_reply: tid=0 2024-04-22T13:36:41.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.693+0000 7f0d0b7b5740 1 client.26162 shutdown 2024-04-22T13:36:41.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.694+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 msgr2=0x55e90da1a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.694+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40049b0 tx=0x7f0cf4005250 comp rx=0 tx=0).stop 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 shutdown_connections 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90da3e7d0 0x55e90da40c60 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da1a920 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da14290 0x55e90da1a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/2155586595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da138e0 0x55e90da19ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.696+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 >> 172.21.15.87:0/2155586595 conn(0x55e90da0dfb0 msgr2=0x55e90da0fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.697+0000 7f0ce9ffb640 0 client.26162 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.702+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 shutdown_connections 2024-04-22T13:36:41.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.705+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/2155586595 wait complete. 2024-04-22T13:36:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.708+0000 7f0d0b7b5740 20 client.26162 trim_cache size 0 max 16384 2024-04-22T13:36:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.765+0000 7f0d0b7b5740 1 Processor -- start 2024-04-22T13:36:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.765+0000 7f0d0b7b5740 1 -- start start 2024-04-22T13:36:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.765+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da1c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.765+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da0fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35488/0 (socket says 172.21.15.87:35488) 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d01d9b640 1 -- 172.21.15.87:0/644652621 learned_addr learned my addr 172.21.15.87:0/644652621 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b6f30 con 0x55e90da14290 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d0259c640 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da1c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b4800 con 0x55e90da1ce10 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.766+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b2700 con 0x55e90da138e0 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 -- 172.21.15.87:0/644652621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 msgr2=0x55e90da1c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da1c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 -- 172.21.15.87:0/644652621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 msgr2=0x55e90da0fb10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da0fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 -- 172.21.15.87:0/644652621 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e90da14000 con 0x55e90da14290 2024-04-22T13:36:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.767+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80bd660 tx=0x7f0cf802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc97949d14e16343 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.768+0000 7f0cebfff640 1 -- 172.21.15.87:0/644652621 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf80d3030 con 0x55e90da14290 2024-04-22T13:36:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.768+0000 7f0cebfff640 1 -- 172.21.15.87:0/644652621 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf80bddd0 con 0x55e90da14290 2024-04-22T13:36:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.768+0000 7f0cebfff640 1 -- 172.21.15.87:0/644652621 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf80bd6d0 con 0x55e90da14290 2024-04-22T13:36:41.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 msgr2=0x55e90da28c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80bd660 tx=0x7f0cf802bb70 comp rx=0 tx=0).stop 2024-04-22T13:36:41.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 shutdown_connections 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da0fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da28c40 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/644652621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da1c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.769+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 >> 172.21.15.87:0/644652621 conn(0x55e90da0dfb0 msgr2=0x55e90da0f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.770+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 shutdown_connections 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.770+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/644652621 wait complete. 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.770+0000 7f0d0b7b5740 1 -- start start 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.770+0000 7f0d0b7b5740 1 Processor -- start 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da25340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da16c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d01d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35498/0 (socket says 172.21.15.87:35498) 2024-04-22T13:36:41.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d01d9b640 1 -- 172.21.15.87:0/3471516909 learned_addr learned my addr 172.21.15.87:0/3471516909 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b2cc0 con 0x55e90da14290 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b4130 con 0x55e90da1ce10 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d041b3390 con 0x55e90da138e0 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.771+0000 7f0d0b7b5740 10 client.0 _ll_register_callbacks cb 0x7f0cf400b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 -- 172.21.15.87:0/3471516909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 msgr2=0x55e90da25340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da25340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 -- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 msgr2=0x55e90da16c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da16c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90da17210 con 0x55e90da14290 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d01d9b640 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80bd3d0 tx=0x7f0cf8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d51e0a77fa25f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0ce9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf80d3050 con 0x55e90da14290 2024-04-22T13:36:41.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f0cf80c5eb0 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf80bddd0 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0cf80ce050 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d0b7b5740 10 client.17025 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.772+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e90daa4be0 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cf80bddd0 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0cf803c070 con 0x55e90da14290 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 10 client.17025 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 20 client.17025 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90daa5250 0x55e90daa7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 4 client.17025 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:36:41.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.773+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e90d97f790 con 0x55e90daa5250 2024-04-22T13:36:41.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.775+0000 7f0d0259c640 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90daa5250 0x55e90daa7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:41.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.777+0000 7f0d0259c640 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90daa5250 0x55e90daa7670 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19aaef8ae3fdbb29 server_cookie=f99b99b2c6a4d8ea in_seq=0 out_seq=0 2024-04-22T13:36:41.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.777+0000 7f0ce9ffb640 10 client.17025 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.847+0000 7f0ce9ffb640 1 -- 172.21.15.87:0/3471516909 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e90d97f790 con 0x55e90daa5250 2024-04-22T13:36:41.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.847+0000 7f0ce9ffb640 10 client.17025 handle_command_reply: tid=0 2024-04-22T13:36:41.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.847+0000 7f0d0b7b5740 1 client.17025 shutdown 2024-04-22T13:36:41.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.848+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 msgr2=0x55e90da16710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.848+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80bd3d0 tx=0x7f0cf8005f90 comp rx=0 tx=0).stop 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.849+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 shutdown_connections 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e90daa5250 0x55e90daa7670 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e90da1ce10 0x55e90da16c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0ce9ffb640 0 client.17025 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e90da14290 0x55e90da16710 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 --2- 172.21.15.87:0/3471516909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e90da138e0 0x55e90da25340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 >> 172.21.15.87:0/3471516909 conn(0x55e90da0dfb0 msgr2=0x55e90da0fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 shutdown_connections 2024-04-22T13:36:41.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 1 -- 172.21.15.87:0/3471516909 wait complete. 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.850+0000 7f0d0b7b5740 20 client.17025 trim_cache size 0 max 16384 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "eaf1b230-a529-4804-a693-f7d5f0717cc5", 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:36:41.848 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.851+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 msgr2=0x7f0cec0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.851+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 0x7f0cec0a63b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55e90da0f020 tx=0x7f0cfc0095c0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.851+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 msgr2=0x7f0d0406dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.851+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8037040 tx=0x7f0cf80042f0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 shutdown_connections 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cec0a3ef0 0x7f0cec0a63b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0d0406e290 0x7f0d0406a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d04106150 0x7f0d0406dc40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 --2- 172.21.15.87:0/1424908978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d04101080 0x7f0d04073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:41.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 >> 172.21.15.87:0/1424908978 conn(0x7f0d04077ef0 msgr2=0x7f0d040ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:41.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 shutdown_connections 2024-04-22T13:36:41.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:41.852+0000 7f0d08ae4640 1 -- 172.21.15.87:0/1424908978 wait complete. 2024-04-22T13:36:41.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:36:41.875 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 -- 172.21.15.87:0/3540938637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 msgr2=0x7fab54110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 --2- 172.21.15.87:0/3540938637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab54110ce0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009ca0 tx=0x7fab4c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 -- 172.21.15.87:0/3540938637 shutdown_connections 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 --2- 172.21.15.87:0/3540938637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab54110ce0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 --2- 172.21.15.87:0/3540938637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54101660 0x7fab5410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 --2- 172.21.15.87:0/3540938637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54100d40 0x7fab54101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 -- 172.21.15.87:0/3540938637 >> 172.21.15.87:0/3540938637 conn(0x7fab54078000 msgr2=0x7fab54078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 -- 172.21.15.87:0/3540938637 shutdown_connections 2024-04-22T13:36:42.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.032+0000 7fab58c9a640 1 -- 172.21.15.87:0/3540938637 wait complete. 2024-04-22T13:36:42.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 Processor -- start 2024-04-22T13:36:42.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 -- start start 2024-04-22T13:36:42.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54100d40 0x7fab541a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54101660 0x7fab541a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab54114430 con 0x7fab5410e8f0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab541142b0 con 0x7fab54101660 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab58c9a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab541145b0 con 0x7fab54100d40 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35524/0 (socket says 172.21.15.87:35524) 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.033+0000 7fab52d9d640 1 -- 172.21.15.87:0/1370919695 learned_addr learned my addr 172.21.15.87:0/1370919695 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab5259c640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54100d40 0x7fab541a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab51d9b640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54101660 0x7fab541a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 -- 172.21.15.87:0/1370919695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54100d40 msgr2=0x7fab541a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54100d40 0x7fab541a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 -- 172.21.15.87:0/1370919695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54101660 msgr2=0x7fab541a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54101660 0x7fab541a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 -- 172.21.15.87:0/1370919695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5419b010 con 0x7fab5410e8f0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab52d9d640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fab4c02c700 tx=0x7fab4c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab437fe640 1 -- 172.21.15.87:0/1370919695 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c041020 con 0x7fab5410e8f0 2024-04-22T13:36:42.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab437fe640 1 -- 172.21.15.87:0/1370919695 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab4c034ca0 con 0x7fab5410e8f0 2024-04-22T13:36:42.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab437fe640 1 -- 172.21.15.87:0/1370919695 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c039040 con 0x7fab5410e8f0 2024-04-22T13:36:42.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5419b2a0 con 0x7fab5410e8f0 2024-04-22T13:36:42.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.034+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab541a7430 con 0x7fab5410e8f0 2024-04-22T13:36:42.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.037+0000 7fab437fe640 1 -- 172.21.15.87:0/1370919695 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fab4c044020 con 0x7fab5410e8f0 2024-04-22T13:36:42.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.038+0000 7fab437fe640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 0x7fab3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.038+0000 7fab437fe640 1 -- 172.21.15.87:0/1370919695 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fab4c0f24e0 con 0x7fab5410e8f0 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.038+0000 7fab5259c640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 0x7fab3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 Processor -- start 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5259c640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 0x7fab3c0a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab44009990 tx=0x7fab44009110 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 -- start start 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267d2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267d2a80 0x55d5267d4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d52672e1e0 con 0x55d5267c9ea0 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d5266cb390 con 0x55d5267d2a80 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.039+0000 7fab5b969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d52672a1a0 con 0x55d5267c9580 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab51d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267d2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:51666/0 (socket says 172.21.15.87:51666) 2024-04-22T13:36:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 -- 172.21.15.87:0/3609403933 learned_addr learned my addr 172.21.15.87:0/3609403933 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 -- 172.21.15.87:0/3609403933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267d2a80 msgr2=0x55d5267d4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267d2a80 0x55d5267d4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 -- 172.21.15.87:0/3609403933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 msgr2=0x55d5267d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267d2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 -- 172.21.15.87:0/3609403933 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d52650a7d0 con 0x55d5267c9580 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab51d9b640 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267d2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.040+0000 7fab5259c640 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab44009fd0 tx=0x7fab44011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce6fec24f24f416 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.041+0000 7fab40ff9640 1 -- 172.21.15.87:0/3609403933 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4401b040 con 0x55d5267c9580 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.041+0000 7fab40ff9640 1 -- 172.21.15.87:0/3609403933 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab44024070 con 0x55d5267c9580 2024-04-22T13:36:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.041+0000 7fab40ff9640 1 -- 172.21.15.87:0/3609403933 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4401f3f0 con 0x55d5267c9580 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.041+0000 7fab5b969740 1 -- 172.21.15.87:0/3609403933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 msgr2=0x55d5267c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.041+0000 7fab5b969740 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab44009fd0 tx=0x7fab44011000 comp rx=0 tx=0).stop 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- 172.21.15.87:0/3609403933 shutdown_connections 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267d2a80 0x55d5267d4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267d2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- 172.21.15.87:0/3609403933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267c9960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- 172.21.15.87:0/3609403933 >> 172.21.15.87:0/3609403933 conn(0x55d5267c3cb0 msgr2=0x55d5267c4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- 172.21.15.87:0/3609403933 shutdown_connections 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- 172.21.15.87:0/3609403933 wait complete. 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- start start 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 Processor -- start 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9580 0x55d5267cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267d2a80 0x55d5267d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d5267d5870 con 0x55d5267c9ea0 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d5266e72f0 con 0x55d5267c9580 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d5267d59f0 con 0x55d5267d2a80 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.042+0000 7fab5b969740 10 client.0 _ll_register_callbacks cb 0x55d5266f4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35534/0 (socket says 172.21.15.87:35534) 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 -- 172.21.15.87:0/3038316953 learned_addr learned my addr 172.21.15.87:0/3038316953 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab52d9d640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267d2a80 0x55d5267d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 -- 172.21.15.87:0/3038316953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267d2a80 msgr2=0x55d5267d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267d2a80 0x55d5267d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 -- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9580 msgr2=0x55d5267cfa90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9580 0x55d5267cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 -- 172.21.15.87:0/3038316953 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5264403f0 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab51d9b640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fab480136e0 tx=0x7fab48013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79f99bc1fd05d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab48004460 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab44024070 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fab480119c0 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fab48011bc0 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab5b969740 10 client.17037 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.043+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5264000f0 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab44024070 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fab48024020 con 0x55d5267c9ea0 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 10 client.17037 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 20 client.17037 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d5267e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 4 client.17037 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d52673bc00 con 0x55d5267e2a80 2024-04-22T13:36:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.044+0000 7fab5259c640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d5267e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.045+0000 7fab5259c640 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d5267e4eb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302dcafa5c1b90a1 server_cookie=afaba4311ca54a8c in_seq=0 out_seq=0 2024-04-22T13:36:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.045+0000 7fab39ffb640 10 client.17037 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.046+0000 7fab39ffb640 1 -- 172.21.15.87:0/3038316953 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d52673bc00 con 0x55d5267e2a80 2024-04-22T13:36:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.046+0000 7fab39ffb640 10 client.17037 handle_command_reply: tid=0 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 client.17037 shutdown 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 msgr2=0x55d5267cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fab480136e0 tx=0x7fab48013ba0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 shutdown_connections 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d5267e4eb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab39ffb640 0 client.17037 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267d2a80 0x55d5267d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267c9ea0 0x55d5267cffd0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 --2- 172.21.15.87:0/3038316953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9580 0x55d5267cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 >> 172.21.15.87:0/3038316953 conn(0x55d5267c3cb0 msgr2=0x55d5267c5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 shutdown_connections 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 1 -- 172.21.15.87:0/3038316953 wait complete. 2024-04-22T13:36:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.047+0000 7fab5b969740 20 client.17037 trim_cache size 0 max 16384 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 Processor -- start 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 -- start start 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267de760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab541b2550 con 0x55d5267d2a80 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab541afe20 con 0x55d5267c9ea0 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5b969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab541adc70 con 0x55d5267c9580 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35540/0 (socket says 172.21.15.87:35540) 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 -- 172.21.15.87:0/4059899627 learned_addr learned my addr 172.21.15.87:0/4059899627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab51d9b640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab5259c640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 -- 172.21.15.87:0/4059899627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 msgr2=0x55d5267d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 -- 172.21.15.87:0/4059899627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 msgr2=0x55d5267de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267de760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 -- 172.21.15.87:0/4059899627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5267c37e0 con 0x55d5267d2a80 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab51d9b640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab52d9d640 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009430 tx=0x7fab4c0bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e921d5e8b4418ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab3bfff640 1 -- 172.21.15.87:0/4059899627 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0bf430 con 0x55d5267d2a80 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.065+0000 7fab3bfff640 1 -- 172.21.15.87:0/4059899627 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab4c0c37e0 con 0x55d5267d2a80 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab3bfff640 1 -- 172.21.15.87:0/4059899627 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0bf690 con 0x55d5267d2a80 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- 172.21.15.87:0/4059899627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 msgr2=0x55d5267ea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009430 tx=0x7fab4c0bb040 comp rx=0 tx=0).stop 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- 172.21.15.87:0/4059899627 shutdown_connections 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ea7c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 --2- 172.21.15.87:0/4059899627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- 172.21.15.87:0/4059899627 >> 172.21.15.87:0/4059899627 conn(0x55d5267c3cb0 msgr2=0x55d5267cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- 172.21.15.87:0/4059899627 shutdown_connections 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- 172.21.15.87:0/4059899627 wait complete. 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.066+0000 7fab5b969740 1 -- start start 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 Processor -- start 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267db190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab541ae2e0 con 0x55d5267d2a80 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab541af750 con 0x55d5267c9ea0 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab541ae9b0 con 0x55d5267c9580 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 10 client.0 _ll_register_callbacks cb 0x7fab480096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35554/0 (socket says 172.21.15.87:35554) 2024-04-22T13:36:42.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab51d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 -- 172.21.15.87:0/917344199 learned_addr learned my addr 172.21.15.87:0/917344199 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 -- 172.21.15.87:0/917344199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 msgr2=0x55d5267db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267db190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 -- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 msgr2=0x55d5267cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 -- 172.21.15.87:0/917344199 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d526848030 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab52d9d640 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0bf130 tx=0x7fab4c0bf4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3041849b79cc9c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0dc030 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fab4c0de2c0 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.067+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab4c0c37e0 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fab4c0c80a0 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 10 client.17043 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d526849120 con 0x55d5267d2a80 2024-04-22T13:36:42.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab4c0c37e0 con 0x55d5267d2a80 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fab4c03c030 con 0x55d5267d2a80 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 10 client.17043 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 10 client.17043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 20 client.17043 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d52684b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 4 client.17043 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d526735a70 con 0x55d5267e2a80 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5259c640 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d52684b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab39ffb640 10 client.17043 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.068+0000 7fab5259c640 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d52684b8a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da40b71334675680 server_cookie=c1719cac8e06e647 in_seq=0 out_seq=0 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab39ffb640 1 -- 172.21.15.87:0/917344199 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55d526735a70 con 0x55d5267e2a80 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab39ffb640 10 client.17043 handle_command_reply: tid=0 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 client.17043 shutdown 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 msgr2=0x55d5267ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0bf130 tx=0x7fab4c0bf4a0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 shutdown_connections 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d5267e2a80 0x55d52684b8a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d5267d2a80 0x55d5267ccca0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5267c9ea0 0x55d5267cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab39ffb640 0 client.17043 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 --2- 172.21.15.87:0/917344199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d5267c9580 0x55d5267db190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 >> 172.21.15.87:0/917344199 conn(0x55d5267c3cb0 msgr2=0x55d5267c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 shutdown_connections 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 1 -- 172.21.15.87:0/917344199 wait complete. 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.069+0000 7fab5b969740 20 client.17043 trim_cache size 0 max 16384 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:42.072 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: "eaf1b230-a529-4804-a693-f7d5f0717cc5": { 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "eaf1b230-a529-4804-a693-f7d5f0717cc5", 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.073 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 msgr2=0x7fab3c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 0x7fab3c0a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab44009990 tx=0x7fab44009110 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 msgr2=0x7fab5419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fab4c02c700 tx=0x7fab4c034670 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 shutdown_connections 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab3c0a3da0 0x7fab3c0a6260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab5410e8f0 0x7fab5419a750 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab54101660 0x7fab541a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 --2- 172.21.15.87:0/1370919695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab54100d40 0x7fab541a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 >> 172.21.15.87:0/1370919695 conn(0x7fab54078000 msgr2=0x7fab5410feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 shutdown_connections 2024-04-22T13:36:42.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.070+0000 7fab58c9a640 1 -- 172.21.15.87:0/1370919695 wait complete. 2024-04-22T13:36:42.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:41 smithi087 ceph-mon[27999]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 109 op/s 2024-04-22T13:36:42.077 INFO:tasks.cephfs.filesystem:scrub status for tag:eaf1b230-a529-4804-a693-f7d5f0717cc5 - {'path': '/', 'tag': 'eaf1b230-a529-4804-a693-f7d5f0717cc5', 'options': 'recursive,force'} 2024-04-22T13:36:42.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:41 smithi120 ceph-mon[31101]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 109 op/s 2024-04-22T13:36:42.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:41 smithi132 ceph-mon[31531]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 109 op/s 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.238+0000 7f4fcbfff640 1 -- 172.21.15.87:0/2318199831 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x565239dd3980 con 0x565239ef03c0 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.238+0000 7f4fcbfff640 10 client.26150 handle_command_reply: tid=0 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.238+0000 7f4ff19dd740 1 client.26150 shutdown 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 msgr2=0x565239e7cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e7cde0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0000f80 tx=0x7f4fe002c8c0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 shutdown_connections 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565239ef03c0 0x565239ef27e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x565239e798a0 0x565239e7d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239e70cc0 0x565239e7cde0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fcbfff640 0 client.26150 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 --2- 172.21.15.87:0/2318199831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565239e703a0 0x565239e819b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 >> 172.21.15.87:0/2318199831 conn(0x565239e6aad0 msgr2=0x565239e6c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 shutdown_connections 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 1 -- 172.21.15.87:0/2318199831 wait complete. 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4ff19dd740 20 client.26150 trim_cache size 0 max 16384 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 22, 2024-04-22T13:36:42.237 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "6bb0429": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 22, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 90.0, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:36:42.238 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:36:42.239 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 msgr2=0x7f4fcc0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 0x7f4fcc0a63f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x565239e6fe70 tx=0x7f4fd8014000 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 msgr2=0x7f4fe81a2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800ddc0 tx=0x7f4fd800a620 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 shutdown_connections 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.239+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4fcc0a3f30 0x7f4fcc0a63f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4fe81a7870 0x7f4fe81a7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe810eb30 0x7f4fe81a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 --2- 172.21.15.87:0/221981065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fe810df00 0x7f4fe81a2fb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 >> 172.21.15.87:0/221981065 conn(0x7f4fe806ce60 msgr2=0x7f4fe810bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 shutdown_connections 2024-04-22T13:36:42.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:36:42.240+0000 7f4fedab3640 1 -- 172.21.15.87:0/221981065 wait complete. 2024-04-22T13:36:42.248 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 6bb0429, quiesce took 2.9 seconds. Will release after: 30.0 2024-04-22T13:36:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:42 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:36:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:42 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:43.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:42 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:36:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:42 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:36:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:42 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:43.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:42 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:36:43.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:42 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:36:43.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:42 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:43.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:42 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:36:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:43 smithi120 ceph-mon[31101]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.0 MiB/s wr, 107 op/s 2024-04-22T13:36:44.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:43 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:43 smithi132 ceph-mon[31531]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.0 MiB/s wr, 107 op/s 2024-04-22T13:36:44.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:43 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:44.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:43 smithi087 ceph-mon[27999]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.0 MiB/s wr, 107 op/s 2024-04-22T13:36:44.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:43 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:36:46.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:45 smithi120 ceph-mon[31101]: pgmap v150: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 122 MiB/s wr, 1.23k op/s 2024-04-22T13:36:46.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:45 smithi132 ceph-mon[31531]: pgmap v150: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 122 MiB/s wr, 1.23k op/s 2024-04-22T13:36:46.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:45 smithi087 ceph-mon[27999]: pgmap v150: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 122 MiB/s wr, 1.23k op/s 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.619+0000 7fe69107a640 1 -- 172.21.15.87:0/436348259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c10c520 msgr2=0x7fe68c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.619+0000 7fe69107a640 1 --2- 172.21.15.87:0/436348259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c10c520 0x7fe68c10e9c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe684009900 tx=0x7fe68402bd60 comp rx=0 tx=0).stop 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 -- 172.21.15.87:0/436348259 shutdown_connections 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 --2- 172.21.15.87:0/436348259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c10c520 0x7fe68c10e9c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 --2- 172.21.15.87:0/436348259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c103b00 0x7fe68c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 --2- 172.21.15.87:0/436348259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c1031e0 0x7fe68c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 -- 172.21.15.87:0/436348259 >> 172.21.15.87:0/436348259 conn(0x7fe68c077fd0 msgr2=0x7fe68c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 -- 172.21.15.87:0/436348259 shutdown_connections 2024-04-22T13:36:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 -- 172.21.15.87:0/436348259 wait complete. 2024-04-22T13:36:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.620+0000 7fe69107a640 1 Processor -- start 2024-04-22T13:36:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 -- start start 2024-04-22T13:36:46.618 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c1031e0 0x7fe68c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 0x7fe68c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c112190 con 0x7fe68c103b00 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c112010 con 0x7fe68c1031e0 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe69107a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c112310 con 0x7fe68c10c520 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35576/0 (socket says 172.21.15.87:35576) 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 -- 172.21.15.87:0/2511197222 learned_addr learned my addr 172.21.15.87:0/2511197222 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c1031e0 0x7fe68c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68b59e640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 0x7fe68c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 -- 172.21.15.87:0/2511197222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 msgr2=0x7fe68c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 0x7fe68c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 -- 172.21.15.87:0/2511197222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c1031e0 msgr2=0x7fe68c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c1031e0 0x7fe68c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68a59c640 1 -- 172.21.15.87:0/2511197222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c19d5e0 con 0x7fe68c103b00 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.621+0000 7fe68b59e640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 0x7fe68c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe68a59c640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe68000d950 tx=0x7fe68000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe673fff640 1 -- 172.21.15.87:0/2511197222 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680017070 con 0x7fe68c103b00 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe673fff640 1 -- 172.21.15.87:0/2511197222 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe680009c30 con 0x7fe68c103b00 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c19d8d0 con 0x7fe68c103b00 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe673fff640 1 -- 172.21.15.87:0/2511197222 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000f620 con 0x7fe68c103b00 2024-04-22T13:36:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.622+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c1a5160 con 0x7fe68c103b00 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.624+0000 7fe673fff640 1 -- 172.21.15.87:0/2511197222 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe680010460 con 0x7fe68c103b00 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.624+0000 7fe693c6a740 1 Processor -- start 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.624+0000 7fe693c6a740 1 -- start start 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.624+0000 7fe673fff640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 0x7fe6740a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b58d0 0x561db97b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561db97198d0 con 0x561db97b4b60 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561db96b6e80 con 0x561db97be0e0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe693c6a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561db9715890 con 0x561db97b58d0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68b59e640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 0x7fe6740a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe673fff640 1 -- 172.21.15.87:0/2511197222 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe6800d0340 con 0x7fe68c103b00 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35586/0 (socket says 172.21.15.87:35586) 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 -- 172.21.15.87:0/3111852620 learned_addr learned my addr 172.21.15.87:0/3111852620 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b58d0 0x561db97b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68b59e640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 -- 172.21.15.87:0/3111852620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b58d0 msgr2=0x561db97b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b58d0 0x561db97b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 -- 172.21.15.87:0/3111852620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 msgr2=0x561db97c04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97c04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 -- 172.21.15.87:0/3111852620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db96d8320 con 0x561db97b4b60 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68b59e640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68b59e640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 0x7fe6740a6280 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe684005ab0 tx=0x7fe684004420 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe68a59c640 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a4a90 tx=0x7fe68009bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e116bb4db26e85ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.625+0000 7fe6717fa640 1 -- 172.21.15.87:0/3111852620 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe680020030 con 0x561db97b4b60 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe6717fa640 1 -- 172.21.15.87:0/3111852620 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68009ca50 con 0x561db97b4b60 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe6717fa640 1 -- 172.21.15.87:0/3111852620 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800a4fc0 con 0x561db97b4b60 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 -- 172.21.15.87:0/3111852620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 msgr2=0x561db97b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a4a90 tx=0x7fe68009bfb0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 -- 172.21.15.87:0/3111852620 shutdown_connections 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b4b60 0x561db97b4fe0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 --2- 172.21.15.87:0/3111852620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b58d0 0x561db97b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 -- 172.21.15.87:0/3111852620 >> 172.21.15.87:0/3111852620 conn(0x561db977bc90 msgr2=0x561db97af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 -- 172.21.15.87:0/3111852620 shutdown_connections 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.626+0000 7fe693c6a740 1 -- 172.21.15.87:0/3111852620 wait complete. 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 -- start start 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 Processor -- start 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97bda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97bb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561db97c0db0 con 0x561db97b58d0 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561db96a1eb0 con 0x561db97be0e0 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561db97c0f30 con 0x561db97b4b60 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe693c6a740 10 client.0 _ll_register_callbacks cb 0x561db96ceb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35588/0 (socket says 172.21.15.87:35588) 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 -- 172.21.15.87:0/626511895 learned_addr learned my addr 172.21.15.87:0/626511895 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68b59e640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97bda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 -- 172.21.15.87:0/626511895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 msgr2=0x561db97bda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97bda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 -- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 msgr2=0x561db97bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97bb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68a59c640 1 -- 172.21.15.87:0/626511895 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db942c210 con 0x561db97b58d0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97bda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.627+0000 7fe68b59e640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe68a59c640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a08c0 tx=0x7fe680009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de820d96de8fa760 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800b4020 con 0x561db97b58d0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe6800cdd80 con 0x561db97b58d0 2024-04-22T13:36:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe68009ca50 con 0x561db97b58d0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe680020070 con 0x561db97b58d0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 10 client.17055 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561db94f8620 con 0x561db97b58d0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68009ca50 con 0x561db97b58d0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6800980a0 con 0x561db97b58d0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 10 client.17055 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 10 client.17055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 20 client.17055 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db97cdee0 0x561db97d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 4 client.17055 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.628+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561db972d880 con 0x561db97cdee0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.629+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db97cdee0 0x561db97d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.629+0000 7fe66eff5640 10 client.17055 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.629+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db97cdee0 0x561db97d02d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d82b6a3ab61d6c server_cookie=b071891299f8d906 in_seq=0 out_seq=0 2024-04-22T13:36:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe66eff5640 1 -- 172.21.15.87:0/626511895 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561db972d880 con 0x561db97cdee0 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe66eff5640 10 client.17055 handle_command_reply: tid=0 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 client.17055 shutdown 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 msgr2=0x561db97bade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a08c0 tx=0x7fe680009e60 comp rx=0 tx=0).stop 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 shutdown_connections 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db97cdee0 0x561db97d02d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97be0e0 0x561db97bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe66eff5640 0 client.17055 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97bade0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 --2- 172.21.15.87:0/626511895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97bda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 >> 172.21.15.87:0/626511895 conn(0x561db977bc90 msgr2=0x561db97b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 shutdown_connections 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 1 -- 172.21.15.87:0/626511895 wait complete. 2024-04-22T13:36:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.630+0000 7fe693c6a740 20 client.17055 trim_cache size 0 max 16384 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 Processor -- start 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 -- start start 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 0x561db97c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97be0e0 0x561db97d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1b01f0 con 0x561db97b58d0 2024-04-22T13:36:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1adac0 con 0x561db97b4b60 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe693c6a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1ab910 con 0x561db97be0e0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35594/0 (socket says 172.21.15.87:35594) 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 -- 172.21.15.87:0/1665679320 learned_addr learned my addr 172.21.15.87:0/1665679320 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 0x561db97c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68b59e640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97be0e0 0x561db97d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 -- 172.21.15.87:0/1665679320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97be0e0 msgr2=0x561db97d5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97be0e0 0x561db97d5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 -- 172.21.15.87:0/1665679320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 msgr2=0x561db97c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 0x561db97c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 -- 172.21.15.87:0/1665679320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db97aaf30 con 0x561db97b58d0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 0x561db97c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.645+0000 7fe68a59c640 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe68009c2e0 tx=0x7fe680010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431d83f681109432 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe670ff9640 1 -- 172.21.15.87:0/1665679320 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800b4070 con 0x561db97b58d0 2024-04-22T13:36:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe670ff9640 1 -- 172.21.15.87:0/1665679320 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68009ca50 con 0x561db97b58d0 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe670ff9640 1 -- 172.21.15.87:0/1665679320 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800989c0 con 0x561db97b58d0 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- 172.21.15.87:0/1665679320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 msgr2=0x561db97b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe68009c2e0 tx=0x7fe680010cb0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- 172.21.15.87:0/1665679320 shutdown_connections 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97be0e0 0x561db97d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97b58d0 0x561db97b65d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 --2- 172.21.15.87:0/1665679320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b4b60 0x561db97c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- 172.21.15.87:0/1665679320 >> 172.21.15.87:0/1665679320 conn(0x561db977bc90 msgr2=0x561db97b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- 172.21.15.87:0/1665679320 shutdown_connections 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- 172.21.15.87:0/1665679320 wait complete. 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.646+0000 7fe693c6a740 1 -- start start 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 Processor -- start 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b58d0 0x561db97b8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1abf80 con 0x561db97be0e0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1ad3f0 con 0x561db97b58d0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1ac650 con 0x561db97b4b60 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe693c6a740 10 client.0 _ll_register_callbacks cb 0x7fe67c007910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35602/0 (socket says 172.21.15.87:35602) 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 -- 172.21.15.87:0/4157690384 learned_addr learned my addr 172.21.15.87:0/4157690384 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68a59c640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b58d0 0x561db97b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 -- 172.21.15.87:0/4157690384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 msgr2=0x561db97b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97b7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 -- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b58d0 msgr2=0x561db97b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b58d0 0x561db97b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 -- 172.21.15.87:0/4157690384 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db9833740 con 0x561db97be0e0 2024-04-22T13:36:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe68b59e640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe684036040 tx=0x7fe68402bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667192dabafbf3fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe66eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.647+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684052070 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe6840046c0 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe68009ca50 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe68405e050 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 10 client.17061 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561db9834910 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe68009ca50 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe684003a30 con 0x561db97be0e0 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 10 client.17061 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 10 client.17061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 20 client.17061 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db98350c0 0x561db98374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 4 client.17061 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:36:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561db96b93d0 con 0x561db98350c0 2024-04-22T13:36:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.648+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db98350c0 0x561db98374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.649+0000 7fe66eff5640 10 client.17061 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.649+0000 7fe68ad9d640 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db98350c0 0x561db98374e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5033788d8795ca server_cookie=ccc0297384c57866 in_seq=0 out_seq=0 2024-04-22T13:36:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.657+0000 7fe66eff5640 1 -- 172.21.15.87:0/4157690384 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561db96b93d0 con 0x561db98350c0 2024-04-22T13:36:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.657+0000 7fe66eff5640 10 client.17061 handle_command_reply: tid=0 2024-04-22T13:36:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.657+0000 7fe693c6a740 1 client.17061 shutdown 2024-04-22T13:36:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.657+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 msgr2=0x561db9833200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.657+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe684036040 tx=0x7fe68402bd60 comp rx=0 tx=0).stop 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 shutdown_connections 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561db98350c0 0x561db98374e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561db97be0e0 0x561db9833200 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db97b58d0 0x561db97b8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 --2- 172.21.15.87:0/4157690384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561db97b4b60 0x561db97b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 >> 172.21.15.87:0/4157690384 conn(0x561db977bc90 msgr2=0x561db97b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe66eff5640 0 client.17061 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 shutdown_connections 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 1 -- 172.21.15.87:0/4157690384 wait complete. 2024-04-22T13:36:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.658+0000 7fe693c6a740 20 client.17061 trim_cache size 0 max 16384 2024-04-22T13:36:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:36:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:36:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:36:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 msgr2=0x7fe6740a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 0x7fe6740a6280 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe684005ab0 tx=0x7fe684004420 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 msgr2=0x7fe68c198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe68000d950 tx=0x7fe68000de10 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 shutdown_connections 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6740a3dc0 0x7fe6740a6280 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe68c10c520 0x7fe68c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe68c103b00 0x7fe68c198b30 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 --2- 172.21.15.87:0/2511197222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe68c1031e0 0x7fe68c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 >> 172.21.15.87:0/2511197222 conn(0x7fe68c077fd0 msgr2=0x7fe68c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 shutdown_connections 2024-04-22T13:36:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:46.659+0000 7fe69107a640 1 -- 172.21.15.87:0/2511197222 wait complete. 2024-04-22T13:36:47.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:46 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:36:47.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:46 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:36:47.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:46 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:36:48.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:47 smithi120 ceph-mon[31101]: pgmap v151: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 169 MiB/s wr, 1.65k op/s 2024-04-22T13:36:48.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:47 smithi132 ceph-mon[31531]: pgmap v151: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 169 MiB/s wr, 1.65k op/s 2024-04-22T13:36:48.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:47 smithi087 ceph-mon[27999]: pgmap v151: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 169 MiB/s wr, 1.65k op/s 2024-04-22T13:36:50.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:49 smithi120 ceph-mon[31101]: pgmap v152: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 188 MiB/s wr, 1.78k op/s 2024-04-22T13:36:50.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:49 smithi132 ceph-mon[31531]: pgmap v152: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 188 MiB/s wr, 1.78k op/s 2024-04-22T13:36:50.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:49 smithi087 ceph-mon[27999]: pgmap v152: 97 pgs: 97 active+clean; 32 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 188 MiB/s wr, 1.78k op/s 2024-04-22T13:36:52.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:51 smithi120 ceph-mon[31101]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 200 MiB/s wr, 1.85k op/s 2024-04-22T13:36:52.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:51 smithi132 ceph-mon[31531]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 200 MiB/s wr, 1.85k op/s 2024-04-22T13:36:52.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:51 smithi087 ceph-mon[27999]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 200 MiB/s wr, 1.85k op/s 2024-04-22T13:36:54.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:53 smithi120 ceph-mon[31101]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 192 MiB/s wr, 1.75k op/s 2024-04-22T13:36:54.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:53 smithi132 ceph-mon[31531]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 192 MiB/s wr, 1.75k op/s 2024-04-22T13:36:54.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:53 smithi087 ceph-mon[27999]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 192 MiB/s wr, 1.75k op/s 2024-04-22T13:36:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:55 smithi120 ceph-mon[31101]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 211 MiB/s wr, 1.88k op/s 2024-04-22T13:36:56.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:55 smithi132 ceph-mon[31531]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 211 MiB/s wr, 1.88k op/s 2024-04-22T13:36:56.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:55 smithi087 ceph-mon[27999]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 211 MiB/s wr, 1.88k op/s 2024-04-22T13:36:58.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:57 smithi120 ceph-mon[31101]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 108 MiB/s wr, 840 op/s 2024-04-22T13:36:58.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:57 smithi132 ceph-mon[31531]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 108 MiB/s wr, 840 op/s 2024-04-22T13:36:58.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:57 smithi087 ceph-mon[27999]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 108 MiB/s wr, 840 op/s 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 -- 172.21.15.87:0/1442004395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 msgr2=0x7f202810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 --2- 172.21.15.87:0/1442004395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f202810dd40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f2020009ca0 tx=0x7f202002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 -- 172.21.15.87:0/1442004395 shutdown_connections 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 --2- 172.21.15.87:0/1442004395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f202810dd40 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 --2- 172.21.15.87:0/1442004395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101aa0 0x7f2028105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 --2- 172.21.15.87:0/1442004395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20281010f0 0x7f20281014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 -- 172.21.15.87:0/1442004395 >> 172.21.15.87:0/1442004395 conn(0x7f2028077fd0 msgr2=0x7f20280783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 -- 172.21.15.87:0/1442004395 shutdown_connections 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 -- 172.21.15.87:0/1442004395 wait complete. 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.756+0000 7f202d7a4640 1 Processor -- start 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 -- start start 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20281010f0 0x7f20281986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f202819cf70 0x7f202819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f20281121f0 con 0x7f2028106170 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2028112070 con 0x7f20281010f0 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f202d7a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2028112370 con 0x7f202819cf70 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34064/0 (socket says 172.21.15.87:34064) 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 -- 172.21.15.87:0/2103284824 learned_addr learned my addr 172.21.15.87:0/2103284824 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f2026ffd640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20281010f0 0x7f20281986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20277fe640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f202819cf70 0x7f202819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 -- 172.21.15.87:0/2103284824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f202819cf70 msgr2=0x7f202819d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f202819cf70 0x7f202819d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 -- 172.21.15.87:0/2103284824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20281010f0 msgr2=0x7f20281986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20281010f0 0x7f20281986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 -- 172.21.15.87:0/2103284824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202819da60 con 0x7f2028106170 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.757+0000 7f20267fc640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d770 tx=0x7f201c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.758+0000 7f200ffff640 1 -- 172.21.15.87:0/2103284824 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c017070 con 0x7f2028106170 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.758+0000 7f200ffff640 1 -- 172.21.15.87:0/2103284824 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c009c30 con 0x7f2028106170 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.758+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202819dd50 con 0x7f2028106170 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.758+0000 7f200ffff640 1 -- 172.21.15.87:0/2103284824 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00f620 con 0x7f2028106170 2024-04-22T13:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.758+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281a5540 con 0x7f2028106170 2024-04-22T13:36:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.760+0000 7f200ffff640 1 -- 172.21.15.87:0/2103284824 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f201c01a070 con 0x7f2028106170 2024-04-22T13:36:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.760+0000 7f2030397740 1 Processor -- start 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.760+0000 7f200ffff640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 0x7f20100a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.760+0000 7f2030397740 1 -- start start 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20277fe640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 0x7f20100a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f200ffff640 1 -- 172.21.15.87:0/2103284824 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f201c0d0270 con 0x7f2028106170 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56d8d0 0x56321f56c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f5760e0 0x56321f5784d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56321f4d18d0 con 0x56321f56cb60 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56321f46ee80 con 0x56321f5760e0 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2030397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56321f4cd890 con 0x56321f56d8d0 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34070/0 (socket says 172.21.15.87:34070) 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 -- 172.21.15.87:0/1626124560 learned_addr learned my addr 172.21.15.87:0/1626124560 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f2026ffd640 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56d8d0 0x56321f56c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20277fe640 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f5760e0 0x56321f5784d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 -- 172.21.15.87:0/1626124560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56d8d0 msgr2=0x56321f56c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56d8d0 0x56321f56c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 -- 172.21.15.87:0/1626124560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f5760e0 msgr2=0x56321f5784d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20277fe640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 0x7f20100a6280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2020005cc0 tx=0x7f2020009a00 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f5760e0 0x56321f5784d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 -- 172.21.15.87:0/1626124560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321f490320 con 0x56321f56cb60 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.761+0000 7f20267fc640 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f201c0ce030 tx=0x7f201c09bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625fd0c3f78851cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f200d7fa640 1 -- 172.21.15.87:0/1626124560 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c021040 con 0x56321f56cb60 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f200d7fa640 1 -- 172.21.15.87:0/1626124560 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c0cf050 con 0x56321f56cb60 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f200d7fa640 1 -- 172.21.15.87:0/1626124560 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c09b6d0 con 0x56321f56cb60 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 -- 172.21.15.87:0/1626124560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 msgr2=0x56321f56cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f201c0ce030 tx=0x7f201c09bce0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 -- 172.21.15.87:0/1626124560 shutdown_connections 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f5760e0 0x56321f5784d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56cb60 0x56321f56cfe0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 --2- 172.21.15.87:0/1626124560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56d8d0 0x56321f56c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 -- 172.21.15.87:0/1626124560 >> 172.21.15.87:0/1626124560 conn(0x56321f533c90 msgr2=0x56321f567850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 -- 172.21.15.87:0/1626124560 shutdown_connections 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.762+0000 7f2030397740 1 -- 172.21.15.87:0/1626124560 wait complete. 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 -- start start 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 Processor -- start 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f575ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56d8d0 0x56321f572e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56321f578db0 con 0x56321f5760e0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56321f459eb0 con 0x56321f56d8d0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56321f578f30 con 0x56321f56cb60 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2030397740 10 client.0 _ll_register_callbacks cb 0x56321f486b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34080/0 (socket says 172.21.15.87:34080) 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 -- 172.21.15.87:0/3386251007 learned_addr learned my addr 172.21.15.87:0/3386251007 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20267fc640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56d8d0 0x56321f572e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f2026ffd640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f575ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 -- 172.21.15.87:0/3386251007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 msgr2=0x56321f575ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f575ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 -- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56d8d0 msgr2=0x56321f572e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56d8d0 0x56321f572e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.763+0000 7f20277fe640 1 -- 172.21.15.87:0/3386251007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321f1e4210 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f20277fe640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2020009ca0 tx=0x7f202002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473897fe8c24392 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020050060 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2020004040 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f201c0cf050 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2020047070 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 10 client.17073 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56321f2b0620 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c0cf050 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2020053020 con 0x56321f5760e0 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 10 client.17073 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 20 client.17073 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f586120 0x56321f588510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 4 client.17073 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.764+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56321f4e5880 con 0x56321f586120 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.765+0000 7f2026ffd640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f586120 0x56321f588510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.765+0000 7f2026ffd640 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f586120 0x56321f588510 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e889a007a4b8916 server_cookie=5a998b75c2c115a4 in_seq=0 out_seq=0 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.765+0000 7f200aff5640 10 client.17073 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f200aff5640 1 -- 172.21.15.87:0/3386251007 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56321f4e5880 con 0x56321f586120 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f200aff5640 10 client.17073 handle_command_reply: tid=0 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 client.17073 shutdown 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 msgr2=0x56321f573470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2020009ca0 tx=0x7f202002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 shutdown_connections 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f586120 0x56321f588510 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f5760e0 0x56321f573470 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56d8d0 0x56321f572e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 --2- 172.21.15.87:0/3386251007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f575ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f200aff5640 0 client.17073 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 >> 172.21.15.87:0/3386251007 conn(0x56321f533c90 msgr2=0x56321f5695a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 shutdown_connections 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 1 -- 172.21.15.87:0/3386251007 wait complete. 2024-04-22T13:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.766+0000 7f2030397740 20 client.17073 trim_cache size 0 max 16384 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 Processor -- start 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 -- start start 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 0x56321f581cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56cb60 0x56321f57e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b0210 con 0x56321f56d8d0 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20281adae0 con 0x56321f56cb60 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2030397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20281ab950 con 0x56321f533c90 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34092/0 (socket says 172.21.15.87:34092) 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 -- 172.21.15.87:0/2336044668 learned_addr learned my addr 172.21.15.87:0/2336044668 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2026ffd640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 0x56321f581cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20267fc640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56cb60 0x56321f57e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 -- 172.21.15.87:0/2336044668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 msgr2=0x56321f581cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 0x56321f581cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 -- 172.21.15.87:0/2336044668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56cb60 msgr2=0x56321f57e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56cb60 0x56321f57e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 -- 172.21.15.87:0/2336044668 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321f56be30 con 0x56321f56d8d0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f2026ffd640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 0x56321f581cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f20277fe640 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2020032540 tx=0x7f20200095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d753ef13319d8380 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f200cff9640 1 -- 172.21.15.87:0/2336044668 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020050040 con 0x56321f56d8d0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f200cff9640 1 -- 172.21.15.87:0/2336044668 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2020042d00 con 0x56321f56d8d0 2024-04-22T13:36:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.781+0000 7f200cff9640 1 -- 172.21.15.87:0/2336044668 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020039040 con 0x56321f56d8d0 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- 172.21.15.87:0/2336044668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 msgr2=0x56321f575590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2020032540 tx=0x7f20200095c0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- 172.21.15.87:0/2336044668 shutdown_connections 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f575590 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f56cb60 0x56321f57e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 --2- 172.21.15.87:0/2336044668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f533c90 0x56321f581cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- 172.21.15.87:0/2336044668 >> 172.21.15.87:0/2336044668 conn(0x56321f56e830 msgr2=0x56321f571730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- 172.21.15.87:0/2336044668 shutdown_connections 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- 172.21.15.87:0/2336044668 wait complete. 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 -- start start 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.782+0000 7f2030397740 1 Processor -- start 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 0x56321f5e4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f5e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f20281abfa0 con 0x56321f56d8d0 2024-04-22T13:36:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20281ad410 con 0x56321f533c90 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20281ac670 con 0x56321f56cb60 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 10 client.0 _ll_register_callbacks cb 0x7f2020009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34104/0 (socket says 172.21.15.87:34104) 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 -- 172.21.15.87:0/1486165198 learned_addr learned my addr 172.21.15.87:0/1486165198 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20267fc640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f5e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2026ffd640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 0x56321f5e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 -- 172.21.15.87:0/1486165198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 msgr2=0x56321f5e4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f5e4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 -- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 msgr2=0x56321f5e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 0x56321f5e4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 -- 172.21.15.87:0/1486165198 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321f5efc00 con 0x56321f56d8d0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2026ffd640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 0x56321f5e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f20277fe640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2020006010 tx=0x7f2020055da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c713f6e4c3250ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:36:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f200aff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020062070 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f2020055650 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.783+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2020042d00 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2020050040 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 10 client.17079 fetch_fsmap learned FSMap version 23 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56321f5f0eb0 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2020042d00 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2020065030 con 0x56321f56d8d0 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 10 client.17079 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 10 client.17079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 20 client.17079 populate_metadata read hostname 'smithi087' 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f5760e0 0x56321f5f3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 4 client.17079 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:36:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56321f4713d0 con 0x56321f5760e0 2024-04-22T13:36:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.784+0000 7f2026ffd640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f5760e0 0x56321f5f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:36:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.785+0000 7f2026ffd640 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f5760e0 0x56321f5f3390 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df963c7d82656c42 server_cookie=6e85db2671db5d6 in_seq=0 out_seq=0 2024-04-22T13:36:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.785+0000 7f200aff5640 10 client.17079 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f200aff5640 1 -- 172.21.15.87:0/1486165198 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56321f4713d0 con 0x56321f5760e0 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f200aff5640 10 client.17079 handle_command_reply: tid=0 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 client.17079 shutdown 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 msgr2=0x56321f5ef690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2020006010 tx=0x7f2020055da0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 shutdown_connections 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56321f5760e0 0x56321f5f3390 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56321f56d8d0 0x56321f5ef690 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f200aff5640 0 client.17079 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56321f56cb60 0x56321f5e4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 --2- 172.21.15.87:0/1486165198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56321f533c90 0x56321f5e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 >> 172.21.15.87:0/1486165198 conn(0x56321f56e830 msgr2=0x56321f566040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 shutdown_connections 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 1 -- 172.21.15.87:0/1486165198 wait complete. 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.789+0000 7f2030397740 20 client.17079 trim_cache size 0 max 16384 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:36:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:36:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 msgr2=0x7f20100a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 0x7f20100a6280 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2020005cc0 tx=0x7f2020009a00 comp rx=0 tx=0).stop 2024-04-22T13:36:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 msgr2=0x7f2028198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d770 tx=0x7f201c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 shutdown_connections 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f20100a3dc0 0x7f20100a6280 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f202819cf70 0x7f202819d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2028106170 0x7f2028198bf0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 --2- 172.21.15.87:0/2103284824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20281010f0 0x7f20281986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 >> 172.21.15.87:0/2103284824 conn(0x7f2028077fd0 msgr2=0x7f20280fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 shutdown_connections 2024-04-22T13:36:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:36:59.790+0000 7f202d7a4640 1 -- 172.21.15.87:0/2103284824 wait complete. 2024-04-22T13:37:00.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:36:59 smithi120 ceph-mon[31101]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 78 MiB/s wr, 526 op/s 2024-04-22T13:37:00.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:36:59 smithi132 ceph-mon[31531]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 78 MiB/s wr, 526 op/s 2024-04-22T13:37:00.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:36:59 smithi087 ceph-mon[27999]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 78 MiB/s wr, 526 op/s 2024-04-22T13:37:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:01 smithi120 ceph-mon[31101]: pgmap v158: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 73 MiB/s wr, 493 op/s 2024-04-22T13:37:02.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:01 smithi132 ceph-mon[31531]: pgmap v158: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 73 MiB/s wr, 493 op/s 2024-04-22T13:37:02.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:01 smithi087 ceph-mon[27999]: pgmap v158: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 73 MiB/s wr, 493 op/s 2024-04-22T13:37:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:03 smithi120 ceph-mon[31101]: pgmap v159: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 411 op/s 2024-04-22T13:37:04.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:03 smithi132 ceph-mon[31531]: pgmap v159: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 411 op/s 2024-04-22T13:37:04.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:03 smithi087 ceph-mon[27999]: pgmap v159: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 61 MiB/s wr, 411 op/s 2024-04-22T13:37:06.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:05 smithi120 ceph-mon[31101]: pgmap v160: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 70 MiB/s wr, 500 op/s 2024-04-22T13:37:06.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:05 smithi132 ceph-mon[31531]: pgmap v160: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 70 MiB/s wr, 500 op/s 2024-04-22T13:37:06.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:05 smithi087 ceph-mon[27999]: pgmap v160: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 70 MiB/s wr, 500 op/s 2024-04-22T13:37:07.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:37:07.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:37:07.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:37:07.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:06 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:37:07.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:37:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:37:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:37:07.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:06 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:37:07.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:37:07.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:37:07.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:37:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:06 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:37:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:07 smithi120 ceph-mon[31101]: pgmap v161: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 460 op/s 2024-04-22T13:37:08.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:07 smithi132 ceph-mon[31531]: pgmap v161: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 460 op/s 2024-04-22T13:37:08.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:07 smithi087 ceph-mon[27999]: pgmap v161: 97 pgs: 97 active+clean; 33 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 460 op/s 2024-04-22T13:37:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:09 smithi120 ceph-mon[31101]: pgmap v162: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 60 MiB/s wr, 438 op/s 2024-04-22T13:37:10.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:09 smithi132 ceph-mon[31531]: pgmap v162: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 60 MiB/s wr, 438 op/s 2024-04-22T13:37:10.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:09 smithi087 ceph-mon[27999]: pgmap v162: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 60 MiB/s wr, 438 op/s 2024-04-22T13:37:12.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:37:12.078 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:37:12.103 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:11 smithi087 ceph-mon[27999]: pgmap v163: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 326 op/s 2024-04-22T13:37:12.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:11 smithi120 ceph-mon[31101]: pgmap v163: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 326 op/s 2024-04-22T13:37:12.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 -- 172.21.15.87:0/3822883999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78201031b0 msgr2=0x7f7820103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 --2- 172.21.15.87:0/3822883999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78201031b0 0x7f7820103590 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7814009900 tx=0x7f781402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 -- 172.21.15.87:0/3822883999 shutdown_connections 2024-04-22T13:37:12.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 --2- 172.21.15.87:0/3822883999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 0x7f782010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 --2- 172.21.15.87:0/3822883999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7820103ad0 0x7f782010bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 --2- 172.21.15.87:0/3822883999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f78201031b0 0x7f7820103590 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 -- 172.21.15.87:0/3822883999 >> 172.21.15.87:0/3822883999 conn(0x7f78200771d0 msgr2=0x7f78200775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 -- 172.21.15.87:0/3822883999 shutdown_connections 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.195+0000 7f78271b2640 1 -- 172.21.15.87:0/3822883999 wait complete. 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 Processor -- start 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 -- start start 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 0x7f7820198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 0x7f782019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78201121d0 con 0x7f7820103ad0 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7820112050 con 0x7f78201031b0 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f78271b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7820112350 con 0x7f782010c580 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f781ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f781ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48726/0 (socket says 172.21.15.87:48726) 2024-04-22T13:37:12.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f781ffff640 1 -- 172.21.15.87:0/1623950313 learned_addr learned my addr 172.21.15.87:0/1623950313 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f782574f640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 0x7f782019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.196+0000 7f7824f4e640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 0x7f7820198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 -- 172.21.15.87:0/1623950313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 msgr2=0x7f782019cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 0x7f782019cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 -- 172.21.15.87:0/1623950313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 msgr2=0x7f7820198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 0x7f7820198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 -- 172.21.15.87:0/1623950313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782019d720 con 0x7f7820103ad0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f7824f4e640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 0x7f7820198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781ffff640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f780c00d950 tx=0x7f780c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781dffb640 1 -- 172.21.15.87:0/1623950313 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c017030 con 0x7f7820103ad0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782019da10 con 0x7f7820103ad0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781dffb640 1 -- 172.21.15.87:0/1623950313 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780c004020 con 0x7f7820103ad0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f781dffb640 1 -- 172.21.15.87:0/1623950313 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c0047a0 con 0x7f7820103ad0 2024-04-22T13:37:12.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.197+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78201a52a0 con 0x7f7820103ad0 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.199+0000 7f781dffb640 1 -- 172.21.15.87:0/1623950313 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f780c009a30 con 0x7f7820103ad0 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 Processor -- start 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 -- start start 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781dffb640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 0x7f78100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5580 0x5623b04a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5623b040a1e0 con 0x5623b04a5ea0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5623b03a7390 con 0x5623b04aea80 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7829da4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5623b04061a0 con 0x5623b04a5580 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781dffb640 1 -- 172.21.15.87:0/1623950313 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f780c0d0210 con 0x7f7820103ad0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f782574f640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 0x7f78100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48734/0 (socket says 172.21.15.87:48734) 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781ffff640 1 -- 172.21.15.87:0/347932568 learned_addr learned my addr 172.21.15.87:0/347932568 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f7824f4e640 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5580 0x5623b04a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f782574f640 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781ffff640 1 -- 172.21.15.87:0/347932568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5580 msgr2=0x5623b04a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.200+0000 7f781ffff640 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5580 0x5623b04a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f781ffff640 1 -- 172.21.15.87:0/347932568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 msgr2=0x5623b04b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f781ffff640 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f781ffff640 1 -- 172.21.15.87:0/347932568 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b01e67d0 con 0x5623b04a5ea0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f782574f640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 0x7f78100a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f7818013b60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f781ffff640 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f780c005fd0 tx=0x7f780c014c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56034a470bf72c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f780affd640 1 -- 172.21.15.87:0/347932568 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c014e70 con 0x5623b04a5ea0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f780affd640 1 -- 172.21.15.87:0/347932568 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780c09c920 con 0x5623b04a5ea0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.201+0000 7f780affd640 1 -- 172.21.15.87:0/347932568 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c009d20 con 0x5623b04a5ea0 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- 172.21.15.87:0/347932568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 msgr2=0x5623b04ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f780c005fd0 tx=0x7f780c014c90 comp rx=0 tx=0).stop 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- 172.21.15.87:0/347932568 shutdown_connections 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5ea0 0x5623b04ae540 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 --2- 172.21.15.87:0/347932568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5580 0x5623b04a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- 172.21.15.87:0/347932568 >> 172.21.15.87:0/347932568 conn(0x5623b049fcb0 msgr2=0x5623b04a0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- 172.21.15.87:0/347932568 shutdown_connections 2024-04-22T13:37:12.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- 172.21.15.87:0/347932568 wait complete. 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 -- start start 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.202+0000 7f7829da4740 1 Processor -- start 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5ea0 0x5623b04abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04ac530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5623b04b1870 con 0x5623b04a5580 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5623b03c32f0 con 0x5623b04aea80 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5623b04b19f0 con 0x5623b04a5ea0 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 10 client.0 _ll_register_callbacks cb 0x5623b03d0580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48740/0 (socket says 172.21.15.87:48740) 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 -- 172.21.15.87:0/285846450 learned_addr learned my addr 172.21.15.87:0/285846450 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f781ffff640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5ea0 0x5623b04abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f782574f640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 -- 172.21.15.87:0/285846450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5ea0 msgr2=0x5623b04abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5ea0 0x5623b04abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 -- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 msgr2=0x5623b04ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 -- 172.21.15.87:0/285846450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b011c3f0 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f782574f640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7824f4e640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7814009900 tx=0x7f78140052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e787b4e75ee37b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f77fffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814042070 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f781402ca80 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.203+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f780c09c920 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f781404e050 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 10 client.17091 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5623b00dc0f0 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f780c09c920 con 0x5623b04a5580 2024-04-22T13:37:12.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7814004c90 con 0x5623b04a5580 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 10 client.17091 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 20 client.17091 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b04bea10 0x5623b04c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 4 client.17091 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623b0417c00 con 0x5623b04bea10 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.204+0000 7f781ffff640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b04bea10 0x5623b04c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f781ffff640 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b04bea10 0x5623b04c0eb0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c5f8670f56a049a server_cookie=da18e08446182401 in_seq=0 out_seq=0 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f77fffff640 10 client.17091 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f77fffff640 1 -- 172.21.15.87:0/285846450 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623b0417c00 con 0x5623b04bea10 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f77fffff640 10 client.17091 handle_command_reply: tid=0 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f7829da4740 1 client.17091 shutdown 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 msgr2=0x5623b04abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.205+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7814009900 tx=0x7f78140052f0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 shutdown_connections 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b04bea10 0x5623b04c0eb0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04aea80 0x5623b04ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04a5ea0 0x5623b04abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f77fffff640 0 client.17091 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 --2- 172.21.15.87:0/285846450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04abab0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 >> 172.21.15.87:0/285846450 conn(0x5623b049fcb0 msgr2=0x5623b04a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 shutdown_connections 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 1 -- 172.21.15.87:0/285846450 wait complete. 2024-04-22T13:37:12.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.206+0000 7f7829da4740 20 client.17091 trim_cache size 0 max 16384 2024-04-22T13:37:12.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:11 smithi132 ceph-mon[31531]: pgmap v163: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 43 MiB/s wr, 326 op/s 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 Processor -- start 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 -- start start 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04ba730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b049ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78201b0300 con 0x5623b04a5580 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78201adbd0 con 0x5623b04a5ea0 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7829da4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78201aba40 con 0x5623b04aea80 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48752/0 (socket says 172.21.15.87:48752) 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 -- 172.21.15.87:0/943487955 learned_addr learned my addr 172.21.15.87:0/943487955 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f782574f640 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b049ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 -- 172.21.15.87:0/943487955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 msgr2=0x5623b049ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b049ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 -- 172.21.15.87:0/943487955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 msgr2=0x5623b04ba730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f781ffff640 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04ba730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04ba730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 -- 172.21.15.87:0/943487955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b04a5cd0 con 0x5623b04a5580 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.223+0000 7f7824f4e640 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7814005940 tx=0x7f7814005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db54dfafd50adbd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f780a7fc640 1 -- 172.21.15.87:0/943487955 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814057070 con 0x5623b04a5580 2024-04-22T13:37:12.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f780a7fc640 1 -- 172.21.15.87:0/943487955 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781404b030 con 0x5623b04a5580 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f780a7fc640 1 -- 172.21.15.87:0/943487955 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814042040 con 0x5623b04a5580 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 -- 172.21.15.87:0/943487955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 msgr2=0x5623b04ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7814005940 tx=0x7f7814005a20 comp rx=0 tx=0).stop 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 -- 172.21.15.87:0/943487955 shutdown_connections 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b049ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04ba730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 --2- 172.21.15.87:0/943487955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04ae1e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 -- 172.21.15.87:0/943487955 >> 172.21.15.87:0/943487955 conn(0x5623b049fcb0 msgr2=0x5623b049e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 -- 172.21.15.87:0/943487955 shutdown_connections 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.224+0000 7f7829da4740 1 -- 172.21.15.87:0/943487955 wait complete. 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 -- start start 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 Processor -- start 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b04b3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f78201ac090 con 0x5623b04a5580 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78201ad500 con 0x5623b04a5ea0 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78201ac760 con 0x5623b04aea80 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7829da4740 10 client.0 _ll_register_callbacks cb 0x7f78180093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48756/0 (socket says 172.21.15.87:48756) 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 -- 172.21.15.87:0/3583838382 learned_addr learned my addr 172.21.15.87:0/3583838382 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f782574f640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b04b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f781ffff640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 -- 172.21.15.87:0/3583838382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 msgr2=0x5623b04b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b04b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 -- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 msgr2=0x5623b04b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f7824f4e640 1 -- 172.21.15.87:0/3583838382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b0524150 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.225+0000 7f781ffff640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7824f4e640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7814055040 tx=0x7f781402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7a08e84bd9e84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814057070 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f7814003a30 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781404b030 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7814042040 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 10 client.17097 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5623b0525440 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781404b030 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7814004c90 con 0x5623b04a5580 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 10 client.17097 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 20 client.17097 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b0525af0 0x5623b0527ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 4 client.17097 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.226+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623b0411a70 con 0x5623b0525af0 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f781ffff640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b0525af0 0x5623b0527ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f77fffff640 10 client.17097 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f781ffff640 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b0525af0 0x5623b0527ee0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e13877fe7fbda439 server_cookie=8d08d505c991c04c in_seq=0 out_seq=0 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f77fffff640 1 -- 172.21.15.87:0/3583838382 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5623b0411a70 con 0x5623b0525af0 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f77fffff640 10 client.17097 handle_command_reply: tid=0 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 client.17097 shutdown 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 msgr2=0x5623b04b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7814055040 tx=0x7f781402cb30 comp rx=0 tx=0).stop 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 shutdown_connections 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5623b0525af0 0x5623b0527ee0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5623b04aea80 0x5623b04b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5623b04a5ea0 0x5623b04b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f77fffff640 0 client.17097 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.227+0000 7f7829da4740 1 --2- 172.21.15.87:0/3583838382 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5623b04a5580 0x5623b04b27c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 >> 172.21.15.87:0/3583838382 conn(0x5623b049fcb0 msgr2=0x5623b049ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 shutdown_connections 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f7829da4740 1 -- 172.21.15.87:0/3583838382 wait complete. 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f7829da4740 20 client.17097 trim_cache size 0 max 16384 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:37:12.230 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 msgr2=0x7f78100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 0x7f78100a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f7818013b60 comp rx=0 tx=0).stop 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 msgr2=0x7f7820198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f780c00d950 tx=0x7f780c00de10 comp rx=0 tx=0).stop 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 shutdown_connections 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f78100a3d20 0x7f78100a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f782010c580 0x7f782019cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7820103ad0 0x7f7820198c70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 --2- 172.21.15.87:0/1623950313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78201031b0 0x7f7820198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 >> 172.21.15.87:0/1623950313 conn(0x7f78200771d0 msgr2=0x7f78200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 shutdown_connections 2024-04-22T13:37:12.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.228+0000 7f78271b2640 1 -- 172.21.15.87:0/1623950313 wait complete. 2024-04-22T13:37:12.235 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:37:12.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:37:12.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:37:12.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 -- 172.21.15.87:0/1885752701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101ad0 msgr2=0x7fc43410dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1885752701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101ad0 0x7fc43410dcb0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009cc0 tx=0x7fc42c02e310 comp rx=0 tx=0).stop 2024-04-22T13:37:12.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 -- 172.21.15.87:0/1885752701 shutdown_connections 2024-04-22T13:37:12.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1885752701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101ad0 0x7fc43410dcb0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1885752701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434101130 0x7fc434101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1885752701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434106f10 0x7fc4341072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 -- 172.21.15.87:0/1885752701 >> 172.21.15.87:0/1885752701 conn(0x7fc434077e30 msgr2=0x7fc434078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 -- 172.21.15.87:0/1885752701 shutdown_connections 2024-04-22T13:37:12.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7fc43ab50640 1 -- 172.21.15.87:0/1885752701 wait complete. 2024-04-22T13:37:12.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 -- 172.21.15.87:0/1696200466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18001011a0 msgr2=0x7f1800101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 --2- 172.21.15.87:0/1696200466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18001011a0 0x7f1800101620 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009890 tx=0x7f17f002bb70 comp rx=0 tx=0).stop 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 -- 172.21.15.87:0/1696200466 shutdown_connections 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 --2- 172.21.15.87:0/1696200466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1800101b60 0x7f180010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 --2- 172.21.15.87:0/1696200466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18001011a0 0x7f1800101620 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 --2- 172.21.15.87:0/1696200466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800105e30 0x7f1800106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.381+0000 7f1807b7e640 1 -- 172.21.15.87:0/1696200466 >> 172.21.15.87:0/1696200466 conn(0x7f1800077e30 msgr2=0x7f1800078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 -- 172.21.15.87:0/1696200466 shutdown_connections 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 -- 172.21.15.87:0/1696200466 wait complete. 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 Processor -- start 2024-04-22T13:37:12.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 -- start start 2024-04-22T13:37:12.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 Processor -- start 2024-04-22T13:37:12.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 -- start start 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 0x7fc434196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 0x7fc43419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc434110040 con 0x7fc434101130 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410fec0 con 0x7fc434106f10 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc43ab50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4341101c0 con 0x7fc434101ad0 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc4388ec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc4388ec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48794/0 (socket says 172.21.15.87:48794) 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7fc4388ec640 1 -- 172.21.15.87:0/1472373261 learned_addr learned my addr 172.21.15.87:0/1472373261 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800101b60 0x7f1800197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.382+0000 7f1807b7e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1800105e30 0x7f180019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f1807b7e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f18001110c0 con 0x7f18001011a0 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f1807b7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1800110f40 con 0x7f1800101b60 2024-04-22T13:37:12.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f1807b7e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1800111240 con 0x7f1800105e30 2024-04-22T13:37:12.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc433fff640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 0x7fc434196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48796/0 (socket says 172.21.15.87:48796) 2024-04-22T13:37:12.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 -- 172.21.15.87:0/2759979648 learned_addr learned my addr 172.21.15.87:0/2759979648 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4390ed640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 0x7fc43419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180611b640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1800105e30 0x7f180019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 -- 172.21.15.87:0/1472373261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 msgr2=0x7fc434196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 0x7fc434196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f1805119640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800101b60 0x7f1800197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 -- 172.21.15.87:0/1472373261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 msgr2=0x7fc43419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 0x7fc43419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 -- 172.21.15.87:0/1472373261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43419b570 con 0x7fc434101130 2024-04-22T13:37:12.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4390ed640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 0x7fc43419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc433fff640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 0x7fc434196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 -- 172.21.15.87:0/2759979648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1800105e30 msgr2=0x7f180019bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1800105e30 0x7f180019bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 -- 172.21.15.87:0/2759979648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800101b60 msgr2=0x7f1800197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800101b60 0x7f1800197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 -- 172.21.15.87:0/2759979648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180019c630 con 0x7f18001011a0 2024-04-22T13:37:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc4388ec640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc42800e9f0 tx=0x7fc42800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f180591a640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00d950 tx=0x7f17fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42800cd80 con 0x7fc434101130 2024-04-22T13:37:12.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43419b860 con 0x7fc434101130 2024-04-22T13:37:12.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4280054d0 con 0x7fc434101130 2024-04-22T13:37:12.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428010690 con 0x7fc434101130 2024-04-22T13:37:12.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc017030 con 0x7f18001011a0 2024-04-22T13:37:12.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4341a30f0 con 0x7fc434101130 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17fc004020 con 0x7f18001011a0 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.383+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180019c920 con 0x7f18001011a0 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.384+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18000690e0 con 0x7f18001011a0 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.384+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0047a0 con 0x7f18001011a0 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.385+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434069230 con 0x7fc434101130 2024-04-22T13:37:12.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.385+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e4005290 con 0x7f18001011a0 2024-04-22T13:37:12.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.386+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc428004e90 con 0x7fc434101130 2024-04-22T13:37:12.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.386+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f17fc0041d0 con 0x7f18001011a0 2024-04-22T13:37:12.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.386+0000 7fc431ffb640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 0x7fc4240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.386+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fc4280d0c50 con 0x7fc434101130 2024-04-22T13:37:12.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.386+0000 7fc433fff640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 0x7fc4240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.387+0000 7f17f6ffd640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 0x7f17ec0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.387+0000 7f1805119640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 0x7f17ec0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.387+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f17fc0d05c0 con 0x7f18001011a0 2024-04-22T13:37:12.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.387+0000 7fc433fff640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 0x7fc4240a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4200058c0 tx=0x7fc420005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.387+0000 7f1805119640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 0x7f17ec0a8540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097b0 tx=0x7f17f00058d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.391+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f17fc09cdd0 con 0x7f18001011a0 2024-04-22T13:37:12.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.396+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4280d6020 con 0x7fc434101130 2024-04-22T13:37:12.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.609+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17e4005580 con 0x7f18001011a0 2024-04-22T13:37:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.609+0000 7f17f6ffd640 1 -- 172.21.15.87:0/2759979648 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f17fc09c6d0 con 0x7f18001011a0 2024-04-22T13:37:12.605 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:37:12.605 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:37:12.606 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 msgr2=0x7f17ec0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 0x7f17ec0a8540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097b0 tx=0x7f17f00058d0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 msgr2=0x7f1800197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00d950 tx=0x7f17fc00de10 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 shutdown_connections 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17ec0a6080 0x7f17ec0a8540 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1800105e30 0x7f180019bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800101b60 0x7f1800197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.611+0000 7f1807b7e640 1 --2- 172.21.15.87:0/2759979648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18001011a0 0x7f1800197640 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.612+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 >> 172.21.15.87:0/2759979648 conn(0x7f1800077e30 msgr2=0x7f18000fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.612+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 shutdown_connections 2024-04-22T13:37:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.612+0000 7f1807b7e640 1 -- 172.21.15.87:0/2759979648 wait complete. 2024-04-22T13:37:12.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.614+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4341978d0 con 0x7fc434101130 2024-04-22T13:37:12.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.615+0000 7fc431ffb640 1 -- 172.21.15.87:0/1472373261 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fc4280bbe20 con 0x7fc434101130 2024-04-22T13:37:12.611 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:37:12.611 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:37:12.611 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 msgr2=0x7fc4240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 0x7fc4240a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4200058c0 tx=0x7fc420005850 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 msgr2=0x7fc434196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc42800e9f0 tx=0x7fc42800edf0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 shutdown_connections 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4240a3d20 0x7fc4240a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc434106f10 0x7fc43419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc434101ad0 0x7fc434196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 --2- 172.21.15.87:0/1472373261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc434101130 0x7fc434196580 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 >> 172.21.15.87:0/1472373261 conn(0x7fc434077e30 msgr2=0x7fc4340ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 shutdown_connections 2024-04-22T13:37:12.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.617+0000 7fc43ab50640 1 -- 172.21.15.87:0/1472373261 wait complete. 2024-04-22T13:37:12.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:37:12.624 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 6bb0429 --release --await' 2024-04-22T13:37:12.625 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 6bb0429 --release --await 2024-04-22T13:37:12.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 -- 172.21.15.87:0/883331541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c1010a0 msgr2=0x7f9d4c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 --2- 172.21.15.87:0/883331541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c101480 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009900 tx=0x7f9d4402bc30 comp rx=0 tx=0).stop 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 -- 172.21.15.87:0/883331541 shutdown_connections 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 --2- 172.21.15.87:0/883331541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c106170 0x7f9d4c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 --2- 172.21.15.87:0/883331541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c101a50 0x7f9d4c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 --2- 172.21.15.87:0/883331541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c101480 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.725+0000 7f9d51740640 1 -- 172.21.15.87:0/883331541 >> 172.21.15.87:0/883331541 conn(0x7f9d4c0771d0 msgr2=0x7f9d4c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- 172.21.15.87:0/883331541 shutdown_connections 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- 172.21.15.87:0/883331541 wait complete. 2024-04-22T13:37:12.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 Processor -- start 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- start start 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c106170 0x7f9d4c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1110d0 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c110f50 con 0x7f9d4c1010a0 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.726+0000 7f9d51740640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c111250 con 0x7f9d4c106170 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48808/0 (socket says 172.21.15.87:48808) 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 -- 172.21.15.87:0/2953915815 learned_addr learned my addr 172.21.15.87:0/2953915815 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c106170 0x7f9d4c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4affd640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 -- 172.21.15.87:0/2953915815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c106170 msgr2=0x7f9d4c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c106170 0x7f9d4c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 -- 172.21.15.87:0/2953915815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c1010a0 msgr2=0x7f9d4c1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c1995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 -- 172.21.15.87:0/2953915815 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1a4e70 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d4b7fe640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00e970 tx=0x7f9d3c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d43fff640 1 -- 172.21.15.87:0/2953915815 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00ccc0 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c1a5160 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d43fff640 1 -- 172.21.15.87:0/2953915815 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c00ce60 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d43fff640 1 -- 172.21.15.87:0/2953915815 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c010430 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.727+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c1a5660 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d43fff640 1 -- 172.21.15.87:0/2953915815 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d3c004ff0 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d5440d740 1 Processor -- start 2024-04-22T13:37:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d5440d740 1 -- start start 2024-04-22T13:37:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c73b0 0x5620c12c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c7d60 0x5620c12d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.730+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d5440d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5620c122c1f0 con 0x5620c12d0e60 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d5440d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5620c11c94b0 con 0x5620c12c73b0 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d5440d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5620c12281b0 con 0x5620c12c7d60 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48818/0 (socket says 172.21.15.87:48818) 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/3641105910 learned_addr learned my addr 172.21.15.87:0/3641105910 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4affd640 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c7d60 0x5620c12d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c73b0 0x5620c12c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d43fff640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 0x7f9d380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/3641105910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c7d60 msgr2=0x5620c12d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c7d60 0x5620c12d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/3641105910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c73b0 msgr2=0x5620c12c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c73b0 0x5620c12c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/3641105910 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620c11bb1b0 con 0x5620c12d0e60 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4b7fe640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 0x7f9d380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9d340117a0 tx=0x7f9d34005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19edc4cc2a6fe4dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d43fff640 1 -- 172.21.15.87:0/2953915815 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9d3c0d0b20 con 0x7f9d4c19a0c0 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d417fa640 1 -- 172.21.15.87:0/3641105910 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34011050 con 0x5620c12d0e60 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d417fa640 1 -- 172.21.15.87:0/3641105910 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d34003e20 con 0x5620c12d0e60 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.731+0000 7f9d417fa640 1 -- 172.21.15.87:0/3641105910 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34004220 con 0x5620c12d0e60 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d4b7fe640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 0x7f9d380a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c099500 tx=0x7f9d3c0bd040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 -- 172.21.15.87:0/3641105910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 msgr2=0x5620c12d12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9d340117a0 tx=0x7f9d34005bd0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 -- 172.21.15.87:0/3641105910 shutdown_connections 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d12c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c7d60 0x5620c12d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 --2- 172.21.15.87:0/3641105910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c73b0 0x5620c12c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 -- 172.21.15.87:0/3641105910 >> 172.21.15.87:0/3641105910 conn(0x5620c12c1ba0 msgr2=0x5620c12c1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 -- 172.21.15.87:0/3641105910 shutdown_connections 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.732+0000 7f9d5440d740 1 -- 172.21.15.87:0/3641105910 wait complete. 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 -- start start 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 Processor -- start 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5620c12d7da0 con 0x5620c12c7d60 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5620c11e5500 con 0x5620c12d0e60 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5620c12d7f20 con 0x5620c12c73b0 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d5440d740 10 client.0 _ll_register_callbacks cb 0x5620c11b3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48822/0 (socket says 172.21.15.87:48822) 2024-04-22T13:37:12.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 -- 172.21.15.87:0/4257498875 learned_addr learned my addr 172.21.15.87:0/4257498875 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 -- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 msgr2=0x5620c12d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12d7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 -- 172.21.15.87:0/4257498875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 msgr2=0x5620c12d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4affd640 1 -- 172.21.15.87:0/4257498875 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620c0f3e3f0 con 0x5620c12c7d60 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.733+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d4affd640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009900 tx=0x7f9d4402c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60452b8d39862f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4402c610 con 0x5620c12c7d60 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9d44005030 con 0x5620c12c7d60 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d34003e20 con 0x5620c12c7d60 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9d44005230 con 0x5620c12c7d60 2024-04-22T13:37:12.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 10 client.17121 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5620c10087d0 con 0x5620c12c7d60 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d34003e20 con 0x5620c12c7d60 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9d44003680 con 0x5620c12c7d60 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 10 client.17121 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 10 client.17121 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 20 client.17121 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c12e5070 0x5620c12e7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 4 client.17121 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.734+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620c11cb930 con 0x5620c12e5070 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.735+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c12e5070 0x5620c12e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.735+0000 7f9d327fc640 10 client.17121 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.735+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c12e5070 0x5620c12e7460 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49d3b8a92eb32e2d server_cookie=cd4e11644b7ad7a2 in_seq=0 out_seq=0 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d327fc640 1 -- 172.21.15.87:0/4257498875 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620c11cb930 con 0x5620c12e5070 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d327fc640 10 client.17121 handle_command_reply: tid=0 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 client.17121 shutdown 2024-04-22T13:37:12.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 msgr2=0x5620c12d4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009900 tx=0x7f9d4402c9f0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 shutdown_connections 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c12e5070 0x5620c12e7460 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d4660 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 --2- 172.21.15.87:0/4257498875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d327fc640 0 client.17121 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 >> 172.21.15.87:0/4257498875 conn(0x5620c12c1ba0 msgr2=0x5620c12c38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 shutdown_connections 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 1 -- 172.21.15.87:0/4257498875 wait complete. 2024-04-22T13:37:12.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.736+0000 7f9d5440d740 20 client.17121 trim_cache size 0 max 16384 2024-04-22T13:37:12.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.749+0000 7f9d5440d740 1 Processor -- start 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.749+0000 7f9d5440d740 1 -- start start 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 0x5620c12d2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1ae100 con 0x5620c12d0e60 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1ab9d0 con 0x5620c12c7d60 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d5440d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1a9860 con 0x5620c12c73b0 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60466/0 (socket says 172.21.15.87:60466) 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 0x5620c12d2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/1836714132 learned_addr learned my addr 172.21.15.87:0/1836714132 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/1836714132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 msgr2=0x5620c12d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 0x5620c12d2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/1836714132 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 msgr2=0x5620c12d6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 -- 172.21.15.87:0/1836714132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620c12d4f00 con 0x5620c12c73b0 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4affd640 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 0x5620c12d2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.750+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400fd00 tx=0x7f9d3400fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc83d06271c59a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d40ff9640 1 -- 172.21.15.87:0/1836714132 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400fda0 con 0x5620c12c73b0 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d40ff9640 1 -- 172.21.15.87:0/1836714132 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d34003e20 con 0x5620c12c73b0 2024-04-22T13:37:12.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d40ff9640 1 -- 172.21.15.87:0/1836714132 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3402e680 con 0x5620c12c73b0 2024-04-22T13:37:12.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 -- 172.21.15.87:0/1836714132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 msgr2=0x5620c12e0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400fd00 tx=0x7f9d3400fc90 comp rx=0 tx=0).stop 2024-04-22T13:37:12.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 -- 172.21.15.87:0/1836714132 shutdown_connections 2024-04-22T13:37:12.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12d0e60 0x5620c12d6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12c7d60 0x5620c12d2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 --2- 172.21.15.87:0/1836714132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12e0ca0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 -- 172.21.15.87:0/1836714132 >> 172.21.15.87:0/1836714132 conn(0x5620c12c1ba0 msgr2=0x5620c12c3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 -- 172.21.15.87:0/1836714132 shutdown_connections 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.751+0000 7f9d5440d740 1 -- 172.21.15.87:0/1836714132 wait complete. 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 -- start start 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 Processor -- start 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1a9eb0 con 0x5620c12c7d60 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1ab300 con 0x5620c12d0e60 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1aa560 con 0x5620c12c73b0 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d5440d740 10 client.0 _ll_register_callbacks cb 0x7f9d44004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48836/0 (socket says 172.21.15.87:48836) 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 -- 172.21.15.87:0/383051488 learned_addr learned my addr 172.21.15.87:0/383051488 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 -- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 msgr2=0x5620c12dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12dd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 -- 172.21.15.87:0/383051488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 msgr2=0x5620c12d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4affd640 1 -- 172.21.15.87:0/383051488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620c12d2590 con 0x5620c12c7d60 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.752+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d4affd640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x5620c12d3ec0 tx=0x7f9d44003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ea52d0741cf8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4402c610 con 0x5620c12c7d60 2024-04-22T13:37:12.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f9d44034490 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d34003e20 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9d44063050 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 10 client.17127 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5620c134b600 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d34003e20 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9d440046c0 con 0x5620c12c7d60 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 10 client.17127 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 10 client.17127 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 20 client.17127 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c134bd80 0x5620c134e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 4 client.17127 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.753+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620c1239c10 con 0x5620c134bd80 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c134bd80 0x5620c134e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d327fc640 10 client.17127 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d4a7fc640 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c134bd80 0x5620c134e1a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=91872482b8c1b60a server_cookie=d8a9f9861bd52177 in_seq=0 out_seq=0 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d327fc640 1 -- 172.21.15.87:0/383051488 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620c1239c10 con 0x5620c134bd80 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d327fc640 10 client.17127 handle_command_reply: tid=0 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 client.17127 shutdown 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 msgr2=0x5620c12d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x5620c12d3ec0 tx=0x7f9d44003ef0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 shutdown_connections 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5620c134bd80 0x5620c134e1a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5620c12d0e60 0x5620c12d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d327fc640 0 client.17127 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5620c12c7d60 0x5620c12d1820 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 --2- 172.21.15.87:0/383051488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620c12c73b0 0x5620c12dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 >> 172.21.15.87:0/383051488 conn(0x5620c12c1ba0 msgr2=0x5620c12ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 shutdown_connections 2024-04-22T13:37:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 1 -- 172.21.15.87:0/383051488 wait complete. 2024-04-22T13:37:12.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.754+0000 7f9d5440d740 20 client.17127 trim_cache size 0 max 16384 2024-04-22T13:37:12.757 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 msgr2=0x7f9d380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 0x7f9d380a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c099500 tx=0x7f9d3c0bd040 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 msgr2=0x7f9d4c1a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00e970 tx=0x7f9d3c00ed70 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 shutdown_connections 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d380a3d20 0x7f9d380a61e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d4c19a0c0 0x7f9d4c1a4670 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c106170 0x7f9d4c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 --2- 172.21.15.87:0/2953915815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4c1010a0 0x7f9d4c1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 >> 172.21.15.87:0/2953915815 conn(0x7f9d4c0771d0 msgr2=0x7f9d4c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 shutdown_connections 2024-04-22T13:37:12.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.755+0000 7f9d51740640 1 -- 172.21.15.87:0/2953915815 wait complete. 2024-04-22T13:37:12.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:37:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.778+0000 7fe6596ac640 1 -- 172.21.15.87:0/3861198839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654102190 msgr2=0x7fe654102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.778+0000 7fe6596ac640 1 --2- 172.21.15.87:0/3861198839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654102190 0x7fe654102570 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe644009900 tx=0x7fe64402bc30 comp rx=0 tx=0).stop 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- 172.21.15.87:0/3861198839 shutdown_connections 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- 172.21.15.87:0/3861198839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe654107260 0x7fe65410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- 172.21.15.87:0/3861198839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102b40 0x7fe654106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- 172.21.15.87:0/3861198839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654102190 0x7fe654102570 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- 172.21.15.87:0/3861198839 >> 172.21.15.87:0/3861198839 conn(0x7fe6540fd990 msgr2=0x7fe6540ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- 172.21.15.87:0/3861198839 shutdown_connections 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- 172.21.15.87:0/3861198839 wait complete. 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 Processor -- start 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- start start 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102190 0x7fe65419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6541985e0 0x7fe654198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541121d0 con 0x7fe654107260 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe654112050 con 0x7fe654102190 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.779+0000 7fe6596ac640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe654112350 con 0x7fe6541985e0 2024-04-22T13:37:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48846/0 (socket says 172.21.15.87:48846) 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 -- 172.21.15.87:0/1673452282 learned_addr learned my addr 172.21.15.87:0/1673452282 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe652ffd640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102190 0x7fe65419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe6537fe640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6541985e0 0x7fe654198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 -- 172.21.15.87:0/1673452282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6541985e0 msgr2=0x7fe654198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6541985e0 0x7fe654198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 -- 172.21.15.87:0/1673452282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102190 msgr2=0x7fe65419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102190 0x7fe65419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 -- 172.21.15.87:0/1673452282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe654199100 con 0x7fe654107260 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe64bfff640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e9c0 tx=0x7fe63c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe650ff9640 1 -- 172.21.15.87:0/1673452282 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00cd80 con 0x7fe654107260 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6541993f0 con 0x7fe654107260 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe650ff9640 1 -- 172.21.15.87:0/1673452282 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63c0054d0 con 0x7fe654107260 2024-04-22T13:37:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe650ff9640 1 -- 172.21.15.87:0/1673452282 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c010430 con 0x7fe654107260 2024-04-22T13:37:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.780+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6541a5690 con 0x7fe654107260 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe650ff9640 1 -- 172.21.15.87:0/1673452282 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe63c004fd0 con 0x7fe654107260 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 Processor -- start 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 -- start start 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bb96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc96b1e500 con 0x55fc96bb9c30 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc96ac46a0 con 0x55fc96bb9310 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.783+0000 7fe65c378740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc96b1a4c0 con 0x55fc96bc2810 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe650ff9640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 0x7fe6400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48852/0 (socket says 172.21.15.87:48852) 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 -- 172.21.15.87:0/3122361554 learned_addr learned my addr 172.21.15.87:0/3122361554 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe6537fe640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 0x7fe6400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe6537fe640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bb96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 -- 172.21.15.87:0/3122361554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 msgr2=0x55fc96bc4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 -- 172.21.15.87:0/3122361554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 msgr2=0x55fc96bb96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bb96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 -- 172.21.15.87:0/3122361554 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc96ae35e0 con 0x55fc96bb9c30 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe650ff9640 1 -- 172.21.15.87:0/1673452282 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe63c0d0bf0 con 0x7fe654107260 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe6537fe640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 0x7fe6400a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009fd0 tx=0x7fe64c009d80 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe644009900 tx=0x7fe64402c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0238d607ba2b5cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.784+0000 7fe6497fa640 1 -- 172.21.15.87:0/3122361554 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe644045020 con 0x55fc96bb9c30 2024-04-22T13:37:12.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe6497fa640 1 -- 172.21.15.87:0/3122361554 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe64402c7e0 con 0x55fc96bb9c30 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe6497fa640 1 -- 172.21.15.87:0/3122361554 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6440046c0 con 0x55fc96bb9c30 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 -- 172.21.15.87:0/3122361554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 msgr2=0x55fc96bc22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe644009900 tx=0x7fe64402c610 comp rx=0 tx=0).stop 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 -- 172.21.15.87:0/3122361554 shutdown_connections 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc22d0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 --2- 172.21.15.87:0/3122361554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bb96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 -- 172.21.15.87:0/3122361554 >> 172.21.15.87:0/3122361554 conn(0x55fc96bb3a40 msgr2=0x55fc96bb3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 -- 172.21.15.87:0/3122361554 shutdown_connections 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.785+0000 7fe65c378740 1 -- 172.21.15.87:0/3122361554 wait complete. 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 -- start start 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 Processor -- start 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 0x55fc96bbf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bbfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc96bc55a0 con 0x55fc96bb9310 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc96ad7a60 con 0x55fc96bb9c30 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc96bc5720 con 0x55fc96bc2810 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe65c378740 10 client.0 _ll_register_callbacks cb 0x55fc96830110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48862/0 (socket says 172.21.15.87:48862) 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 -- 172.21.15.87:0/3871092754 learned_addr learned my addr 172.21.15.87:0/3871092754 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 0x55fc96bbf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 -- 172.21.15.87:0/3871092754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 msgr2=0x55fc96bbfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bbfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 -- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 msgr2=0x55fc96bbf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 0x55fc96bbf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe652ffd640 1 -- 172.21.15.87:0/3871092754 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc96830920 con 0x55fc96bb9310 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.786+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 0x55fc96bbf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:12.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe64402c170 tx=0x7fe644000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1033504ff0ee7965 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6440035f0 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe644003790 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe64402c7e0 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe644059050 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 10 client.17139 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc968fcd30 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe64402c7e0 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe644004dc0 con 0x55fc96bb9310 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 10 client.17139 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 10 client.17139 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 20 client.17139 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96bd2720 0x55fc96bd4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 4 client.17139 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.787+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc96af1a10 con 0x55fc96bd2720 2024-04-22T13:37:12.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.788+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96bd2720 0x55fc96bd4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.788+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96bd2720 0x55fc96bd4b10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2ea416ed76f913 server_cookie=4b6e474e3c5eda84 in_seq=0 out_seq=0 2024-04-22T13:37:12.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.788+0000 7fe63a7fc640 10 client.17139 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3871092754 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc96af1a10 con 0x55fc96bd2720 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe63a7fc640 10 client.17139 handle_command_reply: tid=0 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 client.17139 shutdown 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 msgr2=0x55fc96bc22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe64402c170 tx=0x7fe644000bc0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 shutdown_connections 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96bd2720 0x55fc96bd4b10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bbfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9c30 0x55fc96bbf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe63a7fc640 0 client.17139 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 --2- 172.21.15.87:0/3871092754 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9310 0x55fc96bc22a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 >> 172.21.15.87:0/3871092754 conn(0x55fc96bb3a40 msgr2=0x55fc96bb5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 shutdown_connections 2024-04-22T13:37:12.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 1 -- 172.21.15.87:0/3871092754 wait complete. 2024-04-22T13:37:12.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.789+0000 7fe65c378740 20 client.17139 trim_cache size 0 max 16384 2024-04-22T13:37:12.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 Processor -- start 2024-04-22T13:37:12.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 -- start start 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 0x55fc96bb2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bc2810 0x55fc96bc22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541ae0b0 con 0x55fc96bb9c30 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541ab980 con 0x55fc96bc2810 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe65c378740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541a9870 con 0x55fc96bb9310 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.807+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48866/0 (socket says 172.21.15.87:48866) 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 -- 172.21.15.87:0/2885769738 learned_addr learned my addr 172.21.15.87:0/2885769738 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe652ffd640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 0x55fc96bb2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe6537fe640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bc2810 0x55fc96bc22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 -- 172.21.15.87:0/2885769738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 msgr2=0x55fc96bb2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 0x55fc96bb2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 -- 172.21.15.87:0/2885769738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bc2810 msgr2=0x55fc96bc22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bc2810 0x55fc96bc22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 -- 172.21.15.87:0/2885769738 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc96bb1fb0 con 0x55fc96bb9c30 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe652ffd640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 0x55fc96bb2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe64bfff640 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe63c09c850 tx=0x7fe63c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eca62bf1708fb81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe648ff9640 1 -- 172.21.15.87:0/2885769738 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c09d2a0 con 0x55fc96bb9c30 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe648ff9640 1 -- 172.21.15.87:0/2885769738 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63c00a250 con 0x55fc96bb9c30 2024-04-22T13:37:12.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.808+0000 7fe648ff9640 1 -- 172.21.15.87:0/2885769738 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c099210 con 0x55fc96bb9c30 2024-04-22T13:37:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- 172.21.15.87:0/2885769738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 msgr2=0x55fc96bc1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe63c09c850 tx=0x7fe63c007850 comp rx=0 tx=0).stop 2024-04-22T13:37:12.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- 172.21.15.87:0/2885769738 shutdown_connections 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bc2810 0x55fc96bc22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc1d60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 --2- 172.21.15.87:0/2885769738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bb9310 0x55fc96bb2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- 172.21.15.87:0/2885769738 >> 172.21.15.87:0/2885769738 conn(0x55fc96bb3a40 msgr2=0x55fc96bb2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- 172.21.15.87:0/2885769738 shutdown_connections 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- 172.21.15.87:0/2885769738 wait complete. 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 -- start start 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.809+0000 7fe65c378740 1 Processor -- start 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541a9ec0 con 0x55fc96bb9c30 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541ab2b0 con 0x55fc96bb9310 2024-04-22T13:37:12.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541aa510 con 0x55fc96bc2810 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe65c378740 10 client.0 _ll_register_callbacks cb 0x7fe64c0126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe64bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56126/0 (socket says 172.21.15.87:56126) 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 -- 172.21.15.87:0/3571604720 learned_addr learned my addr 172.21.15.87:0/3571604720 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe6537fe640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 -- 172.21.15.87:0/3571604720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 msgr2=0x55fc96bc7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 -- 172.21.15.87:0/3571604720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 msgr2=0x55fc96bc70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.810+0000 7fe652ffd640 1 -- 172.21.15.87:0/3571604720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc96bba4a0 con 0x55fc96bb9310 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe652ffd640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe64402c170 tx=0x7fe644005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b978de9c76f8aa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe644045040 con 0x55fc96bb9310 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe63c00a250 con 0x55fc96bb9310 2024-04-22T13:37:12.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fe6440419f0 con 0x55fc96bb9310 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe64402c7e0 con 0x55fc96bb9310 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 10 client.26258 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc96c39850 con 0x55fc96bb9310 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63c00a250 con 0x55fc96bb9310 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6440035f0 con 0x55fc96bb9310 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 10 client.26258 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 10 client.26258 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 20 client.26258 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96c3a000 0x55fc96c3c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 4 client.26258 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6bb0429", "release": true, "await": true}] 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.811+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6bb0429", "release": true, "await": true}) -- 0x55fc96b2bf20 con 0x55fc96c3a000 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.812+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96c3a000 0x55fc96c3c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.812+0000 7fe63a7fc640 10 client.26258 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.812+0000 7fe64bfff640 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96c3a000 0x55fc96c3c420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffca6e8d071a35eb server_cookie=bb9325cd04e3a1da in_seq=0 out_seq=0 2024-04-22T13:37:12.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.821+0000 7fe63a7fc640 1 -- 172.21.15.87:0/3571604720 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55fc96b2bf20 con 0x55fc96c3a000 2024-04-22T13:37:12.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.821+0000 7fe63a7fc640 10 client.26258 handle_command_reply: tid=0 2024-04-22T13:37:12.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.821+0000 7fe65c378740 1 client.26258 shutdown 2024-04-22T13:37:12.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 msgr2=0x55fc96bcad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe64402c170 tx=0x7fe644005bc0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 shutdown_connections 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc96c3a000 0x55fc96c3c420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc96bc2810 0x55fc96bc7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc96bb9c30 0x55fc96bc70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 --2- 172.21.15.87:0/3571604720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc96bb9310 0x55fc96bcad40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 >> 172.21.15.87:0/3571604720 conn(0x55fc96bb3a40 msgr2=0x55fc96bb2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe63a7fc640 0 client.26258 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 shutdown_connections 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.822+0000 7fe65c378740 1 -- 172.21.15.87:0/3571604720 wait complete. 2024-04-22T13:37:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.823+0000 7fe65c378740 20 client.26258 trim_cache size 0 max 16384 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 24, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "6bb0429": { 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 24, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 90.0, 2024-04-22T13:37:12.820 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:12.821 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.824+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 msgr2=0x7fe6400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.824+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 0x7fe6400a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009fd0 tx=0x7fe64c009d80 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.824+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 msgr2=0x7fe65419e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.824+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e9c0 tx=0x7fe63c00ee80 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 shutdown_connections 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6400a3d20 0x7fe6400a61e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6541985e0 0x7fe654198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe654107260 0x7fe65419e970 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 --2- 172.21.15.87:0/1673452282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe654102190 0x7fe65419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 >> 172.21.15.87:0/1673452282 conn(0x7fe6540fd990 msgr2=0x7fe6540fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 shutdown_connections 2024-04-22T13:37:12.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.825+0000 7fe6596ac640 1 -- 172.21.15.87:0/1673452282 wait complete. 2024-04-22T13:37:12.840 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 6bb0429, seconds elapsed: 33.5 2024-04-22T13:37:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 -- 172.21.15.87:0/2520334906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff330101900 msgr2=0x7ff330105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 --2- 172.21.15.87:0/2520334906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff330101900 0x7ff330105940 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff320009860 tx=0x7ff32002bb70 comp rx=0 tx=0).stop 2024-04-22T13:37:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 -- 172.21.15.87:0/2520334906 shutdown_connections 2024-04-22T13:37:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 --2- 172.21.15.87:0/2520334906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 --2- 172.21.15.87:0/2520334906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff330101900 0x7ff330105940 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 --2- 172.21.15.87:0/2520334906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 0x7ff330101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.909+0000 7ff336384640 1 -- 172.21.15.87:0/2520334906 >> 172.21.15.87:0/2520334906 conn(0x7ff3300fc7d0 msgr2=0x7ff3300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.910+0000 7ff336384640 1 -- 172.21.15.87:0/2520334906 shutdown_connections 2024-04-22T13:37:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.910+0000 7ff336384640 1 -- 172.21.15.87:0/2520334906 wait complete. 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.910+0000 7ff336384640 1 Processor -- start 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 -- start start 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 0x7ff330109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010fe40 con 0x7ff33010aa20 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010fcc0 con 0x7ff330106020 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff336384640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010ffc0 con 0x7ff330100f50 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff335b83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff335b83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48888/0 (socket says 172.21.15.87:48888) 2024-04-22T13:37:12.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.911+0000 7ff335b83640 1 -- 172.21.15.87:0/2526102504 learned_addr learned my addr 172.21.15.87:0/2526102504 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff334b81640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335382640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 0x7ff330109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 -- 172.21.15.87:0/2526102504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 msgr2=0x7ff330109fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 0x7ff330109fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 -- 172.21.15.87:0/2526102504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 msgr2=0x7ff33010a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 -- 172.21.15.87:0/2526102504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301ab3c0 con 0x7ff33010aa20 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff334b81640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.912+0000 7ff335b83640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff32800efc0 tx=0x7ff32800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.913+0000 7ff3267fc640 1 -- 172.21.15.87:0/2526102504 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328003970 con 0x7ff33010aa20 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.913+0000 7ff3267fc640 1 -- 172.21.15.87:0/2526102504 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff328004020 con 0x7ff33010aa20 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.913+0000 7ff3267fc640 1 -- 172.21.15.87:0/2526102504 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32800b940 con 0x7ff33010aa20 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.913+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301ab6b0 con 0x7ff33010aa20 2024-04-22T13:37:12.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.913+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301abb80 con 0x7ff33010aa20 2024-04-22T13:37:12.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff3267fc640 1 -- 172.21.15.87:0/2526102504 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff32801d020 con 0x7ff33010aa20 2024-04-22T13:37:12.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff33a1d1740 1 Processor -- start 2024-04-22T13:37:12.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff33a1d1740 1 -- start start 2024-04-22T13:37:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a66630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.918+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a69050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60572/0 (socket says 172.21.15.87:60572) 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 -- 172.21.15.87:0/1553365506 learned_addr learned my addr 172.21.15.87:0/1553365506 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a9409c21f0 con 0x55a940a66c00 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a94095f4b0 con 0x55a940a5e080 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a9409be1b0 con 0x55a940a5d6d0 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff3267fc640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 0x7ff31c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a69050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff335382640 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a66630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 -- 172.21.15.87:0/1553365506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 msgr2=0x55a940a66630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a66630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 -- 172.21.15.87:0/1553365506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 msgr2=0x55a940a69050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a69050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff334b81640 1 -- 172.21.15.87:0/1553365506 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9409511b0 con 0x55a940a5d6d0 2024-04-22T13:37:12.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.919+0000 7ff3267fc640 1 -- 172.21.15.87:0/2526102504 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff3280d0510 con 0x7ff33010aa20 2024-04-22T13:37:12.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.920+0000 7ff335b83640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 0x7ff31c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.920+0000 7ff334b81640 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff320005e00 tx=0x7ff320005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cde7971d07628df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.920+0000 7ff31b7fe640 1 -- 172.21.15.87:0/1553365506 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320039040 con 0x55a940a5d6d0 2024-04-22T13:37:12.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.920+0000 7ff31b7fe640 1 -- 172.21.15.87:0/1553365506 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff320034e40 con 0x55a940a5d6d0 2024-04-22T13:37:12.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.920+0000 7ff31b7fe640 1 -- 172.21.15.87:0/1553365506 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32002cb10 con 0x55a940a5d6d0 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.921+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 msgr2=0x55a940a5dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.921+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff320005e00 tx=0x7ff320005f90 comp rx=0 tx=0).stop 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff335b83640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 0x7ff31c0a63b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff32800cd00 tx=0x7ff32800ceb0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 shutdown_connections 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a69050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a66630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1553365506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a5dab0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 >> 172.21.15.87:0/1553365506 conn(0x55a940a57da0 msgr2=0x55a940a58180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.922+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 shutdown_connections 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.923+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1553365506 wait complete. 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.923+0000 7ff33a1d1740 1 -- start start 2024-04-22T13:37:12.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.923+0000 7ff33a1d1740 1 Processor -- start 2024-04-22T13:37:12.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.924+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a63950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a64090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a940a69b10 con 0x55a940a66c00 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60586/0 (socket says 172.21.15.87:60586) 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 -- 172.21.15.87:0/721817099 learned_addr learned my addr 172.21.15.87:0/721817099 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a94097b500 con 0x55a940a5e080 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a940a69c90 con 0x55a940a5d6d0 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff335382640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a63950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff33a1d1740 10 client.0 _ll_register_callbacks cb 0x55a940949b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a64090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 -- 172.21.15.87:0/721817099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 msgr2=0x55a940a63950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a63950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 -- 172.21.15.87:0/721817099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 msgr2=0x55a940a64090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a64090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.925+0000 7ff334b81640 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9406d43f0 con 0x55a940a5d6d0 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.926+0000 7ff334b81640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff320009780 tx=0x7ff32002bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6278abae36bd578c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.926+0000 7ff318ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.926+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320039070 con 0x55a940a5d6d0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.926+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff32002ca20 con 0x55a940a5d6d0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.926+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a940961930 con 0x55a940a5d6d0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.927+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff320005040 con 0x55a940a5d6d0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.927+0000 7ff33a1d1740 10 client.26323 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.927+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a94079e7d0 con 0x55a940a5d6d0 2024-04-22T13:37:12.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.927+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a940961930 con 0x55a940a5d6d0 2024-04-22T13:37:12.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.928+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff320004c30 con 0x55a940a5d6d0 2024-04-22T13:37:12.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.928+0000 7ff33a1d1740 10 client.26323 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.928+0000 7ff33a1d1740 10 client.26323 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:37:12.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.929+0000 7ff33a1d1740 20 client.26323 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.929+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940a76a90 0x55a940a78e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.929+0000 7ff33a1d1740 4 client.26323 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:12.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.929+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9409cfc10 con 0x55a940a76a90 2024-04-22T13:37:12.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.929+0000 7ff335382640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940a76a90 0x55a940a78e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.930+0000 7ff335382640 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940a76a90 0x55a940a78e80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c727a450da8e0d8a server_cookie=cb8e8d02c9606b82 in_seq=0 out_seq=0 2024-04-22T13:37:12.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.930+0000 7ff318ff9640 10 client.26323 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:12.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.930+0000 7ff318ff9640 1 -- 172.21.15.87:0/721817099 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9409cfc10 con 0x55a940a76a90 2024-04-22T13:37:12.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.930+0000 7ff318ff9640 10 client.26323 handle_command_reply: tid=0 2024-04-22T13:37:12.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.930+0000 7ff33a1d1740 1 client.26323 shutdown 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 msgr2=0x55a940a665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff320009780 tx=0x7ff32002bb70 comp rx=0 tx=0).stop 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 shutdown_connections 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940a76a90 0x55a940a78e80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a63950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/721817099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a665c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 >> 172.21.15.87:0/721817099 conn(0x55a940a57da0 msgr2=0x55a940a59af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff318ff9640 0 client.26323 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:12.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 shutdown_connections 2024-04-22T13:37:12.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.931+0000 7ff33a1d1740 1 -- 172.21.15.87:0/721817099 wait complete. 2024-04-22T13:37:12.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.932+0000 7ff33a1d1740 20 client.26323 trim_cache size 0 max 16384 2024-04-22T13:37:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.938+0000 7f577b15f640 1 -- 172.21.15.87:0/15181217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 msgr2=0x7f577410e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.938+0000 7f577b15f640 1 --2- 172.21.15.87:0/15181217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577410e880 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f576802bcd0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.938+0000 7f577b15f640 1 -- 172.21.15.87:0/15181217 shutdown_connections 2024-04-22T13:37:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.938+0000 7f577b15f640 1 --2- 172.21.15.87:0/15181217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 0x7f5774115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.939+0000 7f577b15f640 1 --2- 172.21.15.87:0/15181217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f5774108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.939+0000 7f577b15f640 1 --2- 172.21.15.87:0/15181217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577410e880 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.939+0000 7f577b15f640 1 -- 172.21.15.87:0/15181217 >> 172.21.15.87:0/15181217 conn(0x7f5774077df0 msgr2=0x7f5774106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.939+0000 7f577b15f640 1 -- 172.21.15.87:0/15181217 shutdown_connections 2024-04-22T13:37:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.939+0000 7f577b15f640 1 -- 172.21.15.87:0/15181217 wait complete. 2024-04-22T13:37:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.940+0000 7f577b15f640 1 Processor -- start 2024-04-22T13:37:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.940+0000 7f577b15f640 1 -- start start 2024-04-22T13:37:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 0x7f57741a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5774118690 con 0x7f577410e4a0 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5774118510 con 0x7f5774108670 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f577b15f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5774118810 con 0x7f5774109010 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56188/0 (socket says 172.21.15.87:56188) 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 -- 172.21.15.87:0/1732041114 learned_addr learned my addr 172.21.15.87:0/1732041114 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5773fff640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 0x7f57741a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f57796fc640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 -- 172.21.15.87:0/1732041114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 msgr2=0x7f57741a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 0x7f57741a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 -- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 msgr2=0x7f577419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f5778efb640 1 -- 172.21.15.87:0/1732041114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577419f220 con 0x7f5774108670 2024-04-22T13:37:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.941+0000 7f57796fc640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.942+0000 7f5778efb640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5768000c00 tx=0x7f576802d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.942+0000 7f5771ffb640 1 -- 172.21.15.87:0/1732041114 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768043020 con 0x7f5774108670 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.942+0000 7f5771ffb640 1 -- 172.21.15.87:0/1732041114 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f576802dc10 con 0x7f5774108670 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.942+0000 7f5771ffb640 1 -- 172.21.15.87:0/1732041114 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57680039e0 con 0x7f5774108670 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.942+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577419f4b0 con 0x7f5774108670 2024-04-22T13:37:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.943+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57741ab7b0 con 0x7f5774108670 2024-04-22T13:37:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.947+0000 7f5771ffb640 1 -- 172.21.15.87:0/1732041114 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5768046020 con 0x7f5774108670 2024-04-22T13:37:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 Processor -- start 2024-04-22T13:37:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 -- start start 2024-04-22T13:37:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f5771ffb640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 0x7f57640a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536baac0 0x564a536b9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536c32d0 0x564a536c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564a5361e8d0 con 0x564a536b9d50 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a535bbe80 con 0x564a536c32d0 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f577de2a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564a5361a890 con 0x564a536baac0 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.948+0000 7f5771ffb640 1 -- 172.21.15.87:0/1732041114 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f57680f27a0 con 0x7f5774108670 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5778efb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536baac0 0x564a536b9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48928/0 (socket says 172.21.15.87:48928) 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 -- 172.21.15.87:0/3775998474 learned_addr learned my addr 172.21.15.87:0/3775998474 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f57796fc640 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536c32d0 0x564a536c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 -- 172.21.15.87:0/3775998474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536baac0 msgr2=0x564a536b9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536baac0 0x564a536b9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 -- 172.21.15.87:0/3775998474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536c32d0 msgr2=0x564a536c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536c32d0 0x564a536c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 -- 172.21.15.87:0/3775998474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a535dd320 con 0x564a536b9d50 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.949+0000 7f5773fff640 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f576000bc90 tx=0x7f57600107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0ce2f133dea084 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.950+0000 7f575effd640 1 -- 172.21.15.87:0/3775998474 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760004020 con 0x564a536b9d50 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.950+0000 7f575effd640 1 -- 172.21.15.87:0/3775998474 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57600041c0 con 0x564a536b9d50 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.950+0000 7f575effd640 1 -- 172.21.15.87:0/3775998474 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760004020 con 0x564a536b9d50 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f57796fc640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 0x7f57640a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 -- 172.21.15.87:0/3775998474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 msgr2=0x564a536ba1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f576000bc90 tx=0x7f57600107a0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 -- 172.21.15.87:0/3775998474 shutdown_connections 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536c32d0 0x564a536c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536b9d50 0x564a536ba1d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 --2- 172.21.15.87:0/3775998474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536baac0 0x564a536b9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.951+0000 7f577de2a740 1 -- 172.21.15.87:0/3775998474 >> 172.21.15.87:0/3775998474 conn(0x564a53680c90 msgr2=0x564a536b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.952+0000 7f57796fc640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 0x7f57640a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x564a536b5bd0 tx=0x7f576c004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.952+0000 7f577de2a740 1 -- 172.21.15.87:0/3775998474 shutdown_connections 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.953+0000 7f577de2a740 1 -- 172.21.15.87:0/3775998474 wait complete. 2024-04-22T13:37:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.953+0000 7f577de2a740 1 -- start start 2024-04-22T13:37:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.953+0000 7f577de2a740 1 Processor -- start 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536c32d0 0x564a536c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f5773fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f5773fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56200/0 (socket says 172.21.15.87:56200) 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f5778efb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f5778efb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60614/0 (socket says 172.21.15.87:60614) 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564a536c6050 con 0x564a536c32d0 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a535a6eb0 con 0x564a536baac0 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564a536c61d0 con 0x564a536b9d50 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f577de2a740 10 client.0 _ll_register_callbacks cb 0x564a535d3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.954+0000 7f5773fff640 1 -- 172.21.15.87:0/2717611655 learned_addr learned my addr 172.21.15.87:0/2717611655 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f57796fc640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536c32d0 0x564a536c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 -- 172.21.15.87:0/2717611655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 msgr2=0x564a536c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 -- 172.21.15.87:0/2717611655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536c32d0 msgr2=0x564a536c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536c32d0 0x564a536c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 -- 172.21.15.87:0/2717611655 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a53331210 con 0x564a536baac0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5778efb640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5773fff640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5760004180 tx=0x7f576000bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4cd3efb4aa77a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5753fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576002a060 con 0x564a536baac0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.955+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57600041c0 con 0x564a536baac0 2024-04-22T13:37:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.956+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f5760003a00 con 0x564a536baac0 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.956+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5760005040 con 0x564a536baac0 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.956+0000 7f577de2a740 10 client.26279 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.956+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a533fd620 con 0x564a536baac0 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57600041c0 con 0x564a536baac0 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f576002d020 con 0x564a536baac0 2024-04-22T13:37:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 10 client.26279 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 10 client.26279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 20 client.26279 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a536d31c0 0x564a536d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 4 client.26279 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.957+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a53632880 con 0x564a536d31c0 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.958+0000 7f5778efb640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a536d31c0 0x564a536d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.958+0000 7f5778efb640 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a536d31c0 0x564a536d55b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca3bd77d14bfb60 server_cookie=d346002be455f9ec in_seq=0 out_seq=0 2024-04-22T13:37:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.958+0000 7f5753fff640 10 client.26279 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.959+0000 7f5753fff640 1 -- 172.21.15.87:0/2717611655 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a53632880 con 0x564a536d31c0 2024-04-22T13:37:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.959+0000 7f5753fff640 10 client.26279 handle_command_reply: tid=0 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 client.26279 shutdown 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 msgr2=0x564a536bfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5760004180 tx=0x7f576000bd80 comp rx=0 tx=0).stop 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 shutdown_connections 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a536d31c0 0x564a536d55b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536c32d0 0x564a536c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536baac0 0x564a536bfed0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 --2- 172.21.15.87:0/2717611655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536b9d50 0x564a536c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 >> 172.21.15.87:0/2717611655 conn(0x564a53680c90 msgr2=0x564a536b6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f5753fff640 0 client.26279 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 shutdown_connections 2024-04-22T13:37:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 1 -- 172.21.15.87:0/2717611655 wait complete. 2024-04-22T13:37:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.960+0000 7f577de2a740 20 client.26279 trim_cache size 0 max 16384 2024-04-22T13:37:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 Processor -- start 2024-04-22T13:37:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 -- start start 2024-04-22T13:37:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a5e080 0x55a940a725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a66c00 0x55a940a7bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b6650 con 0x55a940a5e080 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b3f20 con 0x55a940a5d6d0 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.963+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b1df0 con 0x55a940a66c00 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a66c00 0x55a940a7bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56212/0 (socket says 172.21.15.87:56212) 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 -- 172.21.15.87:0/3984293635 learned_addr learned my addr 172.21.15.87:0/3984293635 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff335382640 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a5e080 0x55a940a725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 -- 172.21.15.87:0/3984293635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a66c00 msgr2=0x55a940a7bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a66c00 0x55a940a7bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 -- 172.21.15.87:0/3984293635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a5e080 msgr2=0x55a940a725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a5e080 0x55a940a725f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.964+0000 7ff334b81640 1 -- 172.21.15.87:0/3984293635 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a940a53480 con 0x55a940a5d6d0 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.965+0000 7ff334b81640 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff320003710 tx=0x7ff320000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0550a2fbcf1450 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.965+0000 7ff31affd640 1 -- 172.21.15.87:0/3984293635 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320039020 con 0x55a940a5d6d0 2024-04-22T13:37:12.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.965+0000 7ff31affd640 1 -- 172.21.15.87:0/3984293635 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff32002ca20 con 0x55a940a5d6d0 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 -- 172.21.15.87:0/3984293635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 msgr2=0x55a940a62ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff320003710 tx=0x7ff320000bc0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 -- 172.21.15.87:0/3984293635 shutdown_connections 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a66c00 0x55a940a7bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a5e080 0x55a940a725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/3984293635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5d6d0 0x55a940a62ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.966+0000 7ff33a1d1740 1 -- 172.21.15.87:0/3984293635 >> 172.21.15.87:0/3984293635 conn(0x55a940a57da0 msgr2=0x55a940a71ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.967+0000 7ff33a1d1740 1 -- 172.21.15.87:0/3984293635 shutdown_connections 2024-04-22T13:37:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.967+0000 7ff33a1d1740 1 -- 172.21.15.87:0/3984293635 wait complete. 2024-04-22T13:37:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.967+0000 7ff33a1d1740 1 -- start start 2024-04-22T13:37:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.967+0000 7ff33a1d1740 1 Processor -- start 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a78620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a78b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a790a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b2440 con 0x55a940a66c00 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b3850 con 0x55a940a5e080 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301b2ab0 con 0x55a940a5d6d0 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff33a1d1740 10 client.0 _ll_register_callbacks cb 0x7ff320034ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff334b81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a78620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff334b81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff334b81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a790a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48940/0 (socket says 172.21.15.87:48940) 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff334b81640 1 -- 172.21.15.87:0/1512064627 learned_addr learned my addr 172.21.15.87:0/1512064627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.968+0000 7ff335382640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a78b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.969+0000 7ff335382640 1 -- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 msgr2=0x55a940a78620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.969+0000 7ff335382640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a78620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.969+0000 7ff335382640 1 -- 172.21.15.87:0/1512064627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 msgr2=0x55a940a790a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.969+0000 7ff335382640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a790a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.969+0000 7ff335382640 1 -- 172.21.15.87:0/1512064627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a940adad30 con 0x55a940a5e080 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff334b81640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a78620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff335382640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a78b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55a940a69c90 tx=0x7ff32c00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788587fdab320d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff318ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c01e420 con 0x55a940a5e080 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff32002ca20 con 0x55a940a5e080 2024-04-22T13:37:12.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.970+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7ff32c01e5c0 con 0x55a940a5e080 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff32c02e050 con 0x55a940a5e080 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 10 client.26285 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a940adc090 con 0x55a940a5e080 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff32002ca20 con 0x55a940a5e080 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff32c011440 con 0x55a940a5e080 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 10 client.26285 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 10 client.26285 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 20 client.26285 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940adc7f0 0x55a940adec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 4 client.26285 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.971+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9409c9a80 con 0x55a940adc7f0 2024-04-22T13:37:12.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.972+0000 7ff334b81640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940adc7f0 0x55a940adec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.972+0000 7ff334b81640 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940adc7f0 0x55a940adec10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab9c81fcb1c485b5 server_cookie=8cc217cc024aa0e5 in_seq=0 out_seq=0 2024-04-22T13:37:12.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.972+0000 7ff318ff9640 10 client.26285 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.972+0000 7ff318ff9640 1 -- 172.21.15.87:0/1512064627 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9409c9a80 con 0x55a940adc7f0 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.972+0000 7ff318ff9640 10 client.26285 handle_command_reply: tid=0 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 client.26285 shutdown 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 msgr2=0x55a940a78b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a78b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55a940a69c90 tx=0x7ff32c00be50 comp rx=0 tx=0).stop 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 shutdown_connections 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55a940adc7f0 0x55a940adec10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a940a66c00 0x55a940a790a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a940a5e080 0x55a940a78b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 --2- 172.21.15.87:0/1512064627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a940a5d6d0 0x55a940a78620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 >> 172.21.15.87:0/1512064627 conn(0x55a940a57da0 msgr2=0x55a940a72770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.973+0000 7ff318ff9640 0 client.26285 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.974+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 shutdown_connections 2024-04-22T13:37:12.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.974+0000 7ff33a1d1740 1 -- 172.21.15.87:0/1512064627 wait complete. 2024-04-22T13:37:12.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.974+0000 7ff33a1d1740 20 client.26285 trim_cache size 0 max 16384 2024-04-22T13:37:12.971 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 msgr2=0x7ff31c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 0x7ff31c0a63b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff32800cd00 tx=0x7ff32800ceb0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 msgr2=0x7ff3301aad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff32800efc0 tx=0x7ff32800c800 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 shutdown_connections 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff31c0a3ef0 0x7ff31c0a63b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff33010aa20 0x7ff3301aad80 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff330106020 0x7ff33010a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 --2- 172.21.15.87:0/2526102504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100f50 0x7ff330109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 >> 172.21.15.87:0/2526102504 conn(0x7ff3300fc7d0 msgr2=0x7ff3300fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 shutdown_connections 2024-04-22T13:37:12.972 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:12.975+0000 7ff336384640 1 -- 172.21.15.87:0/2526102504 wait complete. 2024-04-22T13:37:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.985+0000 7f577de2a740 1 Processor -- start 2024-04-22T13:37:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.985+0000 7f577de2a740 1 -- start start 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536c2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b6730 con 0x564a536baac0 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b4000 con 0x564a536b9d50 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f577de2a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b1e70 con 0x564a536c32d0 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f57796fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56228/0 (socket says 172.21.15.87:56228) 2024-04-22T13:37:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f5778efb640 1 -- 172.21.15.87:0/1955147540 learned_addr learned my addr 172.21.15.87:0/1955147540 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.986+0000 7f5773fff640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 -- 172.21.15.87:0/1955147540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 msgr2=0x564a536c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 -- 172.21.15.87:0/1955147540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 msgr2=0x564a536af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 -- 172.21.15.87:0/1955147540 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a536b9020 con 0x564a536baac0 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f57796fc640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5778efb640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.987+0000 7f5773fff640 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536c2600 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x564a536c14a0 tx=0x7f57600107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48632deaad2b9622 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.988+0000 7f575e7fc640 1 -- 172.21.15.87:0/1955147540 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576002a040 con 0x564a536baac0 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.988+0000 7f575e7fc640 1 -- 172.21.15.87:0/1955147540 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5760004e90 con 0x564a536baac0 2024-04-22T13:37:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.988+0000 7f575e7fc640 1 -- 172.21.15.87:0/1955147540 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760004660 con 0x564a536baac0 2024-04-22T13:37:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.990+0000 7f577de2a740 1 -- 172.21.15.87:0/1955147540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 msgr2=0x564a536c2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.990+0000 7f577de2a740 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536c2600 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x564a536c14a0 tx=0x7f57600107a0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.992+0000 7f577de2a740 1 -- 172.21.15.87:0/1955147540 shutdown_connections 2024-04-22T13:37:12.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:12 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2759979648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:12.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:12 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1472373261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.992+0000 7f577de2a740 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.992+0000 7f577de2a740 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536c2600 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.992+0000 7f577de2a740 1 --2- 172.21.15.87:0/1955147540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.992+0000 7f577de2a740 1 -- 172.21.15.87:0/1955147540 >> 172.21.15.87:0/1955147540 conn(0x564a53680c90 msgr2=0x564a536bfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.993+0000 7f577de2a740 1 -- 172.21.15.87:0/1955147540 shutdown_connections 2024-04-22T13:37:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.993+0000 7f577de2a740 1 -- 172.21.15.87:0/1955147540 wait complete. 2024-04-22T13:37:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.993+0000 7f577de2a740 1 -- start start 2024-04-22T13:37:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.994+0000 7f577de2a740 1 Processor -- start 2024-04-22T13:37:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f577de2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f577de2a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f5778efb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56242/0 (socket says 172.21.15.87:56242) 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f5773fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f5778efb640 1 -- 172.21.15.87:0/166768061 learned_addr learned my addr 172.21.15.87:0/166768061 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.995+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536bd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b24c0 con 0x564a536baac0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f57796fc640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536bd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b3930 con 0x564a536b9d50 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57741b2b90 con 0x564a536c32d0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f577de2a740 10 client.0 _ll_register_callbacks cb 0x7f576000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f5778efb640 1 -- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 msgr2=0x564a536bd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f5778efb640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536bd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f5778efb640 1 -- 172.21.15.87:0/166768061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 msgr2=0x564a536bd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f5778efb640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536bd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f5778efb640 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a536bc140 con 0x564a536b9d50 2024-04-22T13:37:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.996+0000 7f57796fc640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536bd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:12.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:37:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f5778efb640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5768001280 tx=0x7f576802bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97d8c665d26db2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f5753fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57680bf080 con 0x564a536b9d50 2024-04-22T13:37:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f5768000ed0 con 0x564a536b9d50 2024-04-22T13:37:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5760004e90 con 0x564a536b9d50 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f57680be980 con 0x564a536b9d50 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f577de2a740 10 client.26288 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.997+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a537385d0 con 0x564a536b9d50 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.998+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57680d7070 con 0x564a536b9d50 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f57680dd030 con 0x564a536b9d50 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 10 client.26288 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 10 client.26288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 20 client.26288 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a53738b60 0x564a5373af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 4 client.26288 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5760004e90 con 0x564a53738b60 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f5773fff640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a53738b60 0x564a5373af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f5773fff640 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a53738b60 0x564a5373af50 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3fbd829866d1204 server_cookie=4121c10d198cbe2a in_seq=0 out_seq=0 2024-04-22T13:37:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:12.999+0000 7f5753fff640 10 client.26288 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.009+0000 7f5753fff640 1 -- 172.21.15.87:0/166768061 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5760004e90 con 0x564a53738b60 2024-04-22T13:37:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.009+0000 7f5753fff640 10 client.26288 handle_command_reply: tid=0 2024-04-22T13:37:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.009+0000 7f577de2a740 1 client.26288 shutdown 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.009+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 msgr2=0x564a536d86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.009+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5768001280 tx=0x7f576802bcd0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 shutdown_connections 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a53738b60 0x564a5373af50 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a536c32d0 0x564a536bd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f5753fff640 0 client.26288 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a536baac0 0x564a536bd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 --2- 172.21.15.87:0/166768061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a536b9d50 0x564a536d86f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 >> 172.21.15.87:0/166768061 conn(0x564a53680c90 msgr2=0x564a536be3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 shutdown_connections 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 1 -- 172.21.15.87:0/166768061 wait complete. 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.010+0000 7f577de2a740 20 client.26288 trim_cache size 0 max 16384 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:37:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:37:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 msgr2=0x7f57640a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 0x7f57640a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x564a536b5bd0 tx=0x7f576c004ab0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 msgr2=0x7f57741a48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5768000c00 tx=0x7f576802d890 comp rx=0 tx=0).stop 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 shutdown_connections 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f57640a3f20 0x7f57640a63e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f577410e4a0 0x7f577419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5774109010 0x7f57741a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 --2- 172.21.15.87:0/1732041114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5774108670 0x7f57741a48f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 >> 172.21.15.87:0/1732041114 conn(0x7f5774077df0 msgr2=0x7f5774078fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 shutdown_connections 2024-04-22T13:37:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:13.012+0000 7f577b15f640 1 -- 172.21.15.87:0/1732041114 wait complete. 2024-04-22T13:37:13.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.168+0000 7f02a5150640 1 -- 172.21.15.87:0/3244106251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a0108f50 msgr2=0x7f02a01093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.168+0000 7f02a5150640 1 --2- 172.21.15.87:0/3244106251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a0108f50 0x7f02a01093b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0290009900 tx=0x7f029002bd10 comp rx=0 tx=0).stop 2024-04-22T13:37:13.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 -- 172.21.15.87:0/3244106251 shutdown_connections 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 --2- 172.21.15.87:0/3244106251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a01099d0 0x7f02a011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 --2- 172.21.15.87:0/3244106251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a0108f50 0x7f02a01093b0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 --2- 172.21.15.87:0/3244106251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a01105f0 0x7f02a01109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 -- 172.21.15.87:0/3244106251 >> 172.21.15.87:0/3244106251 conn(0x7f02a006ced0 msgr2=0x7f02a006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.170+0000 7f02a5150640 1 -- 172.21.15.87:0/3244106251 shutdown_connections 2024-04-22T13:37:13.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.171+0000 7f02a5150640 1 -- 172.21.15.87:0/3244106251 wait complete. 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.171+0000 7f02a5150640 1 Processor -- start 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.171+0000 7f02a5150640 1 -- start start 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 0x7f02a01a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a01105f0 0x7f02a01abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0120f20 con 0x7f02a01099d0 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0120da0 con 0x7f02a01105f0 2024-04-22T13:37:13.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f02a5150640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01210a0 con 0x7f02a0108f50 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.172+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48982/0 (socket says 172.21.15.87:48982) 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 -- 172.21.15.87:0/4293582266 learned_addr learned my addr 172.21.15.87:0/4293582266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 -- 172.21.15.87:0/4293582266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 msgr2=0x7f02a01a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029ffff640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 0x7f02a01a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 0x7f02a01a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 -- 172.21.15.87:0/4293582266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a01105f0 msgr2=0x7f02a01abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a01105f0 0x7f02a01abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029f7fe640 1 -- 172.21.15.87:0/4293582266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a01ac490 con 0x7f02a01099d0 2024-04-22T13:37:13.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.173+0000 7f029ffff640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 0x7f02a01a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:13.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:12 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2759979648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:13.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:12 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1472373261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.174+0000 7f029f7fe640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0290005ab0 tx=0x7f029002daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.174+0000 7f029d7fa640 1 -- 172.21.15.87:0/4293582266 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290043020 con 0x7f02a01099d0 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.174+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a01ac720 con 0x7f02a01099d0 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.174+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a01b4010 con 0x7f02a01099d0 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.175+0000 7f029d7fa640 1 -- 172.21.15.87:0/4293582266 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029002de20 con 0x7f02a01099d0 2024-04-22T13:37:13.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.175+0000 7f029d7fa640 1 -- 172.21.15.87:0/4293582266 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290043020 con 0x7f02a01099d0 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.179+0000 7f02a8fa1740 1 Processor -- start 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.179+0000 7f02a8fa1740 1 -- start start 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de1807630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de1807c00 0x559de180a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559de17631f0 con 0x559de1807c00 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559de17004b0 con 0x559de17fe6d0 2024-04-22T13:37:13.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.180+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559de175f1b0 con 0x559de17ff080 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.182+0000 7f029d7fa640 1 -- 172.21.15.87:0/4293582266 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0290046020 con 0x7f02a01099d0 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.182+0000 7f02a494f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.182+0000 7f02a494f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56260/0 (socket says 172.21.15.87:56260) 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.182+0000 7f02a494f640 1 -- 172.21.15.87:0/3088274130 learned_addr learned my addr 172.21.15.87:0/3088274130 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.183+0000 7f029d7fa640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 0x7f02880a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.183+0000 7f029d7fa640 1 -- 172.21.15.87:0/4293582266 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f02900f2240 con 0x7f02a01099d0 2024-04-22T13:37:13.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.183+0000 7f029ffff640 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de1807630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f029f7fe640 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de1807c00 0x559de180a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 0x7f02880a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 -- 172.21.15.87:0/3088274130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 msgr2=0x559de1807630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de1807630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 -- 172.21.15.87:0/3088274130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de1807c00 msgr2=0x559de180a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de1807c00 0x559de180a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.184+0000 7f02a494f640 1 -- 172.21.15.87:0/3088274130 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de16f21b0 con 0x559de17fe6d0 2024-04-22T13:37:13.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.185+0000 7f02a494f640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 0x7f02880a6490 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0298015fd0 tx=0x7f0298019000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.185+0000 7f02a494f640 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0298015900 tx=0x7f0298015dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499035118ed89d9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.186+0000 7f02867fc640 1 -- 172.21.15.87:0/3088274130 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298015250 con 0x559de17fe6d0 2024-04-22T13:37:13.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.186+0000 7f02867fc640 1 -- 172.21.15.87:0/3088274130 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029800eca0 con 0x559de17fe6d0 2024-04-22T13:37:13.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.186+0000 7f02867fc640 1 -- 172.21.15.87:0/3088274130 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029801db30 con 0x559de17fe6d0 2024-04-22T13:37:13.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.187+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3088274130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 msgr2=0x559de17feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.187+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0298015900 tx=0x7f0298015dc0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3088274130 shutdown_connections 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de1807c00 0x559de180a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de1807630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3088274130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17fe6d0 0x559de17feab0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3088274130 >> 172.21.15.87:0/3088274130 conn(0x559de17f8da0 msgr2=0x559de17f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3088274130 shutdown_connections 2024-04-22T13:37:13.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.188+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3088274130 wait complete. 2024-04-22T13:37:13.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.189+0000 7f02a8fa1740 1 -- start start 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.189+0000 7f02a8fa1740 1 Processor -- start 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 0x559de18074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17ff080 0x559de1804880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1804e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559de180ab10 con 0x559de17ff080 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559de171c500 con 0x559de1807c00 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559de180ac90 con 0x559de17fe6d0 2024-04-22T13:37:13.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a8fa1740 10 client.0 _ll_register_callbacks cb 0x559de16eab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:13.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f02a494f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1804e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.190+0000 7f029ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 0x559de18074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.192+0000 7f029ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 0x559de18074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60664/0 (socket says 172.21.15.87:60664) 2024-04-22T13:37:13.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.192+0000 7f029ffff640 1 -- 172.21.15.87:0/1628694483 learned_addr learned my addr 172.21.15.87:0/1628694483 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:13.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.192+0000 7f029f7fe640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17ff080 0x559de1804880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.193+0000 7f02a494f640 1 -- 172.21.15.87:0/1628694483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 msgr2=0x559de18074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.193+0000 7f02a494f640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 0x559de18074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.193+0000 7f02a494f640 1 -- 172.21.15.87:0/1628694483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17ff080 msgr2=0x559de1804880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.193+0000 7f02a494f640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17ff080 0x559de1804880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.193+0000 7f02a494f640 1 -- 172.21.15.87:0/1628694483 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de14753f0 con 0x559de1807c00 2024-04-22T13:37:13.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.196+0000 7f02a494f640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1804e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0298005c40 tx=0x7f0298012b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66c8e4fc837e208 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.196+0000 7f027f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:13.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.197+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298032060 con 0x559de1807c00 2024-04-22T13:37:13.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.198+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f029800eca0 con 0x559de1807c00 2024-04-22T13:37:13.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.198+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f029800e7f0 con 0x559de1807c00 2024-04-22T13:37:13.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.200+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f029800ee50 con 0x559de1807c00 2024-04-22T13:37:13.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.201+0000 7f02a8fa1740 10 client.26294 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:13.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.201+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559de153f7d0 con 0x559de1807c00 2024-04-22T13:37:13.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.202+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029800eca0 con 0x559de1807c00 2024-04-22T13:37:13.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.203+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0298035020 con 0x559de1807c00 2024-04-22T13:37:13.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 10 client.26294 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:13.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:13.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 20 client.26294 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:13.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de1828da0 0x559de182b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 4 client.26294 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:13.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.204+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559de1702930 con 0x559de1828da0 2024-04-22T13:37:13.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.205+0000 7f029ffff640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de1828da0 0x559de182b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.215+0000 7f029ffff640 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de1828da0 0x559de182b190 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e462e91a21e36beb server_cookie=764280b4b71794df in_seq=0 out_seq=0 2024-04-22T13:37:13.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.215+0000 7f027f7fe640 10 client.26294 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:12 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2759979648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:13.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:12 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1472373261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:13.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.219+0000 7f027f7fe640 1 -- 172.21.15.87:0/1628694483 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559de1702930 con 0x559de1828da0 2024-04-22T13:37:13.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.219+0000 7f027f7fe640 10 client.26294 handle_command_reply: tid=0 2024-04-22T13:37:13.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.221+0000 7f02a8fa1740 1 client.26294 shutdown 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 msgr2=0x559de1804e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1804e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0298005c40 tx=0x7f0298012b50 comp rx=0 tx=0).stop 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 shutdown_connections 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de1828da0 0x559de182b190 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1804e00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17ff080 0x559de1804880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/1628694483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17fe6d0 0x559de18074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 >> 172.21.15.87:0/1628694483 conn(0x559de17f8da0 msgr2=0x559de17faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f027f7fe640 0 client.26294 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 shutdown_connections 2024-04-22T13:37:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.223+0000 7f02a8fa1740 1 -- 172.21.15.87:0/1628694483 wait complete. 2024-04-22T13:37:13.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.226+0000 7f02a8fa1740 20 client.26294 trim_cache size 0 max 16384 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 Processor -- start 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 -- start start 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de1813a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de17f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1806fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f028c010f50 con 0x559de17fe6d0 2024-04-22T13:37:13.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f028c00e820 con 0x559de1807c00 2024-04-22T13:37:13.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.271+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f028c00c670 con 0x559de17ff080 2024-04-22T13:37:13.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.273+0000 7f029ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de1813a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.273+0000 7f029ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de1813a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49012/0 (socket says 172.21.15.87:49012) 2024-04-22T13:37:13.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.273+0000 7f029ffff640 1 -- 172.21.15.87:0/482671164 learned_addr learned my addr 172.21.15.87:0/482671164 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:13.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.274+0000 7f029f7fe640 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de17f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.274+0000 7f02a494f640 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1806fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.277+0000 7f029f7fe640 1 -- 172.21.15.87:0/482671164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 msgr2=0x559de1806fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.277+0000 7f029f7fe640 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1806fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.277+0000 7f029f7fe640 1 -- 172.21.15.87:0/482671164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 msgr2=0x559de1813a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.277+0000 7f029f7fe640 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de1813a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.277+0000 7f029f7fe640 1 -- 172.21.15.87:0/482671164 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de180aa50 con 0x559de17ff080 2024-04-22T13:37:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.279+0000 7f029f7fe640 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de17f7ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02900efb10 tx=0x7f029002c230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=811f4b4d56967b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.279+0000 7f0285ffb640 1 -- 172.21.15.87:0/482671164 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02900bea60 con 0x559de17ff080 2024-04-22T13:37:13.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.279+0000 7f0285ffb640 1 -- 172.21.15.87:0/482671164 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02900c6fd0 con 0x559de17ff080 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.281+0000 7f02a8fa1740 1 -- 172.21.15.87:0/482671164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 msgr2=0x559de17f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.281+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de17f7ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02900efb10 tx=0x7f029002c230 comp rx=0 tx=0).stop 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.281+0000 7f02a8fa1740 1 -- 172.21.15.87:0/482671164 shutdown_connections 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.281+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de1807c00 0x559de1806fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.282+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de17ff080 0x559de17f7ca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.282+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/482671164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de1813a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.282+0000 7f02a8fa1740 1 -- 172.21.15.87:0/482671164 >> 172.21.15.87:0/482671164 conn(0x559de17f8da0 msgr2=0x559de17f7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.282+0000 7f02a8fa1740 1 -- 172.21.15.87:0/482671164 shutdown_connections 2024-04-22T13:37:13.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.282+0000 7f02a8fa1740 1 -- 172.21.15.87:0/482671164 wait complete. 2024-04-22T13:37:13.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.283+0000 7f02a8fa1740 1 -- start start 2024-04-22T13:37:13.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.283+0000 7f02a8fa1740 1 Processor -- start 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de180fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de1807c00 0x559de180b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f028c00cce0 con 0x559de17fe6d0 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f028c00e150 con 0x559de17ff080 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f028c00d3b0 con 0x559de1807c00 2024-04-22T13:37:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.284+0000 7f02a8fa1740 10 client.0 _ll_register_callbacks cb 0x7f0298004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.286+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.286+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56302/0 (socket says 172.21.15.87:56302) 2024-04-22T13:37:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.286+0000 7f029f7fe640 1 -- 172.21.15.87:0/3935088780 learned_addr learned my addr 172.21.15.87:0/3935088780 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.286+0000 7f02a494f640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de1807c00 0x559de180b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029ffff640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de180fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029f7fe640 1 -- 172.21.15.87:0/3935088780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de1807c00 msgr2=0x559de180b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029f7fe640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de1807c00 0x559de180b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029f7fe640 1 -- 172.21.15.87:0/3935088780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 msgr2=0x559de180fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:13.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029f7fe640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de180fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.288+0000 7f029f7fe640 1 -- 172.21.15.87:0/3935088780 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de180ba50 con 0x559de17ff080 2024-04-22T13:37:13.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.292+0000 7f029f7fe640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02900bad30 tx=0x7f029002db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f85d210372d42668 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:13.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.292+0000 7f027f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:13.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.294+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02900bea60 con 0x559de17ff080 2024-04-22T13:37:13.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.294+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f02900c6d50 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.294+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02900c6fd0 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.295+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f02900be360 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.295+0000 7f02a8fa1740 10 client.26300 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.295+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559de188ec60 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02900c6fd0 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f02900c2170 con 0x559de17ff080 2024-04-22T13:37:13.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f02a8fa1740 10 client.26300 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:13.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f02a8fa1740 10 client.26300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:13.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f02a8fa1740 20 client.26300 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:13.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de188f380 0x559de18917a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:13.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.296+0000 7f02a8fa1740 4 client.26300 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:13.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.297+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559de1770c10 con 0x559de188f380 2024-04-22T13:37:13.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.297+0000 7f029ffff640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de188f380 0x559de18917a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.302+0000 7f027f7fe640 10 client.26300 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.302+0000 7f029ffff640 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de188f380 0x559de18917a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8289d19742192c17 server_cookie=196278daba4f5bff in_seq=0 out_seq=0 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f027f7fe640 1 -- 172.21.15.87:0/3935088780 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559de1770c10 con 0x559de188f380 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f027f7fe640 10 client.26300 handle_command_reply: tid=0 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 client.26300 shutdown 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 msgr2=0x559de1810250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02900bad30 tx=0x7f029002db90 comp rx=0 tx=0).stop 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 shutdown_connections 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559de188f380 0x559de18917a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559de1807c00 0x559de180b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559de17ff080 0x559de1810250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 --2- 172.21.15.87:0/3935088780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559de17fe6d0 0x559de180fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 >> 172.21.15.87:0/3935088780 conn(0x559de17f8da0 msgr2=0x559de17ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.303+0000 7f027f7fe640 0 client.26300 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.304+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 shutdown_connections 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.304+0000 7f02a8fa1740 1 -- 172.21.15.87:0/3935088780 wait complete. 2024-04-22T13:37:13.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.304+0000 7f02a8fa1740 20 client.26300 trim_cache size 0 max 16384 2024-04-22T13:37:13.302 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.307+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 msgr2=0x7f02880a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.307+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 0x7f02880a6490 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0298015fd0 tx=0x7f0298019000 comp rx=0 tx=0).stop 2024-04-22T13:37:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.307+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 msgr2=0x7f02a01a79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.307+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0290005ab0 tx=0x7f029002daa0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 shutdown_connections 2024-04-22T13:37:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f02880a3fd0 0x7f02880a6490 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a01105f0 0x7f02a01abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02a01099d0 0x7f02a01a79e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 --2- 172.21.15.87:0/4293582266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02a0108f50 0x7f02a01a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 >> 172.21.15.87:0/4293582266 conn(0x7f02a006ced0 msgr2=0x7f02a0070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 shutdown_connections 2024-04-22T13:37:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:13.308+0000 7f02a5150640 1 -- 172.21.15.87:0/4293582266 wait complete. 2024-04-22T13:37:14.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:13 smithi120 ceph-mon[31101]: pgmap v164: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 29 MiB/s wr, 233 op/s 2024-04-22T13:37:14.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:13 smithi132 ceph-mon[31531]: pgmap v164: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 29 MiB/s wr, 233 op/s 2024-04-22T13:37:14.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:13 smithi087 ceph-mon[27999]: pgmap v164: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 29 MiB/s wr, 233 op/s 2024-04-22T13:37:14.334 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:37:14.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:37:14.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:37:14.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.559+0000 7f89d5e81640 1 -- 172.21.15.87:0/2361014088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0105f70 msgr2=0x7f89d010db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.559+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2361014088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0105f70 0x7f89d010db40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009900 tx=0x7f89c002bc40 comp rx=0 tx=0).stop 2024-04-22T13:37:14.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.562+0000 7f89d5e81640 1 -- 172.21.15.87:0/2361014088 shutdown_connections 2024-04-22T13:37:14.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.562+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2361014088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0105f70 0x7f89d010db40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.562+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2361014088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89d01018a0 0x7f89d0105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.562+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2361014088 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0100ef0 0x7f89d01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.562+0000 7f89d5e81640 1 -- 172.21.15.87:0/2361014088 >> 172.21.15.87:0/2361014088 conn(0x7f89d0077a30 msgr2=0x7f89d00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.564+0000 7f89d5e81640 1 -- 172.21.15.87:0/2361014088 shutdown_connections 2024-04-22T13:37:14.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.566+0000 7f89d5e81640 1 -- 172.21.15.87:0/2361014088 wait complete. 2024-04-22T13:37:14.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.567+0000 7f89d5e81640 1 Processor -- start 2024-04-22T13:37:14.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.567+0000 7f89d5e81640 1 -- start start 2024-04-22T13:37:14.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0100ef0 0x7f89d0116430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89d0105f70 0x7f89d0116970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89d010fea0 con 0x7f89d0111590 2024-04-22T13:37:14.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89d010fd20 con 0x7f89d0105f70 2024-04-22T13:37:14.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.568+0000 7f89d5e81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0110020 con 0x7f89d0100ef0 2024-04-22T13:37:14.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.570+0000 7f89cffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.570+0000 7f89cffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49042/0 (socket says 172.21.15.87:49042) 2024-04-22T13:37:14.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.570+0000 7f89cffff640 1 -- 172.21.15.87:0/2021395727 learned_addr learned my addr 172.21.15.87:0/2021395727 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.572+0000 7f89cffff640 1 -- 172.21.15.87:0/2021395727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0100ef0 msgr2=0x7f89d0116430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:37:14.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.572+0000 7f89cffff640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0100ef0 0x7f89d0116430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.572+0000 7f89cffff640 1 -- 172.21.15.87:0/2021395727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89d0105f70 msgr2=0x7f89d0116970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:37:14.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.573+0000 7f89cffff640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89d0105f70 0x7f89d0116970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.573+0000 7f89cffff640 1 -- 172.21.15.87:0/2021395727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d0112250 con 0x7f89d0111590 2024-04-22T13:37:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.576+0000 7f89cffff640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009850 tx=0x7f89c002ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.578+0000 7f89ccff9640 1 -- 172.21.15.87:0/2021395727 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0043020 con 0x7f89d0111590 2024-04-22T13:37:14.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.578+0000 7f89ccff9640 1 -- 172.21.15.87:0/2021395727 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c00052b0 con 0x7f89d0111590 2024-04-22T13:37:14.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.578+0000 7f89ccff9640 1 -- 172.21.15.87:0/2021395727 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0003ca0 con 0x7f89d0111590 2024-04-22T13:37:14.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.580+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d011c140 con 0x7f89d0111590 2024-04-22T13:37:14.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.580+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d011c640 con 0x7f89d0111590 2024-04-22T13:37:14.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.592+0000 7f89d8a76740 1 Processor -- start 2024-04-22T13:37:14.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89ccff9640 1 -- 172.21.15.87:0/2021395727 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f89c0046020 con 0x7f89d0111590 2024-04-22T13:37:14.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 -- start start 2024-04-22T13:37:14.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb93de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556dbb8a1f00 con 0x556dbb946fb0 2024-04-22T13:37:14.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556dbb83f590 con 0x556dbb93da80 2024-04-22T13:37:14.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.594+0000 7f89d8a76740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556dbb89dec0 con 0x556dbb93e430 2024-04-22T13:37:14.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.595+0000 7f89ccff9640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 0x7f89b80a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.595+0000 7f89ccff9640 1 -- 172.21.15.87:0/2021395727 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f89c00f1fc0 con 0x7f89d0111590 2024-04-22T13:37:14.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.596+0000 7f89cffff640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 0x7f89b80a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.596+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.596+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49046/0 (socket says 172.21.15.87:49046) 2024-04-22T13:37:14.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.596+0000 7f89ceffd640 1 -- 172.21.15.87:0/2510524658 learned_addr learned my addr 172.21.15.87:0/2510524658 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.596+0000 7f89ceffd640 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb93de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.597+0000 7f89cffff640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 0x7f89b80a6360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89c002c1d0 tx=0x7f89c0009260 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.598+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.599+0000 7f89ceffd640 1 -- 172.21.15.87:0/2510524658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 msgr2=0x556dbb9469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.599+0000 7f89ceffd640 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.599+0000 7f89ceffd640 1 -- 172.21.15.87:0/2510524658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 msgr2=0x556dbb93de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.599+0000 7f89ceffd640 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb93de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.599+0000 7f89ceffd640 1 -- 172.21.15.87:0/2510524658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbb67c7e0 con 0x556dbb946fb0 2024-04-22T13:37:14.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.604+0000 7f89ceffd640 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f89c8013580 tx=0x7f89c80018f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21ade17fb6de2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.605+0000 7f89bdffb640 1 -- 172.21.15.87:0/2510524658 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8012040 con 0x556dbb946fb0 2024-04-22T13:37:14.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.605+0000 7f89bdffb640 1 -- 172.21.15.87:0/2510524658 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c8005e30 con 0x556dbb946fb0 2024-04-22T13:37:14.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.605+0000 7f89bdffb640 1 -- 172.21.15.87:0/2510524658 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8018ab0 con 0x556dbb946fb0 2024-04-22T13:37:14.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.608+0000 7f89d8a76740 1 -- 172.21.15.87:0/2510524658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 msgr2=0x556dbb949400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f89c8013580 tx=0x7f89c80018f0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 -- 172.21.15.87:0/2510524658 shutdown_connections 2024-04-22T13:37:14.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946fb0 0x556dbb949400 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 --2- 172.21.15.87:0/2510524658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb93de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.609+0000 7f89d8a76740 1 -- 172.21.15.87:0/2510524658 >> 172.21.15.87:0/2510524658 conn(0x556dbb938150 msgr2=0x556dbb938530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.610+0000 7f89d8a76740 1 -- 172.21.15.87:0/2510524658 shutdown_connections 2024-04-22T13:37:14.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.610+0000 7f89d8a76740 1 -- 172.21.15.87:0/2510524658 wait complete. 2024-04-22T13:37:14.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.611+0000 7f89d8a76740 1 -- start start 2024-04-22T13:37:14.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.611+0000 7f89d8a76740 1 Processor -- start 2024-04-22T13:37:14.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb9440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb944b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556dbb949f20 con 0x556dbb93da80 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556dbb85b580 con 0x556dbb946fb0 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556dbb94a0a0 con 0x556dbb93e430 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89d8a76740 10 client.0 _ll_register_callbacks cb 0x556dbb861650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb9440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb9440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49060/0 (socket says 172.21.15.87:49060) 2024-04-22T13:37:14.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.612+0000 7f89ceffd640 1 -- 172.21.15.87:0/3383263762 learned_addr learned my addr 172.21.15.87:0/3383263762 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.613+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.613+0000 7f89ceffd640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb944b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 -- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 msgr2=0x556dbb944b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb944b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 -- 172.21.15.87:0/3383263762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 msgr2=0x556dbb9440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb9440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 -- 172.21.15.87:0/3383263762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbb6dcf70 con 0x556dbb93e430 2024-04-22T13:37:14.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9445e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x556dbb93e830 tx=0x7f89c400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e48eb31616e7c2f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.614+0000 7f89b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:14.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.615+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4015070 con 0x556dbb93e430 2024-04-22T13:37:14.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.615+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c8005e30 con 0x556dbb93e430 2024-04-22T13:37:14.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.616+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f89c40115f0 con 0x556dbb93e430 2024-04-22T13:37:14.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.619+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f89c401f050 con 0x556dbb93e430 2024-04-22T13:37:14.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.620+0000 7f89d8a76740 10 client.26362 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:14.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.620+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556dbb574170 con 0x556dbb93e430 2024-04-22T13:37:14.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.622+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c8005e30 con 0x556dbb93e430 2024-04-22T13:37:14.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f89c401a970 con 0x556dbb93e430 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 10 client.26362 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 10 client.26362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 20 client.26362 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9579f0 0x556dbb959e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 4 client.26362 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.623+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dbb8af920 con 0x556dbb9579f0 2024-04-22T13:37:14.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.624+0000 7f89ceffd640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9579f0 0x556dbb959e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.637+0000 7f89ceffd640 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9579f0 0x556dbb959e70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=678376b231bf17e0 server_cookie=5eff442015aa8d2 in_seq=0 out_seq=0 2024-04-22T13:37:14.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.637+0000 7f89b2ffd640 10 client.26362 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:14.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.639+0000 7f89b2ffd640 1 -- 172.21.15.87:0/3383263762 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dbb8af920 con 0x556dbb9579f0 2024-04-22T13:37:14.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.639+0000 7f89b2ffd640 10 client.26362 handle_command_reply: tid=0 2024-04-22T13:37:14.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.641+0000 7f89d8a76740 1 client.26362 shutdown 2024-04-22T13:37:14.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.643+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 msgr2=0x556dbb9445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.643+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9445e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x556dbb93e830 tx=0x7f89c400c520 comp rx=0 tx=0).stop 2024-04-22T13:37:14.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 shutdown_connections 2024-04-22T13:37:14.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9579f0 0x556dbb959e70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb944b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb93e430 0x556dbb9445e0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x556dbb93e830 tx=0x7f89c400c520 comp rx=0 tx=0).stop 2024-04-22T13:37:14.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 --2- 172.21.15.87:0/3383263762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb9440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 >> 172.21.15.87:0/3383263762 conn(0x556dbb938150 msgr2=0x556dbb939ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.645+0000 7f89b2ffd640 0 client.26362 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:14.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.648+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 shutdown_connections 2024-04-22T13:37:14.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.650+0000 7f89d8a76740 1 -- 172.21.15.87:0/3383263762 wait complete. 2024-04-22T13:37:14.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.652+0000 7f89d8a76740 20 client.26362 trim_cache size 0 max 16384 2024-04-22T13:37:14.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.697+0000 7f89d8a76740 1 Processor -- start 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.697+0000 7f89d8a76740 1 -- start start 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.697+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.697+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946fb0 0x556dbb945ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946410 0x556dbb95f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89d8a76740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89d01c7570 con 0x556dbb946410 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89d8a76740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0073050 con 0x556dbb93da80 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89d8a76740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0070f50 con 0x556dbb946fb0 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89ceffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56356/0 (socket says 172.21.15.87:56356) 2024-04-22T13:37:14.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.698+0000 7f89ceffd640 1 -- 172.21.15.87:0/4154025650 learned_addr learned my addr 172.21.15.87:0/4154025650 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.699+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946fb0 0x556dbb945ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.699+0000 7f89ceffd640 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946410 0x556dbb95f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.701+0000 7f89ceffd640 1 -- 172.21.15.87:0/4154025650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946fb0 msgr2=0x556dbb945ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.701+0000 7f89ceffd640 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946fb0 0x556dbb945ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.701+0000 7f89ceffd640 1 -- 172.21.15.87:0/4154025650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946410 msgr2=0x556dbb95f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.701+0000 7f89ceffd640 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946410 0x556dbb95f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.701+0000 7f89ceffd640 1 -- 172.21.15.87:0/4154025650 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbb956250 con 0x556dbb93da80 2024-04-22T13:37:14.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.704+0000 7f89ceffd640 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f89c8001890 tx=0x7f89c8007040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeafbf0b6d93e2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.705+0000 7f89bd7fa640 1 -- 172.21.15.87:0/4154025650 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8003b70 con 0x556dbb93da80 2024-04-22T13:37:14.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.705+0000 7f89bd7fa640 1 -- 172.21.15.87:0/4154025650 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c8003d10 con 0x556dbb93da80 2024-04-22T13:37:14.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.706+0000 7f89bd7fa640 1 -- 172.21.15.87:0/4154025650 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c80282a0 con 0x556dbb93da80 2024-04-22T13:37:14.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 -- 172.21.15.87:0/4154025650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 msgr2=0x556dbb952de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f89c8001890 tx=0x7f89c8007040 comp rx=0 tx=0).stop 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 -- 172.21.15.87:0/4154025650 shutdown_connections 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb946410 0x556dbb95f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946fb0 0x556dbb945ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 --2- 172.21.15.87:0/4154025650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb93da80 0x556dbb952de0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.707+0000 7f89d8a76740 1 -- 172.21.15.87:0/4154025650 >> 172.21.15.87:0/4154025650 conn(0x556dbb938150 msgr2=0x556dbb9457c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.709+0000 7f89d8a76740 1 -- 172.21.15.87:0/4154025650 shutdown_connections 2024-04-22T13:37:14.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.709+0000 7f89d8a76740 1 -- 172.21.15.87:0/4154025650 wait complete. 2024-04-22T13:37:14.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.709+0000 7f89d8a76740 1 -- start start 2024-04-22T13:37:14.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.710+0000 7f89d8a76740 1 Processor -- start 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb94f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb94c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0071510 con 0x556dbb93da80 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0072980 con 0x556dbb946fb0 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0071be0 con 0x556dbb946410 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89d8a76740 10 client.0 _ll_register_callbacks cb 0x7f89c4006e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89cf7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89cf7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60740/0 (socket says 172.21.15.87:60740) 2024-04-22T13:37:14.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.711+0000 7f89cf7fe640 1 -- 172.21.15.87:0/1256271866 learned_addr learned my addr 172.21.15.87:0/1256271866 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.712+0000 7f89ceffd640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb94f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.713+0000 7f89ceffd640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb94c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.714+0000 7f89cf7fe640 1 -- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 msgr2=0x556dbb94c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.714+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb94c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.714+0000 7f89cf7fe640 1 -- 172.21.15.87:0/1256271866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 msgr2=0x556dbb94f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.714+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb94f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.714+0000 7f89cf7fe640 1 -- 172.21.15.87:0/1256271866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbb94ce70 con 0x556dbb946410 2024-04-22T13:37:14.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.720+0000 7f89cf7fe640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f89c401af80 tx=0x7f89c401ac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f9788512c918b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.720+0000 7f89b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:14.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.722+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4015040 con 0x556dbb946410 2024-04-22T13:37:14.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.722+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c8003d10 con 0x556dbb946410 2024-04-22T13:37:14.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.722+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f89c4024640 con 0x556dbb946410 2024-04-22T13:37:14.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.725+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f89c4033050 con 0x556dbb946410 2024-04-22T13:37:14.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.727+0000 7f89d8a76740 10 client.26368 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:14.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.727+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556dbb9beee0 con 0x556dbb946410 2024-04-22T13:37:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.732+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89c8003d10 con 0x556dbb946410 2024-04-22T13:37:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.732+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f89c402d820 con 0x556dbb946410 2024-04-22T13:37:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.732+0000 7f89d8a76740 10 client.26368 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.732+0000 7f89d8a76740 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:14.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.733+0000 7f89d8a76740 20 client.26368 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.733+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9bf660 0x556dbb9c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.733+0000 7f89d8a76740 4 client.26368 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:37:14.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.733+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556dbb8a9790 con 0x556dbb9bf660 2024-04-22T13:37:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.735+0000 7f89ceffd640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9bf660 0x556dbb9c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.742+0000 7f89ceffd640 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9bf660 0x556dbb9c1a80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecaaf941e6fc97fa server_cookie=1088b6a3bcfac85f in_seq=0 out_seq=0 2024-04-22T13:37:14.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.742+0000 7f89b2ffd640 10 client.26368 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:14.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.744+0000 7f89b2ffd640 1 -- 172.21.15.87:0/1256271866 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556dbb8a9790 con 0x556dbb9bf660 2024-04-22T13:37:14.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.744+0000 7f89b2ffd640 10 client.26368 handle_command_reply: tid=0 2024-04-22T13:37:14.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.744+0000 7f89d8a76740 1 client.26368 shutdown 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 msgr2=0x556dbb94c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f89c401af80 tx=0x7f89c401ac00 comp rx=0 tx=0).stop 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 shutdown_connections 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dbb9bf660 0x556dbb9c1a80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dbb946fb0 0x556dbb94c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dbb946410 0x556dbb94c2c0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f89c401af80 tx=0x7f89c401ac00 comp rx=0 tx=0).stop 2024-04-22T13:37:14.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 --2- 172.21.15.87:0/1256271866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dbb93da80 0x556dbb94f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.750+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 >> 172.21.15.87:0/1256271866 conn(0x556dbb938150 msgr2=0x556dbb9368b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.752+0000 7f89b2ffd640 0 client.26368 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:14.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.752+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 shutdown_connections 2024-04-22T13:37:14.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.753+0000 7f89d8a76740 1 -- 172.21.15.87:0/1256271866 wait complete. 2024-04-22T13:37:14.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.755+0000 7f89d8a76740 20 client.26368 trim_cache size 0 max 16384 2024-04-22T13:37:14.752 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:14.752 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:37:14.752 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "ed267e10-8d06-4912-a9cb-e86a9590f17e", 2024-04-22T13:37:14.752 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:37:14.752 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:14.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.756+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 msgr2=0x7f89b80a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.756+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 0x7f89b80a6360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89c002c1d0 tx=0x7f89c0009260 comp rx=0 tx=0).stop 2024-04-22T13:37:14.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.756+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 msgr2=0x7f89d0111a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.756+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009850 tx=0x7f89c002ca20 comp rx=0 tx=0).stop 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 shutdown_connections 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89b80a3ea0 0x7f89b80a6360 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89d0111590 0x7f89d0111a40 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89d0105f70 0x7f89d0116970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 --2- 172.21.15.87:0/2021395727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89d0100ef0 0x7f89d0116430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.758+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 >> 172.21.15.87:0/2021395727 conn(0x7f89d0077a30 msgr2=0x7f89d00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.759+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 shutdown_connections 2024-04-22T13:37:14.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.759+0000 7f89d5e81640 1 -- 172.21.15.87:0/2021395727 wait complete. 2024-04-22T13:37:14.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:37:14.775 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:37:14.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.990+0000 7f81ee9ce640 1 -- 172.21.15.87:0/2378883298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e81010a0 msgr2=0x7f81e8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.990+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/2378883298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e81010a0 0x7f81e8101500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009900 tx=0x7f81dc02bc80 comp rx=0 tx=0).stop 2024-04-22T13:37:14.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.992+0000 7f81ee9ce640 1 -- 172.21.15.87:0/2378883298 shutdown_connections 2024-04-22T13:37:14.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.992+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/2378883298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e8101a40 0x7f81e810dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.992+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/2378883298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e81010a0 0x7f81e8101500 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.992+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/2378883298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8106ed0 0x7f81e81072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.992+0000 7f81ee9ce640 1 -- 172.21.15.87:0/2378883298 >> 172.21.15.87:0/2378883298 conn(0x7f81e8077c40 msgr2=0x7f81e8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.994+0000 7f81ee9ce640 1 -- 172.21.15.87:0/2378883298 shutdown_connections 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.994+0000 7f81ee9ce640 1 -- 172.21.15.87:0/2378883298 wait complete. 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 Processor -- start 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 -- start start 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e81010a0 0x7f81e81966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8101a40 0x7f81e8196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81111d0 con 0x7f81e8106ed0 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8111050 con 0x7f81e8101a40 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.995+0000 7f81ee9ce640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8111350 con 0x7f81e81010a0 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81e77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8101a40 0x7f81e8196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49092/0 (socket says 172.21.15.87:49092) 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 -- 172.21.15.87:0/3351010781 learned_addr learned my addr 172.21.15.87:0/3351010781 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81e7fff640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e81010a0 0x7f81e81966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 -- 172.21.15.87:0/3351010781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e81010a0 msgr2=0x7f81e81966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e81010a0 0x7f81e81966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 -- 172.21.15.87:0/3351010781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8101a40 msgr2=0x7f81e8196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8101a40 0x7f81e8196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.996+0000 7f81ecf6b640 1 -- 172.21.15.87:0/3351010781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e81a2a00 con 0x7f81e8106ed0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b970 tx=0x7f81e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81e57fa640 1 -- 172.21.15.87:0/3351010781 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0019070 con 0x7f81e8106ed0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81e57fa640 1 -- 172.21.15.87:0/3351010781 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81e0014440 con 0x7f81e8106ed0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e81a2cf0 con 0x7f81e8106ed0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81e57fa640 1 -- 172.21.15.87:0/3351010781 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00075d0 con 0x7f81e8106ed0 2024-04-22T13:37:14.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:14.997+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81a3210 con 0x7f81e8106ed0 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.001+0000 7f81f15c4740 1 Processor -- start 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.002+0000 7f81f15c4740 1 -- start start 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961ae6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b02e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561961a4b1e0 con 0x561961ae6230 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5619619e8390 con 0x561961ae6b50 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81f15c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561961a471a0 con 0x561961b00a90 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56406/0 (socket says 172.21.15.87:56406) 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81e7fff640 1 -- 172.21.15.87:0/577357774 learned_addr learned my addr 172.21.15.87:0/577357774 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81e77fe640 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b02e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.003+0000 7f81e77fe640 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961ae6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e57fa640 1 -- 172.21.15.87:0/3351010781 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f81e001c070 con 0x7f81e8106ed0 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e7fff640 1 -- 172.21.15.87:0/577357774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 msgr2=0x561961b02e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e7fff640 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b02e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e7fff640 1 -- 172.21.15.87:0/577357774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 msgr2=0x561961ae6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e7fff640 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961ae6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.004+0000 7f81e7fff640 1 -- 172.21.15.87:0/577357774 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619618277d0 con 0x561961ae6b50 2024-04-22T13:37:15.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.008+0000 7f81e7fff640 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f81d800b7f0 tx=0x7f81d800bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8647db299616704 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:15.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.008+0000 7f81e57fa640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 0x7f81d00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.008+0000 7f81ce7fc640 1 -- 172.21.15.87:0/577357774 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800d950 con 0x561961ae6b50 2024-04-22T13:37:15.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.008+0000 7f81ce7fc640 1 -- 172.21.15.87:0/577357774 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d8018070 con 0x561961ae6b50 2024-04-22T13:37:15.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.008+0000 7f81ce7fc640 1 -- 172.21.15.87:0/577357774 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d80123e0 con 0x561961ae6b50 2024-04-22T13:37:15.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.009+0000 7f81e57fa640 1 -- 172.21.15.87:0/3351010781 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f81e00cfc10 con 0x7f81e8106ed0 2024-04-22T13:37:15.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.010+0000 7f81f15c4740 1 -- 172.21.15.87:0/577357774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 msgr2=0x561961b00550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.010+0000 7f81f15c4740 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f81d800b7f0 tx=0x7f81d800bcb0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.011+0000 7f81f15c4740 1 -- 172.21.15.87:0/577357774 shutdown_connections 2024-04-22T13:37:15.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.011+0000 7f81f15c4740 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b02e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.011+0000 7f81f15c4740 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b00550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.011+0000 7f81f15c4740 1 --2- 172.21.15.87:0/577357774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961ae6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.011+0000 7f81f15c4740 1 -- 172.21.15.87:0/577357774 >> 172.21.15.87:0/577357774 conn(0x561961ae0cb0 msgr2=0x561961ae1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.013+0000 7f81e77fe640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 0x7f81d00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.013+0000 7f81f15c4740 1 -- 172.21.15.87:0/577357774 shutdown_connections 2024-04-22T13:37:15.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.014+0000 7f81f15c4740 1 -- 172.21.15.87:0/577357774 wait complete. 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.014+0000 7f81f15c4740 1 -- start start 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 Processor -- start 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6b50 0x561961afdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961b00a90 0x561961afe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561961b038d0 con 0x561961ae6b50 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561961a042f0 con 0x561961b00a90 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561961b03a50 con 0x561961ae6230 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.015+0000 7f81f15c4740 10 client.0 _ll_register_callbacks cb 0x561961a11580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60794/0 (socket says 172.21.15.87:60794) 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 -- 172.21.15.87:0/1241806496 learned_addr learned my addr 172.21.15.87:0/1241806496 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961b00a90 0x561961afe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 -- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961b00a90 msgr2=0x561961afe510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961b00a90 0x561961afe510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 -- 172.21.15.87:0/1241806496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6b50 msgr2=0x561961afdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6b50 0x561961afdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.016+0000 7f81e7fff640 1 -- 172.21.15.87:0/1241806496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56196175d3f0 con 0x561961ae6230 2024-04-22T13:37:15.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.017+0000 7f81e77fe640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 0x7f81d00a6320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f81dc02c290 tx=0x7f81dc005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:15.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.019+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f81d8008010 tx=0x7f81d80136b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b17af5ef3ac781 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:15.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.020+0000 7f81c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:15.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.020+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8022020 con 0x561961ae6230 2024-04-22T13:37:15.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.020+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d8018070 con 0x561961ae6230 2024-04-22T13:37:15.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.020+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f81d800dd60 con 0x561961ae6230 2024-04-22T13:37:15.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.021+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f81d802d050 con 0x561961ae6230 2024-04-22T13:37:15.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.022+0000 7f81f15c4740 10 client.26377 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:15.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.022+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56196171d0f0 con 0x561961ae6230 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d8018070 con 0x561961ae6230 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f81d8013ce0 con 0x561961ae6230 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 10 client.26377 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 10 client.26377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 20 client.26377 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b10a70 0x561961b12e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 4 client.26377 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:15.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.023+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561961a58c00 con 0x561961b10a70 2024-04-22T13:37:15.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.026+0000 7f81e77fe640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b10a70 0x561961b12e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.028+0000 7f81e77fe640 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b10a70 0x561961b12e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ab711f2671cb1d7 server_cookie=ee2f0851eade2428 in_seq=0 out_seq=0 2024-04-22T13:37:15.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.028+0000 7f81c77fe640 10 client.26377 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:15.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.029+0000 7f81c77fe640 1 -- 172.21.15.87:0/1241806496 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561961a58c00 con 0x561961b10a70 2024-04-22T13:37:15.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.029+0000 7f81c77fe640 10 client.26377 handle_command_reply: tid=0 2024-04-22T13:37:15.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 client.26377 shutdown 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 msgr2=0x561961afda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f81d8008010 tx=0x7f81d80136b0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 shutdown_connections 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b10a70 0x561961b12e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961b00a90 0x561961afe510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6b50 0x561961afdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1241806496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961ae6230 0x561961afda90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 >> 172.21.15.87:0/1241806496 conn(0x561961ae0cb0 msgr2=0x561961ae2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81c77fe640 0 client.26377 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.030+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 shutdown_connections 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.031+0000 7f81f15c4740 1 -- 172.21.15.87:0/1241806496 wait complete. 2024-04-22T13:37:15.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.032+0000 7f81f15c4740 20 client.26377 trim_cache size 0 max 16384 2024-04-22T13:37:15.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.064+0000 7f81f15c4740 1 Processor -- start 2024-04-22T13:37:15.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.064+0000 7f81f15c4740 1 -- start start 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961afd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961ae2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81a5340 con 0x561961ae6230 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81a5a10 con 0x561961ae6b50 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81f15c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8073f20 con 0x561961b00a90 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49124/0 (socket says 172.21.15.87:49124) 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 -- 172.21.15.87:0/3776049184 learned_addr learned my addr 172.21.15.87:0/3776049184 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e77fe640 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961afd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 -- 172.21.15.87:0/3776049184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 msgr2=0x561961ae2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961ae2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 -- 172.21.15.87:0/3776049184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 msgr2=0x561961afd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961afd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.065+0000 7f81e7fff640 1 -- 172.21.15.87:0/3776049184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561961af8860 con 0x561961ae6230 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.066+0000 7f81e7fff640 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d800d5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234ec22f822317bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.066+0000 7f81cdffb640 1 -- 172.21.15.87:0/3776049184 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8022030 con 0x561961ae6230 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.066+0000 7f81cdffb640 1 -- 172.21.15.87:0/3776049184 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d8018040 con 0x561961ae6230 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.066+0000 7f81cdffb640 1 -- 172.21.15.87:0/3776049184 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8026d90 con 0x561961ae6230 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 -- 172.21.15.87:0/3776049184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 msgr2=0x561961b0c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d800d5b0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 -- 172.21.15.87:0/3776049184 shutdown_connections 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961ae2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961afd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 --2- 172.21.15.87:0/3776049184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b0c7d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.067+0000 7f81f15c4740 1 -- 172.21.15.87:0/3776049184 >> 172.21.15.87:0/3776049184 conn(0x561961ae0cb0 msgr2=0x561961ae1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:15.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.068+0000 7f81f15c4740 1 -- 172.21.15.87:0/3776049184 shutdown_connections 2024-04-22T13:37:15.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.068+0000 7f81f15c4740 1 -- 172.21.15.87:0/3776049184 wait complete. 2024-04-22T13:37:15.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.068+0000 7f81f15c4740 1 -- start start 2024-04-22T13:37:15.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 Processor -- start 2024-04-22T13:37:15.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b09210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b04270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8072400 con 0x561961ae6230 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8073850 con 0x561961ae6b50 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.069+0000 7f81f15c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8072ab0 con 0x561961b00a90 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81f15c4740 10 client.0 _ll_register_callbacks cb 0x7f81dc005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49132/0 (socket says 172.21.15.87:49132) 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 -- 172.21.15.87:0/1550779120 learned_addr learned my addr 172.21.15.87:0/1550779120 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b04270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e77fe640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b09210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 -- 172.21.15.87:0/1550779120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 msgr2=0x561961b04270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b04270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 -- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 msgr2=0x561961b09210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b09210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 -- 172.21.15.87:0/1550779120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561961ae6980 con 0x561961ae6230 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81ecf6b640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81e7fff640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f81d8026560 tx=0x7f81d8013930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e552f8b9bf468f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.070+0000 7f81c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.071+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8022040 con 0x561961ae6230 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.071+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f81d8026a10 con 0x561961ae6230 2024-04-22T13:37:15.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.071+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d8018040 con 0x561961ae6230 2024-04-22T13:37:15.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.072+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f81d8041050 con 0x561961ae6230 2024-04-22T13:37:15.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.072+0000 7f81f15c4740 10 client.17223 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:15.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.072+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561961b76800 con 0x561961ae6230 2024-04-22T13:37:15.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.074+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d8018040 con 0x561961ae6230 2024-04-22T13:37:15.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.074+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f81d801c420 con 0x561961ae6230 2024-04-22T13:37:15.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.074+0000 7f81f15c4740 10 client.17223 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.074+0000 7f81f15c4740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.075+0000 7f81f15c4740 20 client.17223 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.075+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b76ef0 0x561961b79310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.075+0000 7f81f15c4740 4 client.17223 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.075+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561961a52a70 con 0x561961b76ef0 2024-04-22T13:37:15.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.076+0000 7f81e77fe640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b76ef0 0x561961b79310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:15.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.086+0000 7f81e77fe640 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b76ef0 0x561961b79310 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39f5d9230f2e6bde server_cookie=f9fe6b47f474ad61 in_seq=0 out_seq=0 2024-04-22T13:37:15.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.086+0000 7f81c77fe640 10 client.17223 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:15.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.086+0000 7f81c77fe640 1 -- 172.21.15.87:0/1550779120 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561961a52a70 con 0x561961b76ef0 2024-04-22T13:37:15.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.086+0000 7f81c77fe640 10 client.17223 handle_command_reply: tid=0 2024-04-22T13:37:15.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.086+0000 7f81f15c4740 1 client.17223 shutdown 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 msgr2=0x561961b08cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f81d8026560 tx=0x7f81d8013930 comp rx=0 tx=0).stop 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 shutdown_connections 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561961b76ef0 0x561961b79310 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81c77fe640 0 client.17223 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561961b00a90 0x561961b04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561961ae6b50 0x561961b09210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 --2- 172.21.15.87:0/1550779120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561961ae6230 0x561961b08cd0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.088+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 >> 172.21.15.87:0/1550779120 conn(0x561961ae0cb0 msgr2=0x561961adfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:15.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.089+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 shutdown_connections 2024-04-22T13:37:15.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.089+0000 7f81f15c4740 1 -- 172.21.15.87:0/1550779120 wait complete. 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.090+0000 7f81f15c4740 20 client.17223 trim_cache size 0 max 16384 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "ed267e10-8d06-4912-a9cb-e86a9590f17e": { 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "ed267e10-8d06-4912-a9cb-e86a9590f17e", 2024-04-22T13:37:15.086 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:37:15.087 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:15.087 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:15.087 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:15.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.093+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 msgr2=0x7f81d00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.093+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 0x7f81d00a6320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f81dc02c290 tx=0x7f81dc005fd0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.093+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 msgr2=0x7f81e81a24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:15.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.093+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b970 tx=0x7f81e000be30 comp rx=0 tx=0).stop 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 shutdown_connections 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81d00a3e60 0x7f81d00a6320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81e8106ed0 0x7f81e81a24c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81e8101a40 0x7f81e8196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 --2- 172.21.15.87:0/3351010781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81e81010a0 0x7f81e81966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 >> 172.21.15.87:0/3351010781 conn(0x7f81e8077c40 msgr2=0x7f81e806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 shutdown_connections 2024-04-22T13:37:15.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:15.096+0000 7f81ee9ce640 1 -- 172.21.15.87:0/3351010781 wait complete. 2024-04-22T13:37:15.109 INFO:tasks.cephfs.filesystem:scrub status for tag:ed267e10-8d06-4912-a9cb-e86a9590f17e - {'path': '/', 'tag': 'ed267e10-8d06-4912-a9cb-e86a9590f17e', 'options': 'recursive,force'} 2024-04-22T13:37:16.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:15 smithi120 ceph-mon[31101]: pgmap v165: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 31 MiB/s wr, 265 op/s 2024-04-22T13:37:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:15 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:37:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:15 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:15 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:37:16.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:15 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:15 smithi132 ceph-mon[31531]: pgmap v165: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 31 MiB/s wr, 265 op/s 2024-04-22T13:37:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:15 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:37:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:15 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:15 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:37:16.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:15 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:16.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:15 smithi087 ceph-mon[27999]: pgmap v165: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 31 MiB/s wr, 265 op/s 2024-04-22T13:37:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:15 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:37:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:15 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:15 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:37:16.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:15 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:17.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:16 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:37:17.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:16 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:37:17.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:16 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:37:18.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:17 smithi120 ceph-mon[31101]: pgmap v166: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 43 MiB/s wr, 405 op/s 2024-04-22T13:37:18.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:17 smithi132 ceph-mon[31531]: pgmap v166: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 43 MiB/s wr, 405 op/s 2024-04-22T13:37:18.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:17 smithi087 ceph-mon[27999]: pgmap v166: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 43 MiB/s wr, 405 op/s 2024-04-22T13:37:20.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:19 smithi120 ceph-mon[31101]: pgmap v167: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 209 MiB/s wr, 2.08k op/s 2024-04-22T13:37:20.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:19 smithi132 ceph-mon[31531]: pgmap v167: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 209 MiB/s wr, 2.08k op/s 2024-04-22T13:37:20.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:19 smithi087 ceph-mon[27999]: pgmap v167: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 209 MiB/s wr, 2.08k op/s 2024-04-22T13:37:22.169 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:21 smithi120 ceph-mon[31101]: pgmap v168: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 273 MiB/s wr, 2.71k op/s 2024-04-22T13:37:22.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:21 smithi132 ceph-mon[31531]: pgmap v168: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 273 MiB/s wr, 2.71k op/s 2024-04-22T13:37:22.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:21 smithi087 ceph-mon[27999]: pgmap v168: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 273 MiB/s wr, 2.71k op/s 2024-04-22T13:37:24.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:23 smithi132 ceph-mon[31531]: pgmap v169: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 322 MiB/s wr, 3.16k op/s 2024-04-22T13:37:24.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:23 smithi087 ceph-mon[27999]: pgmap v169: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 322 MiB/s wr, 3.16k op/s 2024-04-22T13:37:24.419 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:23 smithi120 ceph-mon[31101]: pgmap v169: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 322 MiB/s wr, 3.16k op/s 2024-04-22T13:37:25.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:24 smithi132 ceph-mon[31531]: pgmap v170: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 510 MiB/s wr, 4.31k op/s 2024-04-22T13:37:25.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:24 smithi087 ceph-mon[27999]: pgmap v170: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 510 MiB/s wr, 4.31k op/s 2024-04-22T13:37:25.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:24 smithi120 ceph-mon[31101]: pgmap v170: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 510 MiB/s wr, 4.31k op/s 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.215+0000 7f3f98f9f640 1 -- 172.21.15.87:0/3023291187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410aac0 msgr2=0x7f3f9410af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.215+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/3023291187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410aac0 0x7f3f9410af40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c013490 tx=0x7f3f8c0343f0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 -- 172.21.15.87:0/3023291187 shutdown_connections 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/3023291187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410aac0 0x7f3f9410af40 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/3023291187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410a100 0x7f3f9410a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/3023291187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f941097e0 0x7f3f94109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 -- 172.21.15.87:0/3023291187 >> 172.21.15.87:0/3023291187 conn(0x7f3f9406ce90 msgr2=0x7f3f9406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.216+0000 7f3f98f9f640 1 -- 172.21.15.87:0/3023291187 shutdown_connections 2024-04-22T13:37:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.221+0000 7f3f98f9f640 1 -- 172.21.15.87:0/3023291187 wait complete. 2024-04-22T13:37:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.222+0000 7f3f98f9f640 1 Processor -- start 2024-04-22T13:37:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 -- start start 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f941097e0 0x7f3f941b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410a100 0x7f3f941b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f941186a0 con 0x7f3f9410a100 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94118520 con 0x7f3f9410aac0 2024-04-22T13:37:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.223+0000 7f3f98f9f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f94118820 con 0x7f3f941097e0 2024-04-22T13:37:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.228+0000 7f3f93fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.228+0000 7f3f93fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60216/0 (socket says 172.21.15.87:60216) 2024-04-22T13:37:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.228+0000 7f3f93fff640 1 -- 172.21.15.87:0/4051198330 learned_addr learned my addr 172.21.15.87:0/4051198330 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410a100 0x7f3f941b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f93fff640 1 -- 172.21.15.87:0/4051198330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f941097e0 msgr2=0x7f3f941b58a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f93fff640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f941097e0 0x7f3f941b58a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f93fff640 1 -- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410a100 msgr2=0x7f3f941b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f93fff640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410a100 0x7f3f941b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.229+0000 7f3f93fff640 1 -- 172.21.15.87:0/4051198330 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f941b01d0 con 0x7f3f9410aac0 2024-04-22T13:37:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.231+0000 7f3f93fff640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c011040 tx=0x7f3f8c03b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.232+0000 7f3f90ff9640 1 -- 172.21.15.87:0/4051198330 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c040070 con 0x7f3f9410aac0 2024-04-22T13:37:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.232+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f941b0460 con 0x7f3f9410aac0 2024-04-22T13:37:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.232+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f941bc760 con 0x7f3f9410aac0 2024-04-22T13:37:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.232+0000 7f3f90ff9640 1 -- 172.21.15.87:0/4051198330 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8c0130e0 con 0x7f3f9410aac0 2024-04-22T13:37:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.232+0000 7f3f90ff9640 1 -- 172.21.15.87:0/4051198330 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c00f0d0 con 0x7f3f9410aac0 2024-04-22T13:37:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.237+0000 7f3f9cdf0740 1 Processor -- start 2024-04-22T13:37:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.241+0000 7f3f9cdf0740 1 -- start start 2024-04-22T13:37:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.242+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f07810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.240 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.240 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e916e6c8d0 con 0x55e916f07d50 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e916e09e80 con 0x55e916f08ac0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e916e68890 con 0x55e916f112d0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.243+0000 7f3f90ff9640 1 -- 172.21.15.87:0/4051198330 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3f8c0110d0 con 0x7f3f9410aac0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.244+0000 7f3f90ff9640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 0x7f3f7c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.244+0000 7f3f90ff9640 1 -- 172.21.15.87:0/4051198330 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f8c0f2310 con 0x7f3f9410aac0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f93fff640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 0x7f3f7c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f92ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f07810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f92ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f92ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f07810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60226/0 (socket says 172.21.15.87:60226) 2024-04-22T13:37:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f92ffd640 1 -- 172.21.15.87:0/4082627484 learned_addr learned my addr 172.21.15.87:0/4082627484 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.246+0000 7f3f937fe640 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.247+0000 7f3f937fe640 1 -- 172.21.15.87:0/4082627484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 msgr2=0x55e916f136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.247+0000 7f3f937fe640 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.247+0000 7f3f937fe640 1 -- 172.21.15.87:0/4082627484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 msgr2=0x55e916f07810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.247+0000 7f3f937fe640 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f07810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.247+0000 7f3f937fe640 1 -- 172.21.15.87:0/4082627484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e916e2b320 con 0x55e916f07d50 2024-04-22T13:37:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.248+0000 7f3f93fff640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 0x7f3f7c0a6320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0bac90 tx=0x7f3f8c011f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.248+0000 7f3f937fe640 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f081d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3f941b0b90 tx=0x7f3f88009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ee2c0e5f6483f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.249+0000 7f3f79ffb640 1 -- 172.21.15.87:0/4082627484 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88015020 con 0x55e916f07d50 2024-04-22T13:37:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.249+0000 7f3f79ffb640 1 -- 172.21.15.87:0/4082627484 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8800bd70 con 0x55e916f07d50 2024-04-22T13:37:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.250+0000 7f3f79ffb640 1 -- 172.21.15.87:0/4082627484 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88010380 con 0x55e916f07d50 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.251+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/4082627484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 msgr2=0x55e916f081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.251+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f081d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3f941b0b90 tx=0x7f3f88009b20 comp rx=0 tx=0).stop 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.254+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/4082627484 shutdown_connections 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.254+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.254+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f081d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.254+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/4082627484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f07810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.254+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/4082627484 >> 172.21.15.87:0/4082627484 conn(0x55e916ecec90 msgr2=0x55e916f02a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:26.255 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.258+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/4082627484 shutdown_connections 2024-04-22T13:37:26.256 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.260+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/4082627484 wait complete. 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.260+0000 7f3f9cdf0740 1 -- start start 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 Processor -- start 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f07d50 0x55e916f10bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f0df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 0x55e916f0e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e916f14000 con 0x55e916f08ac0 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e916df4eb0 con 0x55e916f112d0 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e916f14180 con 0x55e916f07d50 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.261+0000 7f3f9cdf0740 10 client.0 _ll_register_callbacks cb 0x55e916e21b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.262+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f0df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.262+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f0df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57578/0 (socket says 172.21.15.87:57578) 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.262+0000 7f3f937fe640 1 -- 172.21.15.87:0/870322051 learned_addr learned my addr 172.21.15.87:0/870322051 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:26.258 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.262+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 0x55e916f0e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.263+0000 7f3f92ffd640 1 -- 172.21.15.87:0/870322051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f07d50 msgr2=0x55e916f10bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.263+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f07d50 0x55e916f10bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.263+0000 7f3f92ffd640 1 -- 172.21.15.87:0/870322051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 msgr2=0x55e916f0df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.263+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f0df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.259 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.263+0000 7f3f92ffd640 1 -- 172.21.15.87:0/870322051 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e916b7f210 con 0x55e916f112d0 2024-04-22T13:37:26.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.265+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 0x55e916f0e4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x55e916f087f0 tx=0x7f3f840128c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d47f3b7b083a8d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.261 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.265+0000 7f3f72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:26.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.266+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84019070 con 0x55e916f112d0 2024-04-22T13:37:26.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.267+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f8800bd70 con 0x55e916f112d0 2024-04-22T13:37:26.262 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.267+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f840135b0 con 0x55e916f112d0 2024-04-22T13:37:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.267+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3f84025050 con 0x55e916f112d0 2024-04-22T13:37:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.268+0000 7f3f9cdf0740 10 client.26330 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.268+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e916c4b620 con 0x55e916f112d0 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8800bd70 con 0x55e916f112d0 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3f84015a20 con 0x55e916f112d0 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 10 client.26330 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 10 client.26330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 20 client.26330 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f21810 0x55e916f23c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 4 client.26330 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.269+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e916e80880 con 0x55e916f21810 2024-04-22T13:37:26.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.270+0000 7f3f937fe640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f21810 0x55e916f23c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.275+0000 7f3f72ffd640 10 client.26330 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.275+0000 7f3f937fe640 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f21810 0x55e916f23c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2ce4341bc0e0246 server_cookie=da0d1279383ef3c7 in_seq=0 out_seq=0 2024-04-22T13:37:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.284+0000 7f3f72ffd640 1 -- 172.21.15.87:0/870322051 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e916e80880 con 0x55e916f21810 2024-04-22T13:37:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.284+0000 7f3f72ffd640 10 client.26330 handle_command_reply: tid=0 2024-04-22T13:37:26.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.284+0000 7f3f9cdf0740 1 client.26330 shutdown 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.289+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 msgr2=0x55e916f0e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.289+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 0x55e916f0e4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x55e916f087f0 tx=0x7f3f840128c0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 shutdown_connections 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f21810 0x55e916f23c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f112d0 0x55e916f0e4c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f0df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/870322051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f07d50 0x55e916f10bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 >> 172.21.15.87:0/870322051 conn(0x55e916ecec90 msgr2=0x55e916f04790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f72ffd640 0 client.26330 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:26.286 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.290+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 shutdown_connections 2024-04-22T13:37:26.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.291+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/870322051 wait complete. 2024-04-22T13:37:26.288 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.292+0000 7f3f9cdf0740 20 client.26330 trim_cache size 0 max 16384 2024-04-22T13:37:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 Processor -- start 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 -- start start 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f07d50 0x55e916f1cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f10bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80010f50 con 0x55e916f08ac0 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8000e820 con 0x55e916f07d50 2024-04-22T13:37:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.331+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8000c670 con 0x55e916f112d0 2024-04-22T13:37:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.332+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.332+0000 7f3f937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57582/0 (socket says 172.21.15.87:57582) 2024-04-22T13:37:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.332+0000 7f3f937fe640 1 -- 172.21.15.87:0/484612355 learned_addr learned my addr 172.21.15.87:0/484612355 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.333+0000 7f3f937fe640 1 -- 172.21.15.87:0/484612355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 msgr2=0x55e916f10bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.333+0000 7f3f937fe640 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f10bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.333+0000 7f3f937fe640 1 -- 172.21.15.87:0/484612355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f07d50 msgr2=0x55e916f1cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.333+0000 7f3f937fe640 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f07d50 0x55e916f1cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.333+0000 7f3f937fe640 1 -- 172.21.15.87:0/484612355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e916efcef0 con 0x55e916f08ac0 2024-04-22T13:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.339+0000 7f3f937fe640 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55e916f0d7e0 tx=0x7f3f88009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b146923c5b3dfe05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.340+0000 7f3f797fa640 1 -- 172.21.15.87:0/484612355 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88015040 con 0x55e916f08ac0 2024-04-22T13:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.340+0000 7f3f797fa640 1 -- 172.21.15.87:0/484612355 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8802b070 con 0x55e916f08ac0 2024-04-22T13:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.341+0000 7f3f797fa640 1 -- 172.21.15.87:0/484612355 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88004400 con 0x55e916f08ac0 2024-04-22T13:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.342+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/484612355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 msgr2=0x55e916f106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.342+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55e916f0d7e0 tx=0x7f3f88009060 comp rx=0 tx=0).stop 2024-04-22T13:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.345+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/484612355 shutdown_connections 2024-04-22T13:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.345+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f10bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.345+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f08ac0 0x55e916f106b0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.345+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/484612355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f07d50 0x55e916f1cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.345+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/484612355 >> 172.21.15.87:0/484612355 conn(0x55e916ecec90 msgr2=0x55e916f0c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.349+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/484612355 shutdown_connections 2024-04-22T13:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.352+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/484612355 wait complete. 2024-04-22T13:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.352+0000 7f3f9cdf0740 1 -- start start 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 Processor -- start 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f0b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f0b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8000cce0 con 0x55e916f07d50 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8000e150 con 0x55e916f08ac0 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8000d3b0 con 0x55e916f112d0 2024-04-22T13:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.353+0000 7f3f9cdf0740 10 client.0 _ll_register_callbacks cb 0x7f3f8400c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.356+0000 7f3f92ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57596/0 (socket says 172.21.15.87:57596) 2024-04-22T13:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 -- 172.21.15.87:0/1753634361 learned_addr learned my addr 172.21.15.87:0/1753634361 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 -- 172.21.15.87:0/1753634361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 msgr2=0x55e916f0b740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f0b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 -- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 msgr2=0x55e916f0b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f0b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.357+0000 7f3f92ffd640 1 -- 172.21.15.87:0/1753634361 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e916f0bc80 con 0x55e916f07d50 2024-04-22T13:37:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.359+0000 7f3f92ffd640 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006ee0 tx=0x7f3f8401d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5916412f23cb8836 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.359+0000 7f3f72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.361+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8402e060 con 0x55e916f07d50 2024-04-22T13:37:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.361+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f8802b070 con 0x55e916f07d50 2024-04-22T13:37:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.362+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f8401d9b0 con 0x55e916f07d50 2024-04-22T13:37:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.362+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3f84019050 con 0x55e916f07d50 2024-04-22T13:37:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.362+0000 7f3f9cdf0740 10 client.17241 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.362+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e916f125f0 con 0x55e916f07d50 2024-04-22T13:37:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8802b070 con 0x55e916f07d50 2024-04-22T13:37:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3f84031020 con 0x55e916f07d50 2024-04-22T13:37:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 10 client.17241 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 20 client.17241 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f882a0 0x55e916f8a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 4 client.17241 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:37:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.363+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e916e0c3d0 con 0x55e916f882a0 2024-04-22T13:37:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.368+0000 7f3f937fe640 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f882a0 0x55e916f8a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.372+0000 7f3f937fe640 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f882a0 0x55e916f8a6c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31241ea530200066 server_cookie=2b3df52d345f95a9 in_seq=0 out_seq=0 2024-04-22T13:37:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:26.372+0000 7f3f72ffd640 10 client.17241 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:26 smithi120 ceph-mon[31101]: pgmap v171: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 583 MiB/s wr, 4.67k op/s 2024-04-22T13:37:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:26 smithi132 ceph-mon[31531]: pgmap v171: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 583 MiB/s wr, 4.67k op/s 2024-04-22T13:37:26.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:26 smithi087 ceph-mon[27999]: pgmap v171: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 583 MiB/s wr, 4.67k op/s 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f72ffd640 1 -- 172.21.15.87:0/1753634361 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e916e0c3d0 con 0x55e916f882a0 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f72ffd640 10 client.17241 handle_command_reply: tid=0 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 client.17241 shutdown 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 msgr2=0x55e916f196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006ee0 tx=0x7f3f8401d5e0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 shutdown_connections 2024-04-22T13:37:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e916f882a0 0x55e916f8a6c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e916f112d0 0x55e916f0b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e916f08ac0 0x55e916f0b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 --2- 172.21.15.87:0/1753634361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e916f07d50 0x55e916f196b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 >> 172.21.15.87:0/1753634361 conn(0x55e916ecec90 msgr2=0x55e916f01250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f72ffd640 0 client.17241 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 shutdown_connections 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 1 -- 172.21.15.87:0/1753634361 wait complete. 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.375+0000 7f3f9cdf0740 20 client.17241 trim_cache size 0 max 16384 2024-04-22T13:37:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:37:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:37:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:37:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:37:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 msgr2=0x7f3f7c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:28.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 0x7f3f7c0a6320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0bac90 tx=0x7f3f8c011f90 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 msgr2=0x7f3f941afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c011040 tx=0x7f3f8c03b990 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 shutdown_connections 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f7c0a3e60 0x7f3f7c0a6320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9410aac0 0x7f3f941afa50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f9410a100 0x7f3f941b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 --2- 172.21.15.87:0/4051198330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f941097e0 0x7f3f941b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 >> 172.21.15.87:0/4051198330 conn(0x7f3f9406ce90 msgr2=0x7f3f9410d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 shutdown_connections 2024-04-22T13:37:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:28.377+0000 7f3f98f9f640 1 -- 172.21.15.87:0/4051198330 wait complete. 2024-04-22T13:37:28.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:28 smithi120 ceph-mon[31101]: pgmap v172: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 795 MiB/s wr, 5.47k op/s 2024-04-22T13:37:28.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:28 smithi132 ceph-mon[31531]: pgmap v172: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 795 MiB/s wr, 5.47k op/s 2024-04-22T13:37:28.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:28 smithi087 ceph-mon[27999]: pgmap v172: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 795 MiB/s wr, 5.47k op/s 2024-04-22T13:37:30.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:30 smithi087 ceph-mon[27999]: pgmap v173: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 733 MiB/s wr, 4.22k op/s 2024-04-22T13:37:30.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:30 smithi120 ceph-mon[31101]: pgmap v173: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 733 MiB/s wr, 4.22k op/s 2024-04-22T13:37:30.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:30 smithi132 ceph-mon[31531]: pgmap v173: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 733 MiB/s wr, 4.22k op/s 2024-04-22T13:37:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:32 smithi120 ceph-mon[31101]: pgmap v174: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 702 MiB/s wr, 3.75k op/s 2024-04-22T13:37:32.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:32 smithi132 ceph-mon[31531]: pgmap v174: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 702 MiB/s wr, 3.75k op/s 2024-04-22T13:37:32.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:32 smithi087 ceph-mon[27999]: pgmap v174: 97 pgs: 97 active+clean; 42 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 702 MiB/s wr, 3.75k op/s 2024-04-22T13:37:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:34 smithi120 ceph-mon[31101]: pgmap v175: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 899 MiB/s wr, 4.81k op/s 2024-04-22T13:37:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:34 smithi132 ceph-mon[31531]: pgmap v175: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 899 MiB/s wr, 4.81k op/s 2024-04-22T13:37:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:34 smithi087 ceph-mon[27999]: pgmap v175: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 899 MiB/s wr, 4.81k op/s 2024-04-22T13:37:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:36 smithi120 ceph-mon[31101]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 4.00k op/s 2024-04-22T13:37:36.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:36 smithi132 ceph-mon[31531]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 4.00k op/s 2024-04-22T13:37:36.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:36 smithi087 ceph-mon[27999]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 4.00k op/s 2024-04-22T13:37:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:38 smithi120 ceph-mon[31101]: pgmap v177: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 939 MiB/s wr, 4.45k op/s 2024-04-22T13:37:38.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:38 smithi132 ceph-mon[31531]: pgmap v177: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 939 MiB/s wr, 4.45k op/s 2024-04-22T13:37:38.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:38 smithi087 ceph-mon[27999]: pgmap v177: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 939 MiB/s wr, 4.45k op/s 2024-04-22T13:37:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:40 smithi120 ceph-mon[31101]: pgmap v178: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 802 MiB/s wr, 3.61k op/s 2024-04-22T13:37:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:40 smithi132 ceph-mon[31531]: pgmap v178: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 802 MiB/s wr, 3.61k op/s 2024-04-22T13:37:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:40 smithi087 ceph-mon[27999]: pgmap v178: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 802 MiB/s wr, 3.61k op/s 2024-04-22T13:37:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.640+0000 7f51f6238640 1 -- 172.21.15.87:0/353703494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f010ec20 msgr2=0x7f51f011d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.640+0000 7f51f6238640 1 --2- 172.21.15.87:0/353703494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f010ec20 0x7f51f011d740 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009900 tx=0x7f51e002bd10 comp rx=0 tx=0).stop 2024-04-22T13:37:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.645+0000 7f51f6238640 1 -- 172.21.15.87:0/353703494 shutdown_connections 2024-04-22T13:37:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.645+0000 7f51f6238640 1 --2- 172.21.15.87:0/353703494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f010ec20 0x7f51f011d740 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.645+0000 7f51f6238640 1 --2- 172.21.15.87:0/353703494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f01109f0 0x7f51f010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.645+0000 7f51f6238640 1 --2- 172.21.15.87:0/353703494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f0110610 0x7f51f010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.645+0000 7f51f6238640 1 -- 172.21.15.87:0/353703494 >> 172.21.15.87:0/353703494 conn(0x7f51f006ce90 msgr2=0x7f51f006d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.652+0000 7f51f6238640 1 -- 172.21.15.87:0/353703494 shutdown_connections 2024-04-22T13:37:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.658+0000 7f51f6238640 1 -- 172.21.15.87:0/353703494 wait complete. 2024-04-22T13:37:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.659+0000 7f51f6238640 1 Processor -- start 2024-04-22T13:37:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 -- start start 2024-04-22T13:37:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 0x7f51f011d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f0110610 0x7f51f0117400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0111810 con 0x7f51f0110610 2024-04-22T13:37:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0111690 con 0x7f51f01109f0 2024-04-22T13:37:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.666+0000 7f51f6238640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0111990 con 0x7f51f010ec20 2024-04-22T13:37:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.668+0000 7f51f5a37640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.668+0000 7f51f5a37640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60482/0 (socket says 172.21.15.87:60482) 2024-04-22T13:37:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.668+0000 7f51f5a37640 1 -- 172.21.15.87:0/4272044596 learned_addr learned my addr 172.21.15.87:0/4272044596 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.671+0000 7f51f5236640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 0x7f51f011d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f4a35640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f0110610 0x7f51f0117400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f5a37640 1 -- 172.21.15.87:0/4272044596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 msgr2=0x7f51f011d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f5a37640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 0x7f51f011d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f5a37640 1 -- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f0110610 msgr2=0x7f51f0117400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f5a37640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f0110610 0x7f51f0117400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.676+0000 7f51f5a37640 1 -- 172.21.15.87:0/4272044596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f0118070 con 0x7f51f01109f0 2024-04-22T13:37:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.677+0000 7f51f5236640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 0x7f51f011d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:41.673 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.677+0000 7f51f5a37640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51e002c250 tx=0x7f51e002db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.673 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.677+0000 7f51e67fc640 1 -- 172.21.15.87:0/4272044596 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0043020 con 0x7f51f01109f0 2024-04-22T13:37:41.675 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.678+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f01bc430 con 0x7f51f01109f0 2024-04-22T13:37:41.675 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.678+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f01bc8f0 con 0x7f51f01109f0 2024-04-22T13:37:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.683+0000 7f51fa16a740 1 Processor -- start 2024-04-22T13:37:41.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 -- start start 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487b32d0 0x55fc487b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc4870e8d0 con 0x55fc487a9d50 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc486abe80 con 0x55fc487aaac0 2024-04-22T13:37:41.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.684+0000 7f51fa16a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc4870a890 con 0x55fc487b32d0 2024-04-22T13:37:41.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.685+0000 7f51e67fc640 1 -- 172.21.15.87:0/4272044596 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e00052b0 con 0x7f51f01109f0 2024-04-22T13:37:41.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.686+0000 7f51f4a35640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.687+0000 7f51f4a35640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60486/0 (socket says 172.21.15.87:60486) 2024-04-22T13:37:41.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.687+0000 7f51e67fc640 1 -- 172.21.15.87:0/4272044596 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0003b40 con 0x7f51f01109f0 2024-04-22T13:37:41.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.687+0000 7f51f4a35640 1 -- 172.21.15.87:0/3990867784 learned_addr learned my addr 172.21.15.87:0/3990867784 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:41.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.687+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487b32d0 0x55fc487b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.688+0000 7f51f4a35640 1 -- 172.21.15.87:0/3990867784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487b32d0 msgr2=0x55fc487b56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.688+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487b32d0 0x55fc487b56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.685 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.688+0000 7f51f4a35640 1 -- 172.21.15.87:0/3990867784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 msgr2=0x55fc487aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.685 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.689+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.685 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.689+0000 7f51f4a35640 1 -- 172.21.15.87:0/3990867784 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc486cd320 con 0x55fc487aaac0 2024-04-22T13:37:41.686 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.691+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00bf60 tx=0x7f51ec00b1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a1a001cb9bbafe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.691 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.693+0000 7f51e67fc640 1 -- 172.21.15.87:0/4272044596 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f51e0046020 con 0x7f51f01109f0 2024-04-22T13:37:41.691 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.693+0000 7f51db7fe640 1 -- 172.21.15.87:0/3990867784 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec014850 con 0x55fc487aaac0 2024-04-22T13:37:41.691 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.693+0000 7f51db7fe640 1 -- 172.21.15.87:0/3990867784 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ec010040 con 0x55fc487aaac0 2024-04-22T13:37:41.691 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.693+0000 7f51db7fe640 1 -- 172.21.15.87:0/3990867784 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec017b00 con 0x55fc487aaac0 2024-04-22T13:37:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.695+0000 7f51fa16a740 1 -- 172.21.15.87:0/3990867784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 msgr2=0x55fc487a9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.695+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00bf60 tx=0x7f51ec00b1a0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.696+0000 7f51e67fc640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 0x7f51dc0a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.692 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51e67fc640 1 -- 172.21.15.87:0/4272044596 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f51e00f27a0 con 0x7f51f01109f0 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 -- 172.21.15.87:0/3990867784 shutdown_connections 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487b32d0 0x55fc487b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3990867784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487aaac0 0x55fc487a9810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 -- 172.21.15.87:0/3990867784 >> 172.21.15.87:0/3990867784 conn(0x55fc48770c90 msgr2=0x55fc487a4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:41.693 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.697+0000 7f51fa16a740 1 -- 172.21.15.87:0/3990867784 shutdown_connections 2024-04-22T13:37:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.698+0000 7f51fa16a740 1 -- 172.21.15.87:0/3990867784 wait complete. 2024-04-22T13:37:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.699+0000 7f51fa16a740 1 -- start start 2024-04-22T13:37:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.699+0000 7f51fa16a740 1 Processor -- start 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487b32d0 0x55fc487b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc487b5fa0 con 0x55fc487a9d50 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc48696eb0 con 0x55fc487b32d0 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc487b6120 con 0x55fc487aaac0 2024-04-22T13:37:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51fa16a740 10 client.0 _ll_register_callbacks cb 0x55fc486c3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.700+0000 7f51f4a35640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 0x7f51dc0a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.701+0000 7f51f5236640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.701+0000 7f51f5236640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59210/0 (socket says 172.21.15.87:59210) 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.701+0000 7f51f5236640 1 -- 172.21.15.87:0/3746665293 learned_addr learned my addr 172.21.15.87:0/3746665293 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.701+0000 7f51f5a37640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487b32d0 0x55fc487b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 -- 172.21.15.87:0/3746665293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 msgr2=0x55fc487b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 -- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487b32d0 msgr2=0x55fc487b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487b32d0 0x55fc487b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 -- 172.21.15.87:0/3746665293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc48421210 con 0x55fc487a9d50 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f5236640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f51e800aef0 tx=0x7f51e800e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c12bc05d3a21c3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.702+0000 7f51f4a35640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 0x7f51dc0a6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51ec006e90 tx=0x7f51ec00bef0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.704+0000 7f51d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.704+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8018020 con 0x55fc487a9d50 2024-04-22T13:37:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.705+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f51e8014600 con 0x55fc487a9d50 2024-04-22T13:37:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.705+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51ec010040 con 0x55fc487a9d50 2024-04-22T13:37:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.706+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f51e800ee50 con 0x55fc487a9d50 2024-04-22T13:37:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.706+0000 7f51fa16a740 10 client.17247 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.706+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc484ed620 con 0x55fc487a9d50 2024-04-22T13:37:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.707+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51ec010040 con 0x55fc487a9d50 2024-04-22T13:37:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.707+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f51e801c9c0 con 0x55fc487a9d50 2024-04-22T13:37:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 10 client.17247 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 20 client.17247 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc487c3460 0x55fc487c5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 4 client.17247 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:41.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.708+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc48722880 con 0x55fc487c3460 2024-04-22T13:37:41.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.709+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc487c3460 0x55fc487c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.716+0000 7f51f4a35640 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc487c3460 0x55fc487c5870 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14d84c10b57306f3 server_cookie=6cce40d67ece395b in_seq=0 out_seq=0 2024-04-22T13:37:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.716+0000 7f51d8ff9640 10 client.17247 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.718+0000 7f51d8ff9640 1 -- 172.21.15.87:0/3746665293 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc48722880 con 0x55fc487c3460 2024-04-22T13:37:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51d8ff9640 10 client.17247 handle_command_reply: tid=0 2024-04-22T13:37:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 client.17247 shutdown 2024-04-22T13:37:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 msgr2=0x55fc487b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f51e800aef0 tx=0x7f51e800e490 comp rx=0 tx=0).stop 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 shutdown_connections 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc487c3460 0x55fc487c5870 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487b32d0 0x55fc487b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 --2- 172.21.15.87:0/3746665293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487a9d50 0x55fc487b2cf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 >> 172.21.15.87:0/3746665293 conn(0x55fc48770c90 msgr2=0x55fc487a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.719+0000 7f51d8ff9640 0 client.17247 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.722+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 shutdown_connections 2024-04-22T13:37:41.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.722+0000 7f51fa16a740 1 -- 172.21.15.87:0/3746665293 wait complete. 2024-04-22T13:37:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.722+0000 7f51fa16a740 20 client.17247 trim_cache size 0 max 16384 2024-04-22T13:37:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.771+0000 7f51fa16a740 1 Processor -- start 2024-04-22T13:37:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.771+0000 7f51fa16a740 1 -- start start 2024-04-22T13:37:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.771+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.771+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.771+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51f5a37640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51f5a37640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59212/0 (socket says 172.21.15.87:59212) 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51f5a37640 1 -- 172.21.15.87:0/1829209995 learned_addr learned my addr 172.21.15.87:0/1829209995 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4010f80 con 0x55fc487b32d0 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51d400e850 con 0x55fc487a9d50 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.772+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51d400c6a0 con 0x55fc487aaac0 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 -- 172.21.15.87:0/1829209995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 msgr2=0x55fc487b27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 -- 172.21.15.87:0/1829209995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 msgr2=0x55fc487bef30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 -- 172.21.15.87:0/1829209995 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc487a9020 con 0x55fc487b32d0 2024-04-22T13:37:41.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.773+0000 7f51f5a37640 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f51e00be5c0 tx=0x7f51e00df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d0a0910e7c5ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.784+0000 7f51daffd640 1 -- 172.21.15.87:0/1829209995 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00be8c0 con 0x55fc487b32d0 2024-04-22T13:37:41.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.784+0000 7f51daffd640 1 -- 172.21.15.87:0/1829209995 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e00c2c70 con 0x55fc487b32d0 2024-04-22T13:37:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.786+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 msgr2=0x55fc487b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.786+0000 7f51fa16a740 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f51e00be5c0 tx=0x7f51e00df040 comp rx=0 tx=0).stop 2024-04-22T13:37:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.794+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 shutdown_connections 2024-04-22T13:37:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.794+0000 7f51fa16a740 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b2cf0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.794+0000 7f51fa16a740 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.794+0000 7f51fa16a740 1 --2- 172.21.15.87:0/1829209995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.794+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 >> 172.21.15.87:0/1829209995 conn(0x55fc48770c90 msgr2=0x55fc487ab220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:41.792 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.795+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 shutdown_connections 2024-04-22T13:37:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.795+0000 7f51fa16a740 1 -- 172.21.15.87:0/1829209995 wait complete. 2024-04-22T13:37:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.796+0000 7f51fa16a740 1 -- start start 2024-04-22T13:37:41.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.796+0000 7f51fa16a740 1 Processor -- start 2024-04-22T13:37:41.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51d400cd10 con 0x55fc487b32d0 2024-04-22T13:37:41.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51d400e180 con 0x55fc487a9d50 2024-04-22T13:37:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51d400d3e0 con 0x55fc487aaac0 2024-04-22T13:37:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.797+0000 7f51fa16a740 10 client.0 _ll_register_callbacks cb 0x7f51e8009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.800+0000 7f51f5236640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.800+0000 7f51f4a35640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.798 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.801+0000 7f51f5236640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60498/0 (socket says 172.21.15.87:60498) 2024-04-22T13:37:41.798 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.801+0000 7f51f5236640 1 -- 172.21.15.87:0/2411923657 learned_addr learned my addr 172.21.15.87:0/2411923657 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:41.798 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.801+0000 7f51f5a37640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.798 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.802+0000 7f51f5a37640 1 -- 172.21.15.87:0/2411923657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 msgr2=0x55fc487b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.799 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.802+0000 7f51f5a37640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.799 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.802+0000 7f51f5a37640 1 -- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 msgr2=0x55fc487bb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:41.799 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.802+0000 7f51f5a37640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:41.799 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.802+0000 7f51f5a37640 1 -- 172.21.15.87:0/2411923657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc487b7a50 con 0x55fc487b32d0 2024-04-22T13:37:41.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.809+0000 7f51f5a37640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b7250 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f51e00bea80 tx=0x7f51e002bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a851e3200ee7c118 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:41.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.809+0000 7f51d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:41.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.810+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00c7530 con 0x55fc487b32d0 2024-04-22T13:37:41.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.811+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e00c2c70 con 0x55fc487b32d0 2024-04-22T13:37:41.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.811+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f51e00a90b0 con 0x55fc487b32d0 2024-04-22T13:37:41.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.812+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f51e00c70a0 con 0x55fc487b32d0 2024-04-22T13:37:41.810 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.812+0000 7f51fa16a740 10 client.17253 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:41.810 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.812+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc48829ad0 con 0x55fc487b32d0 2024-04-22T13:37:41.810 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e00c2c70 con 0x55fc487b32d0 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f51e00bd780 con 0x55fc487b32d0 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 10 client.17253 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 20 client.17253 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc4882a0f0 0x55fc4882c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:41.811 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 4 client.17253 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:37:41.812 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.814+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc486ae3d0 con 0x55fc4882a0f0 2024-04-22T13:37:41.814 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.818+0000 7f51f5236640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc4882a0f0 0x55fc4882c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:41.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.827+0000 7f51f5236640 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc4882a0f0 0x55fc4882c510 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c84613c7c087d14 server_cookie=d5c71b122e6bb69f in_seq=0 out_seq=0 2024-04-22T13:37:41.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:41.828+0000 7f51d8ff9640 10 client.17253 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:42.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:42 smithi120 ceph-mon[31101]: pgmap v179: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 731 MiB/s wr, 3.23k op/s 2024-04-22T13:37:42.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:42 smithi132 ceph-mon[31531]: pgmap v179: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 731 MiB/s wr, 3.23k op/s 2024-04-22T13:37:42.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:42 smithi087 ceph-mon[27999]: pgmap v179: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 731 MiB/s wr, 3.23k op/s 2024-04-22T13:37:44.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:44 smithi087 ceph-mon[27999]: pgmap v180: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 941 MiB/s wr, 3.82k op/s 2024-04-22T13:37:44.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:44 smithi120 ceph-mon[31101]: pgmap v180: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 941 MiB/s wr, 3.82k op/s 2024-04-22T13:37:44.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:44 smithi132 ceph-mon[31531]: pgmap v180: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 941 MiB/s wr, 3.82k op/s 2024-04-22T13:37:45.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:37:45.111 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:37:45.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 -- 172.21.15.87:0/548900987 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8076da0 msgr2=0x7f70d8112870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:45.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 --2- 172.21.15.87:0/548900987 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8076da0 0x7f70d8112870 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009900 tx=0x7f70cc02bc50 comp rx=0 tx=0).stop 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 -- 172.21.15.87:0/548900987 shutdown_connections 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 --2- 172.21.15.87:0/548900987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70d8112db0 0x7f70d81151a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 --2- 172.21.15.87:0/548900987 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8076da0 0x7f70d8112870 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 --2- 172.21.15.87:0/548900987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076480 0x7f70d8076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.518+0000 7f70dce3f640 1 -- 172.21.15.87:0/548900987 >> 172.21.15.87:0/548900987 conn(0x7f70d80ffc40 msgr2=0x7f70d8102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.519+0000 7f70dce3f640 1 -- 172.21.15.87:0/548900987 shutdown_connections 2024-04-22T13:37:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.519+0000 7f70dce3f640 1 -- 172.21.15.87:0/548900987 wait complete. 2024-04-22T13:37:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.520+0000 7f70dce3f640 1 Processor -- start 2024-04-22T13:37:45.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 -- start start 2024-04-22T13:37:45.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70d8076480 0x7f70d8073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8112db0 0x7f70d806ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8118890 con 0x7f70d8112db0 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8118710 con 0x7f70d8076da0 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.524+0000 7f70dce3f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8118a10 con 0x7f70d8076480 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.525+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.525+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60512/0 (socket says 172.21.15.87:60512) 2024-04-22T13:37:45.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.525+0000 7f70d5d9b640 1 -- 172.21.15.87:0/1195788128 learned_addr learned my addr 172.21.15.87:0/1195788128 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 -- 172.21.15.87:0/1195788128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70d8076480 msgr2=0x7f70d8073840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70d8076480 0x7f70d8073840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 -- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8112db0 msgr2=0x7f70d806ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8112db0 0x7f70d806ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 -- 172.21.15.87:0/1195788128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d806d4f0 con 0x7f70d8076da0 2024-04-22T13:37:45.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.526+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70cc005ab0 tx=0x7f70cc02c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.527+0000 7f70c77fe640 1 -- 172.21.15.87:0/1195788128 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc043020 con 0x7f70d8076da0 2024-04-22T13:37:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.527+0000 7f70c77fe640 1 -- 172.21.15.87:0/1195788128 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70cc0052b0 con 0x7f70d8076da0 2024-04-22T13:37:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.527+0000 7f70c77fe640 1 -- 172.21.15.87:0/1195788128 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc003c10 con 0x7f70d8076da0 2024-04-22T13:37:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.527+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d8071360 con 0x7f70d8076da0 2024-04-22T13:37:45.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.527+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d8071860 con 0x7f70d8076da0 2024-04-22T13:37:45.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.532+0000 7f70c77fe640 1 -- 172.21.15.87:0/1195788128 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70cc046020 con 0x7f70d8076da0 2024-04-22T13:37:45.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.534+0000 7f70c77fe640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 0x7f70c00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.534+0000 7f70c77fe640 1 -- 172.21.15.87:0/1195788128 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f70cc0f2990 con 0x7f70d8076da0 2024-04-22T13:37:45.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.539+0000 7f70d659c640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 0x7f70c00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.539+0000 7f70d659c640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 0x7f70c00a63e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b990 tx=0x7f70c800b370 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:45.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.546+0000 7f70dfa2d740 1 Processor -- start 2024-04-22T13:37:45.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.547+0000 7f70dfa2d740 1 -- start start 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.547+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9580 0x562e3efb9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.547+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efc2a80 0x562e3efc4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562e3ef1e1e0 con 0x562e3efb9ea0 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e3eebb390 con 0x562e3efb9580 2024-04-22T13:37:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e3ef1a1a0 con 0x562e3efc2a80 2024-04-22T13:37:45.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59254/0 (socket says 172.21.15.87:59254) 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.548+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3886909025 learned_addr learned my addr 172.21.15.87:0/3886909025 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.549+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3886909025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efc2a80 msgr2=0x562e3efc4e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.549+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efc2a80 0x562e3efc4e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.549+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3886909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9580 msgr2=0x562e3efb9960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.549+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9580 0x562e3efb9960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.549+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3886909025 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3ecfa7d0 con 0x562e3efb9ea0 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.550+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0baeb0 tx=0x7f70cc02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7b0f280858b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.551+0000 7f70c4ff9640 1 -- 172.21.15.87:0/3886909025 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0bea50 con 0x562e3efb9ea0 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.551+0000 7f70c4ff9640 1 -- 172.21.15.87:0/3886909025 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70cc0c2e00 con 0x562e3efb9ea0 2024-04-22T13:37:45.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.551+0000 7f70c4ff9640 1 -- 172.21.15.87:0/3886909025 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc005580 con 0x562e3efb9ea0 2024-04-22T13:37:45.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.552+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3886909025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 msgr2=0x562e3efc2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.552+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0baeb0 tx=0x7f70cc02c5f0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.553+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3886909025 shutdown_connections 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.553+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efc2a80 0x562e3efc4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.553+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efc2540 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.553+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3886909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9580 0x562e3efb9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.553+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3886909025 >> 172.21.15.87:0/3886909025 conn(0x562e3efb3cb0 msgr2=0x562e3efb4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:45.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.559+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3886909025 shutdown_connections 2024-04-22T13:37:45.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.565+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3886909025 wait complete. 2024-04-22T13:37:45.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.566+0000 7f70dfa2d740 1 -- start start 2024-04-22T13:37:45.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.567+0000 7f70dfa2d740 1 Processor -- start 2024-04-22T13:37:45.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.571+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9580 0x562e3efbfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562e3efc5630 con 0x562e3efb9580 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e3eed72f0 con 0x562e3efc2a80 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e3efc57b0 con 0x562e3efb9ea0 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70dfa2d740 10 client.0 _ll_register_callbacks cb 0x562e3eee4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.572+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.573+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60640/0 (socket says 172.21.15.87:60640) 2024-04-22T13:37:45.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.573+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3482905462 learned_addr learned my addr 172.21.15.87:0/3482905462 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:45.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.573+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d659c640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9580 0x562e3efbfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 msgr2=0x562e3efc0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3482905462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9580 msgr2=0x562e3efbfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9580 0x562e3efbfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 -- 172.21.15.87:0/3482905462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3ec303f0 con 0x562e3efb9ea0 2024-04-22T13:37:45.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.578+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70cc004650 tx=0x7f70cc02c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95fe7dbe15eaf852 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.582+0000 7f70bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.582+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0f0370 con 0x562e3efb9ea0 2024-04-22T13:37:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.582+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70cc0c2e00 con 0x562e3efb9ea0 2024-04-22T13:37:45.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.582+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f70cc0f0510 con 0x562e3efb9ea0 2024-04-22T13:37:45.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.584+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f70cc0ce050 con 0x562e3efb9ea0 2024-04-22T13:37:45.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.584+0000 7f70dfa2d740 10 client.26392 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:45.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.584+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562e3ebf00f0 con 0x562e3efb9ea0 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.587+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70cc0c2e00 con 0x562e3efb9ea0 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.588+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f70cc0ba7a0 con 0x562e3efb9ea0 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.588+0000 7f70dfa2d740 10 client.26392 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.588+0000 7f70dfa2d740 10 client.26392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.588+0000 7f70dfa2d740 20 client.26392 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.589+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3efd2450 0x562e3efd4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.589+0000 7f70dfa2d740 4 client.26392 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.589+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e3ef2bc00 con 0x562e3efd2450 2024-04-22T13:37:45.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.593+0000 7f70d659c640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3efd2450 0x562e3efd4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:45.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.595+0000 7f70d659c640 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3efd2450 0x562e3efd4870 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c44babc57a0186 server_cookie=33a25fa103dfbfcd in_seq=0 out_seq=0 2024-04-22T13:37:45.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:45.595+0000 7f70bdffb640 10 client.26392 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:46.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:46 smithi120 ceph-mon[31101]: pgmap v181: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 771 MiB/s wr, 2.62k op/s 2024-04-22T13:37:46.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:46 smithi132 ceph-mon[31531]: pgmap v181: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 771 MiB/s wr, 2.62k op/s 2024-04-22T13:37:46.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:46 smithi087 ceph-mon[27999]: pgmap v181: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 771 MiB/s wr, 2.62k op/s 2024-04-22T13:37:46.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f51d8ff9640 1 -- 172.21.15.87:0/2411923657 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc486ae3d0 con 0x55fc4882a0f0 2024-04-22T13:37:46.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f51d8ff9640 10 client.17253 handle_command_reply: tid=0 2024-04-22T13:37:46.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f51fa16a740 1 client.17253 shutdown 2024-04-22T13:37:46.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f70bdffb640 1 -- 172.21.15.87:0/3482905462 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e3ef2bc00 con 0x562e3efd2450 2024-04-22T13:37:46.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f70bdffb640 10 client.26392 handle_command_reply: tid=0 2024-04-22T13:37:46.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.886+0000 7f70dfa2d740 1 client.26392 shutdown 2024-04-22T13:37:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 msgr2=0x55fc487b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b7250 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f51e00bea80 tx=0x7f51e002bd10 comp rx=0 tx=0).stop 2024-04-22T13:37:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 shutdown_connections 2024-04-22T13:37:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc4882a0f0 0x55fc4882c510 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc487b32d0 0x55fc487b7250 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc487aaac0 0x55fc487b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 --2- 172.21.15.87:0/2411923657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc487a9d50 0x55fc487bb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 >> 172.21.15.87:0/2411923657 conn(0x55fc48770c90 msgr2=0x55fc487af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51d8ff9640 0 client.17253 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.891+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 shutdown_connections 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.892+0000 7f51fa16a740 1 -- 172.21.15.87:0/2411923657 wait complete. 2024-04-22T13:37:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.892+0000 7f51fa16a740 20 client.17253 trim_cache size 0 max 16384 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 msgr2=0x562e3efbffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70cc004650 tx=0x7f70cc02c470 comp rx=0 tx=0).stop 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 shutdown_connections 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3efd2450 0x562e3efd4870 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9ea0 0x562e3efbffe0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3482905462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9580 0x562e3efbfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 >> 172.21.15.87:0/3482905462 conn(0x562e3efb3cb0 msgr2=0x562e3efb5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.893+0000 7f70bdffb640 0 client.26392 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.894+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 shutdown_connections 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.894+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3482905462 wait complete. 2024-04-22T13:37:46.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:46.894+0000 7f70dfa2d740 20 client.26392 trim_cache size 0 max 16384 2024-04-22T13:37:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:37:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:37:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:37:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:37:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.897+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 msgr2=0x7f51dc0a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.897+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 0x7f51dc0a6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51ec006e90 tx=0x7f51ec00bef0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.897+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 msgr2=0x7f51f0117940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.897+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51e002c250 tx=0x7f51e002db00 comp rx=0 tx=0).stop 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 shutdown_connections 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f51dc0a3c70 0x7f51dc0a6130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01109f0 0x7f51f0117940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f51f0110610 0x7f51f0117400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 --2- 172.21.15.87:0/4272044596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f010ec20 0x7f51f011d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.903+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 >> 172.21.15.87:0/4272044596 conn(0x7f51f006ce90 msgr2=0x7f51f010d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.910+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 shutdown_connections 2024-04-22T13:37:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:37:46.910+0000 7f51f6238640 1 -- 172.21.15.87:0/4272044596 wait complete. 2024-04-22T13:37:47.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.005+0000 7f70dfa2d740 1 Processor -- start 2024-04-22T13:37:47.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.005+0000 7f70dfa2d740 1 -- start start 2024-04-22T13:37:47.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.006+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.006+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efbe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.006+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.007+0000 7f70d659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.007+0000 7f70d659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60650/0 (socket says 172.21.15.87:60650) 2024-04-22T13:37:47.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.007+0000 7f70d659c640 1 -- 172.21.15.87:0/2696636284 learned_addr learned my addr 172.21.15.87:0/2696636284 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:47.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.008+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efbe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.012+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8010e10 con 0x562e3efb9ea0 2024-04-22T13:37:47.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.012+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800e6e0 con 0x562e3efc2a80 2024-04-22T13:37:47.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.012+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800c5e0 con 0x562e3efb9580 2024-04-22T13:37:47.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.013+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.014+0000 7f70d5d9b640 1 -- 172.21.15.87:0/2696636284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 msgr2=0x562e3efce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:47.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.014+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.014+0000 7f70d5d9b640 1 -- 172.21.15.87:0/2696636284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 msgr2=0x562e3efc0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:47.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.014+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.014+0000 7f70d5d9b640 1 -- 172.21.15.87:0/2696636284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e3efb45f0 con 0x562e3efb9ea0 2024-04-22T13:37:47.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.015+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efbe9e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x562e3efc1070 tx=0x7f70cc02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a349b9f76d7bdb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:47.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.023+0000 7f70bffff640 1 -- 172.21.15.87:0/2696636284 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0be590 con 0x562e3efb9ea0 2024-04-22T13:37:47.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.024+0000 7f70bffff640 1 -- 172.21.15.87:0/2696636284 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70cc0bf150 con 0x562e3efb9ea0 2024-04-22T13:37:47.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.024+0000 7f70bffff640 1 -- 172.21.15.87:0/2696636284 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0c7230 con 0x562e3efb9ea0 2024-04-22T13:37:47.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.025+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 msgr2=0x562e3efbe9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.025+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efbe9e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x562e3efc1070 tx=0x7f70cc02c5f0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 shutdown_connections 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efc2a80 0x562e3efc0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efb9ea0 0x562e3efbe9e0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/2696636284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 >> 172.21.15.87:0/2696636284 conn(0x562e3efb3cb0 msgr2=0x562e3efb2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 shutdown_connections 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.026+0000 7f70dfa2d740 1 -- 172.21.15.87:0/2696636284 wait complete. 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.027+0000 7f70dfa2d740 1 -- start start 2024-04-22T13:37:47.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.027+0000 7f70dfa2d740 1 Processor -- start 2024-04-22T13:37:47.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.028+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efcadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.028+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 0x562e3efc6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.028+0000 7f70dfa2d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 0x562e3efc6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.028+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800cba0 con 0x562e3efc2a80 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800e010 con 0x562e3efb9ea0 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70d659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efcadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70dfa2d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800d270 con 0x562e3efb9580 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70d659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efcadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60654/0 (socket says 172.21.15.87:60654) 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70d659c640 1 -- 172.21.15.87:0/3658080526 learned_addr learned my addr 172.21.15.87:0/3658080526 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70dfa2d740 10 client.0 _ll_register_callbacks cb 0x7f70cc0d6070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 0x562e3efc6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.029+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 0x562e3efc6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 -- 172.21.15.87:0/3658080526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 msgr2=0x562e3efcadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:47.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efcadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 -- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 msgr2=0x562e3efc6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 0x562e3efc6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 -- 172.21.15.87:0/3658080526 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3efc6e50 con 0x562e3efc2a80 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d5d9b640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 0x562e3efc6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70d6d9d640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 0x562e3efc6670 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x562e3efb3670 tx=0x7f70d000bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d02f3354550c881 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.030+0000 7f70bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.031+0000 7f70bdffb640 1 -- 172.21.15.87:0/3658080526 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d001c060 con 0x562e3efc2a80 2024-04-22T13:37:47.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.031+0000 7f70bdffb640 1 -- 172.21.15.87:0/3658080526 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f70d0007950 con 0x562e3efc2a80 2024-04-22T13:37:47.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.038+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70cc0bf150 con 0x562e3efc2a80 2024-04-22T13:37:47.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.040+0000 7f70bdffb640 1 -- 172.21.15.87:0/3658080526 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f70d0007b50 con 0x562e3efc2a80 2024-04-22T13:37:47.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.045+0000 7f70dfa2d740 10 client.17268 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:47.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.045+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({fsmap=23}) -- 0x562e3f038990 con 0x562e3efc2a80 2024-04-22T13:37:47.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.050+0000 7f70bdffb640 1 -- 172.21.15.87:0/3658080526 <== mon.0 v2:172.21.15.87:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f70d0017410 con 0x562e3efc2a80 2024-04-22T13:37:47.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.051+0000 7f70dfa2d740 10 client.17268 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:47.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.051+0000 7f70dfa2d740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:47.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.051+0000 7f70dfa2d740 20 client.17268 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:47.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.051+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3f038f00 0x562e3f03b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.052+0000 7f70dfa2d740 4 client.17268 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:37:47.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.052+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f70cc0bf150 con 0x562e3f038f00 2024-04-22T13:37:47.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.057+0000 7f70d659c640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3f038f00 0x562e3f03b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.075+0000 7f70bdffb640 10 client.17268 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:47.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.075+0000 7f70d659c640 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3f038f00 0x562e3f03b2f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ddeeaff94d62b5 server_cookie=4b4e58925bf9e5b1 in_seq=0 out_seq=0 2024-04-22T13:37:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.080+0000 7f70bdffb640 1 -- 172.21.15.87:0/3658080526 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f70cc0bf150 con 0x562e3f038f00 2024-04-22T13:37:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.080+0000 7f70bdffb640 10 client.17268 handle_command_reply: tid=0 2024-04-22T13:37:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.080+0000 7f70dfa2d740 1 client.17268 shutdown 2024-04-22T13:37:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.080+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 msgr2=0x562e3efc6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.080+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 0x562e3efc6670 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x562e3efb3670 tx=0x7f70d000bc40 comp rx=0 tx=0).stop 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 shutdown_connections 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e3f038f00 0x562e3f03b2f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70bdffb640 0 client.17268 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e3efc2a80 0x562e3efc6670 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3efb9ea0 0x562e3efc6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 --2- 172.21.15.87:0/3658080526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e3efb9580 0x562e3efcadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 >> 172.21.15.87:0/3658080526 conn(0x562e3efb3cb0 msgr2=0x562e3efb4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 shutdown_connections 2024-04-22T13:37:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.081+0000 7f70dfa2d740 1 -- 172.21.15.87:0/3658080526 wait complete. 2024-04-22T13:37:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.082+0000 7f70dfa2d740 20 client.17268 trim_cache size 0 max 16384 2024-04-22T13:37:47.084 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:47.084 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:37:47.084 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:37:47.084 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 msgr2=0x7f70c00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 0x7f70c00a63e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b990 tx=0x7f70c800b370 comp rx=0 tx=0).stop 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 msgr2=0x7f70d806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70cc005ab0 tx=0x7f70cc02c9d0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 shutdown_connections 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f70c00a3f20 0x7f70c00a63e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70d8112db0 0x7f70d806ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70d8076da0 0x7f70d806c910 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 --2- 172.21.15.87:0/1195788128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70d8076480 0x7f70d8073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.088+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 >> 172.21.15.87:0/1195788128 conn(0x7f70d80ffc40 msgr2=0x7f70d81145b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:47.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.089+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 shutdown_connections 2024-04-22T13:37:47.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.089+0000 7f70dce3f640 1 -- 172.21.15.87:0/1195788128 wait complete. 2024-04-22T13:37:47.146 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:37:47.146 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:37:47.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.663+0000 7f88f3fff640 1 -- 172.21.15.87:0/2233225070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 msgr2=0x7f88f41108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.663+0000 7f88f3fff640 1 --2- 172.21.15.87:0/2233225070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 0x7f88f41108b0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009ca0 tx=0x7f88ec02c120 comp rx=0 tx=0).stop 2024-04-22T13:37:47.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.670+0000 7f88f3fff640 1 -- 172.21.15.87:0/2233225070 shutdown_connections 2024-04-22T13:37:47.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.670+0000 7f88f3fff640 1 --2- 172.21.15.87:0/2233225070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 0x7f88f411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.670+0000 7f88f3fff640 1 --2- 172.21.15.87:0/2233225070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.670+0000 7f88f3fff640 1 --2- 172.21.15.87:0/2233225070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 0x7f88f41108b0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.670+0000 7f88f3fff640 1 -- 172.21.15.87:0/2233225070 >> 172.21.15.87:0/2233225070 conn(0x7f88f406ce60 msgr2=0x7f88f406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:47.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.677+0000 7f88f3fff640 1 -- 172.21.15.87:0/2233225070 shutdown_connections 2024-04-22T13:37:47.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.679+0000 7f88f3fff640 1 -- 172.21.15.87:0/2233225070 wait complete. 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.680+0000 7f88f3fff640 1 Processor -- start 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.680+0000 7f88f3fff640 1 -- start start 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 0x7f88f41173e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 0x7f88f4117920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f88f41117d0 con 0x7f88f41104d0 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f88f4111650 con 0x7f88f410e030 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f3fff640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f88f4111950 con 0x7f88f410eb50 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60572/0 (socket says 172.21.15.87:60572) 2024-04-22T13:37:47.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.681+0000 7f88f2ffd640 1 -- 172.21.15.87:0/562937887 learned_addr learned my addr 172.21.15.87:0/562937887 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f27fc640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 0x7f88f41173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f2ffd640 1 -- 172.21.15.87:0/562937887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 msgr2=0x7f88f41173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f2ffd640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 0x7f88f41173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f2ffd640 1 -- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 msgr2=0x7f88f4117920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f2ffd640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 0x7f88f4117920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:47.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.682+0000 7f88f2ffd640 1 -- 172.21.15.87:0/562937887 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f4118050 con 0x7f88f410e030 2024-04-22T13:37:47.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.691+0000 7f88f2ffd640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009dc0 tx=0x7f88ec034cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:47.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.700+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec041020 con 0x7f88f410e030 2024-04-22T13:37:47.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.700+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ec034e40 con 0x7f88f410e030 2024-04-22T13:37:47.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.700+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec039040 con 0x7f88f410e030 2024-04-22T13:37:47.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.700+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f41265b0 con 0x7f88f410e030 2024-04-22T13:37:47.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.700+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f4126a50 con 0x7f88f410e030 2024-04-22T13:37:47.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.703+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f4118220 con 0x7f88f410e030 2024-04-22T13:37:47.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.714+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f88ec044020 con 0x7f88f410e030 2024-04-22T13:37:47.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.715+0000 7f88dbfff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 0x7f88dc0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:47.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.716+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f88ec0f28f0 con 0x7f88f410e030 2024-04-22T13:37:47.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.717+0000 7f88f27fc640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 0x7f88dc0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:47.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.719+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88ec04b020 con 0x7f88f410e030 2024-04-22T13:37:47.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:47.721+0000 7f88f27fc640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 0x7f88dc0a63e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88f4118ba0 tx=0x7f88e400a2d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.406+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88f4068c90 con 0x7f88f410e030 2024-04-22T13:37:48.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.409+0000 7f88dbfff640 1 -- 172.21.15.87:0/562937887 <== mon.2 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f88ec0bf1b0 con 0x7f88f410e030 2024-04-22T13:37:48.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:37:48.407 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:37:48.407 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:37:48.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 msgr2=0x7f88dc0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 0x7f88dc0a63e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88f4118ba0 tx=0x7f88e400a2d0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 msgr2=0x7f88f411d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009dc0 tx=0x7f88ec034cd0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 shutdown_connections 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88dc0a3f20 0x7f88dc0a63e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88f41104d0 0x7f88f4117920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88f410eb50 0x7f88f41173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 --2- 172.21.15.87:0/562937887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88f410e030 0x7f88f411d370 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 >> 172.21.15.87:0/562937887 conn(0x7f88f406ce60 msgr2=0x7f88f40715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 shutdown_connections 2024-04-22T13:37:48.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.441+0000 7f88f3fff640 1 -- 172.21.15.87:0/562937887 wait complete. 2024-04-22T13:37:48.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:37:48.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:48 smithi087 ceph-mon[27999]: pgmap v182: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 992 MiB/s wr, 2.88k op/s 2024-04-22T13:37:48.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:48 smithi120 ceph-mon[31101]: pgmap v182: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 992 MiB/s wr, 2.88k op/s 2024-04-22T13:37:48.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:48 smithi132 ceph-mon[31531]: pgmap v182: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 992 MiB/s wr, 2.88k op/s 2024-04-22T13:37:48.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.914+0000 7f3f8034b640 1 -- 172.21.15.87:0/3123371659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 msgr2=0x7f3f7810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.914+0000 7f3f8034b640 1 --2- 172.21.15.87:0/3123371659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 0x7f3f7810e290 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009cd0 tx=0x7f3f7002c120 comp rx=0 tx=0).stop 2024-04-22T13:37:48.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.918+0000 7f3f8034b640 1 -- 172.21.15.87:0/3123371659 shutdown_connections 2024-04-22T13:37:48.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.918+0000 7f3f8034b640 1 --2- 172.21.15.87:0/3123371659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f7811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.918+0000 7f3f8034b640 1 --2- 172.21.15.87:0/3123371659 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f7810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.918+0000 7f3f8034b640 1 --2- 172.21.15.87:0/3123371659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 0x7f3f7810e290 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.918+0000 7f3f8034b640 1 -- 172.21.15.87:0/3123371659 >> 172.21.15.87:0/3123371659 conn(0x7f3f7806d060 msgr2=0x7f3f7806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:48.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.921+0000 7f3f8034b640 1 -- 172.21.15.87:0/3123371659 shutdown_connections 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.923+0000 7f3f8034b640 1 -- 172.21.15.87:0/3123371659 wait complete. 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 Processor -- start 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 -- start start 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f78118df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 0x7f3f78111e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f781123b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781210d0 con 0x7f3f781107a0 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78120f50 con 0x7f3f7811b730 2024-04-22T13:37:48.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.924+0000 7f3f8034b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78121250 con 0x7f3f7810e7d0 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.925+0000 7f3f7e0e7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f78118df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.925+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f781123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.925+0000 7f3f7e0e7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f78118df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60704/0 (socket says 172.21.15.87:60704) 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.925+0000 7f3f7e0e7640 1 -- 172.21.15.87:0/1430796599 learned_addr learned my addr 172.21.15.87:0/1430796599 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.926+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1430796599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 msgr2=0x7f3f78118df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.926+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f78118df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.926+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 msgr2=0x7f3f78111e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.926+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 0x7f3f78111e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.926+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1430796599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f78112ae0 con 0x7f3f7811b730 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.927+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f781123b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800b3e0 tx=0x7f3f6800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.928+0000 7f3f6f7fe640 1 -- 172.21.15.87:0/1430796599 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68015020 con 0x7f3f7811b730 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.929+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f78116910 con 0x7f3f7811b730 2024-04-22T13:37:48.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.929+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f78116e60 con 0x7f3f7811b730 2024-04-22T13:37:48.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.929+0000 7f3f6f7fe640 1 -- 172.21.15.87:0/1430796599 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f68011ce0 con 0x7f3f7811b730 2024-04-22T13:37:48.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.932+0000 7f3f6f7fe640 1 -- 172.21.15.87:0/1430796599 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68010e50 con 0x7f3f7811b730 2024-04-22T13:37:48.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.950+0000 7f3f6f7fe640 1 -- 172.21.15.87:0/1430796599 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3f68018020 con 0x7f3f7811b730 2024-04-22T13:37:48.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.952+0000 7f3f6f7fe640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 0x7f3f640a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.952+0000 7f3f6f7fe640 1 -- 172.21.15.87:0/1430796599 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f680cfd30 con 0x7f3f7811b730 2024-04-22T13:37:48.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.953+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 0x7f3f640a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.955+0000 7f3f83014740 1 Processor -- start 2024-04-22T13:37:48.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.955+0000 7f3f83014740 1 -- start start 2024-04-22T13:37:48.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.956+0000 7f3f83014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e563a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.956+0000 7f3f83014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 0x5649e5643430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.956+0000 7f3f83014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 0x5649e5645e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f83014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5649e559f1f0 con 0x5649e5643a00 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f83014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5649e553c4b0 con 0x5649e563a4d0 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f83014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5649e559b1b0 con 0x5649e563ae80 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 0x5649e5645e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 0x5649e5645e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59310/0 (socket says 172.21.15.87:59310) 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/3051354368 learned_addr learned my addr 172.21.15.87:0/3051354368 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:48.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.957+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 0x5649e5643430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.958+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/3051354368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 msgr2=0x5649e563a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.958+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e563a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.958+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/3051354368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 msgr2=0x5649e5645e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.958+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 0x5649e5645e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.958+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/3051354368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649e552e1b0 con 0x5649e563ae80 2024-04-22T13:37:48.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.959+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 0x5649e5643430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3f781136c0 tx=0x7f3f74009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ff975e4457da95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:48.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.961+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 0x7f3f640a6580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000f80 tx=0x7f3f70009a10 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:48.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.969+0000 7f3f6cbfe640 1 -- 172.21.15.87:0/3051354368 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74015020 con 0x5649e563ae80 2024-04-22T13:37:48.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.969+0000 7f3f6cbfe640 1 -- 172.21.15.87:0/3051354368 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7400bd70 con 0x5649e563ae80 2024-04-22T13:37:48.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.971+0000 7f3f83014740 1 -- 172.21.15.87:0/3051354368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 msgr2=0x5649e5643430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:48.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.971+0000 7f3f83014740 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 0x5649e5643430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3f781136c0 tx=0x7f3f74009b80 comp rx=0 tx=0).stop 2024-04-22T13:37:48.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.973+0000 7f3f83014740 1 -- 172.21.15.87:0/3051354368 shutdown_connections 2024-04-22T13:37:48.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.973+0000 7f3f83014740 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e5643a00 0x5649e5645e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.973+0000 7f3f83014740 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563ae80 0x5649e5643430 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.973+0000 7f3f83014740 1 --2- 172.21.15.87:0/3051354368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e563a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.973+0000 7f3f83014740 1 -- 172.21.15.87:0/3051354368 >> 172.21.15.87:0/3051354368 conn(0x5649e5634ba0 msgr2=0x5649e5634f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:48.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.978+0000 7f3f83014740 1 -- 172.21.15.87:0/3051354368 shutdown_connections 2024-04-22T13:37:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.981+0000 7f3f83014740 1 -- 172.21.15.87:0/3051354368 wait complete. 2024-04-22T13:37:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.981+0000 7f3f83014740 1 -- start start 2024-04-22T13:37:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 Processor -- start 2024-04-22T13:37:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563a4d0 0x5649e56431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5640530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 0x5649e5640a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5649e56469c0 con 0x5649e563ae80 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5649e5558500 con 0x5649e5643a00 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5649e5646b40 con 0x5649e563a4d0 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.982+0000 7f3f83014740 10 client.0 _ll_register_callbacks cb 0x5649e5526b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.984+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 0x5649e5640a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.984+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 0x5649e5640a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:60626/0 (socket says 172.21.15.87:60626) 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.984+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/2211440343 learned_addr learned my addr 172.21.15.87:0/2211440343 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.985+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5640530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:48.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.988+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563a4d0 msgr2=0x5649e56431a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:48.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.988+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563a4d0 0x5649e56431a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.988+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/2211440343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 msgr2=0x5649e5640a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:48.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.988+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 0x5649e5640a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:48.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.988+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/2211440343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649e52b13f0 con 0x5649e563ae80 2024-04-22T13:37:48.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.990+0000 7f3f7d8e6640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5640530 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5649e5635c10 tx=0x7f3f740118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94d8c1abf1f1c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:48.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.990+0000 7f3f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:48.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.992+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74020070 con 0x5649e563ae80 2024-04-22T13:37:48.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.992+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7400bd70 con 0x5649e563ae80 2024-04-22T13:37:48.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.992+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f7400b8c0 con 0x5649e563ae80 2024-04-22T13:37:48.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:48.995+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3f7400bb00 con 0x5649e563ae80 2024-04-22T13:37:48.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.000+0000 7f3f83014740 10 client.17280 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:48.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.000+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5649e537b7d0 con 0x5649e563ae80 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7400bd70 con 0x5649e563ae80 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3f7402e620 con 0x5649e563ae80 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 10 client.17280 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 10 client.17280 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 20 client.17280 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e5653f10 0x5649e5656390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 4 client.17280 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.002+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649e553e930 con 0x5649e5653f10 2024-04-22T13:37:49.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.008+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e5653f10 0x5649e5656390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.015+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e5653f10 0x5649e5656390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d0584c7168b8c20 server_cookie=81d14d8ea132a765 in_seq=0 out_seq=0 2024-04-22T13:37:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.016+0000 7f3f61ffb640 10 client.17280 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:49.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.017+0000 7f3f61ffb640 1 -- 172.21.15.87:0/2211440343 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649e553e930 con 0x5649e5653f10 2024-04-22T13:37:49.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.017+0000 7f3f61ffb640 10 client.17280 handle_command_reply: tid=0 2024-04-22T13:37:49.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.017+0000 7f3f83014740 1 client.17280 shutdown 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 msgr2=0x5649e5640530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5640530 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5649e5635c10 tx=0x7f3f740118a0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 shutdown_connections 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e5653f10 0x5649e5656390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e5643a00 0x5649e5640a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5640530 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 --2- 172.21.15.87:0/2211440343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e563a4d0 0x5649e56431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 >> 172.21.15.87:0/2211440343 conn(0x5649e5634ba0 msgr2=0x5649e56368f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f61ffb640 0 client.17280 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.018+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 shutdown_connections 2024-04-22T13:37:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.019+0000 7f3f83014740 1 -- 172.21.15.87:0/2211440343 wait complete. 2024-04-22T13:37:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.019+0000 7f3f83014740 20 client.17280 trim_cache size 0 max 16384 2024-04-22T13:37:49.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.062+0000 7f3f83014740 1 Processor -- start 2024-04-22T13:37:49.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.062+0000 7f3f83014740 1 -- start start 2024-04-22T13:37:49.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563a4d0 0x5649e564f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563ae80 0x5649e5642c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c010fb0 con 0x5649e563a4d0 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c00e880 con 0x5649e563ae80 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.065+0000 7f3f83014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c00c6d0 con 0x5649e5643a00 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:60726/0 (socket says 172.21.15.87:60726) 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/4110836809 learned_addr learned my addr 172.21.15.87:0/4110836809 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/4110836809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563ae80 msgr2=0x5649e5642c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563ae80 0x5649e5642c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/4110836809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563a4d0 msgr2=0x5649e564f960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563a4d0 0x5649e564f960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.066+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/4110836809 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649e56349d0 con 0x5649e5643a00 2024-04-22T13:37:49.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.067+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f6809bd80 tx=0x7f3f680bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb73b7f9e69ae9bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.072+0000 7f3f63fff640 1 -- 172.21.15.87:0/4110836809 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680b4020 con 0x5649e5643a00 2024-04-22T13:37:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.072+0000 7f3f63fff640 1 -- 172.21.15.87:0/4110836809 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f680a4a60 con 0x5649e5643a00 2024-04-22T13:37:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.075+0000 7f3f83014740 1 -- 172.21.15.87:0/4110836809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 msgr2=0x5649e56431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.075+0000 7f3f83014740 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f6809bd80 tx=0x7f3f680bc040 comp rx=0 tx=0).stop 2024-04-22T13:37:49.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.079+0000 7f3f83014740 1 -- 172.21.15.87:0/4110836809 shutdown_connections 2024-04-22T13:37:49.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.079+0000 7f3f83014740 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56431a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.079+0000 7f3f83014740 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563ae80 0x5649e5642c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.079+0000 7f3f83014740 1 --2- 172.21.15.87:0/4110836809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563a4d0 0x5649e564f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.079+0000 7f3f83014740 1 -- 172.21.15.87:0/4110836809 >> 172.21.15.87:0/4110836809 conn(0x5649e5634ba0 msgr2=0x5649e56404f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.097+0000 7f3f83014740 1 -- 172.21.15.87:0/4110836809 shutdown_connections 2024-04-22T13:37:49.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.097+0000 7f3f83014740 1 -- 172.21.15.87:0/4110836809 wait complete. 2024-04-22T13:37:49.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.097+0000 7f3f83014740 1 -- start start 2024-04-22T13:37:49.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 Processor -- start 2024-04-22T13:37:49.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e5646e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5647370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c00cd40 con 0x5649e563ae80 2024-04-22T13:37:49.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c00e1b0 con 0x5649e563a4d0 2024-04-22T13:37:49.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c00d410 con 0x5649e5643a00 2024-04-22T13:37:49.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.098+0000 7f3f83014740 10 client.0 _ll_register_callbacks cb 0x7f3f7400bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:49.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.104+0000 7f3f7e8e8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.104+0000 7f3f7d8e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5647370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.104+0000 7f3f7d8e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5647370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59328/0 (socket says 172.21.15.87:59328) 2024-04-22T13:37:49.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.104+0000 7f3f7d8e6640 1 -- 172.21.15.87:0/1145094072 learned_addr learned my addr 172.21.15.87:0/1145094072 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.104+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e5646e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.115+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1145094072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 msgr2=0x5649e5646e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.115+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e5646e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.115+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1145094072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 msgr2=0x5649e5647370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.115+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5647370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.115+0000 7f3f7e8e8640 1 -- 172.21.15.87:0/1145094072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649e56b84c0 con 0x5649e5643a00 2024-04-22T13:37:49.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.122+0000 7f3f7e8e8640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56478b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3f6809baf0 tx=0x7f3f680a0320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7bfa7453002badb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.122+0000 7f3f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:49.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.124+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680a9060 con 0x5649e5643a00 2024-04-22T13:37:49.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.124+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f3f68011a30 con 0x5649e5643a00 2024-04-22T13:37:49.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.133+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f680a4a60 con 0x5649e5643a00 2024-04-22T13:37:49.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.143+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3f680b4070 con 0x5649e5643a00 2024-04-22T13:37:49.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.146+0000 7f3f83014740 10 client.26416 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:49.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.146+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5649e56b9720 con 0x5649e5643a00 2024-04-22T13:37:49.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.150+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f680a4a60 con 0x5649e5643a00 2024-04-22T13:37:49.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.150+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3f680bc0b0 con 0x5649e5643a00 2024-04-22T13:37:49.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 10 client.26416 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:49.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 10 client.26416 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:37:49.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 20 client.26416 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:49.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e56b9cf0 0x5649e56bc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 4 client.26416 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:49.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.152+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649e55acc10 con 0x5649e56b9cf0 2024-04-22T13:37:49.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.158+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e56b9cf0 0x5649e56bc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.181+0000 7f3f7e0e7640 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e56b9cf0 0x5649e56bc0e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f63bfb97df540b server_cookie=fffe1b728783a0f7 in_seq=0 out_seq=0 2024-04-22T13:37:49.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.181+0000 7f3f61ffb640 10 client.26416 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:49.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.186+0000 7f3f61ffb640 1 -- 172.21.15.87:0/1145094072 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649e55acc10 con 0x5649e56b9cf0 2024-04-22T13:37:49.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.186+0000 7f3f61ffb640 10 client.26416 handle_command_reply: tid=0 2024-04-22T13:37:49.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.187+0000 7f3f83014740 1 client.26416 shutdown 2024-04-22T13:37:49.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 msgr2=0x5649e56478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56478b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3f6809baf0 tx=0x7f3f680a0320 comp rx=0 tx=0).stop 2024-04-22T13:37:49.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 shutdown_connections 2024-04-22T13:37:49.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5649e56b9cf0 0x5649e56bc0e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649e5643a00 0x5649e56478b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5649e563ae80 0x5649e5647370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 --2- 172.21.15.87:0/1145094072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e563a4d0 0x5649e5646e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 >> 172.21.15.87:0/1145094072 conn(0x5649e5634ba0 msgr2=0x5649e563ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f61ffb640 0 client.26416 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 shutdown_connections 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.207+0000 7f3f83014740 1 -- 172.21.15.87:0/1145094072 wait complete. 2024-04-22T13:37:49.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.208+0000 7f3f83014740 20 client.26416 trim_cache size 0 max 16384 2024-04-22T13:37:49.212 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:49.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.221+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 msgr2=0x7f3f640a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.221+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 0x7f3f640a6580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000f80 tx=0x7f3f70009a10 comp rx=0 tx=0).stop 2024-04-22T13:37:49.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.221+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 msgr2=0x7f3f781123b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.221+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f781123b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800b3e0 tx=0x7f3f6800b8a0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 shutdown_connections 2024-04-22T13:37:49.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f640a40c0 0x7f3f640a6580 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f7811b730 0x7f3f781123b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f781107a0 0x7f3f78111e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 --2- 172.21.15.87:0/1430796599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f7810e7d0 0x7f3f78118df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.223+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 >> 172.21.15.87:0/1430796599 conn(0x7f3f7806d060 msgr2=0x7f3f7811a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.225+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 shutdown_connections 2024-04-22T13:37:49.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.225+0000 7f3f8034b640 1 -- 172.21.15.87:0/1430796599 wait complete. 2024-04-22T13:37:49.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:37:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:49 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/562937887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:49.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:49 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/562937887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:49.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:49 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/562937887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:37:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.754+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/1100176459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410e7d0 msgr2=0x7f8b3410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.754+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/1100176459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410e7d0 0x7f8b3410ec50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009cd0 tx=0x7f8b2c02c120 comp rx=0 tx=0).stop 2024-04-22T13:37:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.755+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/1100176459 shutdown_connections 2024-04-22T13:37:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.755+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/1100176459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b3411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.755+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/1100176459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410e7d0 0x7f8b3410ec50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.755+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/1100176459 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b341107a0 0x7f8b3410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.755+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/1100176459 >> 172.21.15.87:0/1100176459 conn(0x7f8b3406d060 msgr2=0x7f8b3406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.764+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/1100176459 shutdown_connections 2024-04-22T13:37:49.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.769+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/1100176459 wait complete. 2024-04-22T13:37:49.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.771+0000 7f8b3a7b2640 1 Processor -- start 2024-04-22T13:37:49.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.772+0000 7f8b3a7b2640 1 -- start start 2024-04-22T13:37:49.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b3410e7d0 0x7f8b341a76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 0x7f8b341a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b341abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341210d0 con 0x7f8b3410e7d0 2024-04-22T13:37:49.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34120f50 con 0x7f8b341107a0 2024-04-22T13:37:49.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.773+0000 7f8b3a7b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34121250 con 0x7f8b3411b730 2024-04-22T13:37:49.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.776+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 0x7f8b341a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.776+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 0x7f8b341a7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:55964/0 (socket says 172.21.15.87:55964) 2024-04-22T13:37:49.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.776+0000 7f8b337fe640 1 -- 172.21.15.87:0/564520024 learned_addr learned my addr 172.21.15.87:0/564520024 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.777+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b341abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.785+0000 7f8b38d4f640 1 -- 172.21.15.87:0/564520024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 msgr2=0x7f8b341a7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.785+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 0x7f8b341a7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.785+0000 7f8b38d4f640 1 -- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b3410e7d0 msgr2=0x7f8b341a76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.785+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b3410e7d0 0x7f8b341a76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.785+0000 7f8b38d4f640 1 -- 172.21.15.87:0/564520024 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b341ac730 con 0x7f8b3411b730 2024-04-22T13:37:49.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.792+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b341abfb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800b3e0 tx=0x7f8b2800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.794+0000 7f8b317fa640 1 -- 172.21.15.87:0/564520024 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28015020 con 0x7f8b3411b730 2024-04-22T13:37:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.794+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b341aca20 con 0x7f8b3411b730 2024-04-22T13:37:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.794+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b341b4260 con 0x7f8b3411b730 2024-04-22T13:37:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.795+0000 7f8b317fa640 1 -- 172.21.15.87:0/564520024 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b28011ce0 con 0x7f8b3411b730 2024-04-22T13:37:49.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.795+0000 7f8b317fa640 1 -- 172.21.15.87:0/564520024 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28010e00 con 0x7f8b3411b730 2024-04-22T13:37:49.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.801+0000 7f8b3d3a0740 1 Processor -- start 2024-04-22T13:37:49.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.802+0000 7f8b317fa640 1 -- 172.21.15.87:0/564520024 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b28018020 con 0x7f8b3411b730 2024-04-22T13:37:49.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 -- start start 2024-04-22T13:37:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b317fa640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 0x7f8b1c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd311430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd311a00 0x5561bd313e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5561bd26d1f0 con 0x5561bd3084d0 2024-04-22T13:37:49.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5561bd20a4b0 con 0x5561bd311a00 2024-04-22T13:37:49.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.804+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561bd2691b0 con 0x5561bd308e80 2024-04-22T13:37:49.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.805+0000 7f8b317fa640 1 -- 172.21.15.87:0/564520024 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f8b280cfd20 con 0x7f8b3411b730 2024-04-22T13:37:49.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.807+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.807+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41270/0 (socket says 172.21.15.87:41270) 2024-04-22T13:37:49.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.807+0000 7f8b33fff640 1 -- 172.21.15.87:0/1204315991 learned_addr learned my addr 172.21.15.87:0/1204315991 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.811+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd311a00 0x5561bd313e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.811+0000 7f8b337fe640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 0x7f8b1c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.811+0000 7f8b337fe640 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd311430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.812+0000 7f8b33fff640 1 -- 172.21.15.87:0/1204315991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 msgr2=0x5561bd311430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.812+0000 7f8b33fff640 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd311430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.812+0000 7f8b33fff640 1 -- 172.21.15.87:0/1204315991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd311a00 msgr2=0x5561bd313e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.812+0000 7f8b33fff640 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd311a00 0x5561bd313e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.812+0000 7f8b33fff640 1 -- 172.21.15.87:0/1204315991 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561bd1fc1b0 con 0x5561bd3084d0 2024-04-22T13:37:49.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.820+0000 7f8b33fff640 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8b240097c0 tx=0x7f8b24009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e25b1d461139dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.832+0000 7f8b1a7fc640 1 -- 172.21.15.87:0/1204315991 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24015020 con 0x5561bd3084d0 2024-04-22T13:37:49.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.834+0000 7f8b1a7fc640 1 -- 172.21.15.87:0/1204315991 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b24011420 con 0x5561bd3084d0 2024-04-22T13:37:49.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.834+0000 7f8b1a7fc640 1 -- 172.21.15.87:0/1204315991 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b240105c0 con 0x5561bd3084d0 2024-04-22T13:37:49.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.836+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/1204315991 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 msgr2=0x5561bd3088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.836+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8b240097c0 tx=0x7f8b24009c80 comp rx=0 tx=0).stop 2024-04-22T13:37:49.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.837+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/1204315991 shutdown_connections 2024-04-22T13:37:49.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.837+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd311a00 0x5561bd313e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.837+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd311430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.837+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/1204315991 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd3088b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.837+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/1204315991 >> 172.21.15.87:0/1204315991 conn(0x5561bd302ba0 msgr2=0x5561bd302f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.841+0000 7f8b337fe640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 0x7f8b1c0a6320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c000f80 tx=0x7f8b2c02e040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.846+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/1204315991 shutdown_connections 2024-04-22T13:37:49.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.846+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/1204315991 wait complete. 2024-04-22T13:37:49.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 -- start start 2024-04-22T13:37:49.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 Processor -- start 2024-04-22T13:37:49.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd308e80 0x5561bd30e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5561bd3149c0 con 0x5561bd308e80 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5561bd226500 con 0x5561bd3084d0 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561bd314b40 con 0x5561bd311a00 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.847+0000 7f8b3d3a0740 10 client.0 _ll_register_callbacks cb 0x5561bd1f4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.849+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.849+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd308e80 0x5561bd30e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.849+0000 7f8b38d4f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.849+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:55980/0 (socket says 172.21.15.87:55980) 2024-04-22T13:37:49.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.849+0000 7f8b33fff640 1 -- 172.21.15.87:0/3968834368 learned_addr learned my addr 172.21.15.87:0/3968834368 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.850+0000 7f8b33fff640 1 -- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 msgr2=0x5561bd30ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.850+0000 7f8b33fff640 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.850+0000 7f8b33fff640 1 -- 172.21.15.87:0/3968834368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd308e80 msgr2=0x5561bd30e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.850+0000 7f8b33fff640 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd308e80 0x5561bd30e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.850+0000 7f8b33fff640 1 -- 172.21.15.87:0/3968834368 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561bcf7f3f0 con 0x5561bd3084d0 2024-04-22T13:37:49.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.851+0000 7f8b33fff640 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009550 tx=0x7f8b24009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94501573da5a98f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.857+0000 7f8b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:49.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.857+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24015030 con 0x5561bd3084d0 2024-04-22T13:37:49.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.857+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b24011420 con 0x5561bd3084d0 2024-04-22T13:37:49.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.857+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f8b2400bad0 con 0x5561bd3084d0 2024-04-22T13:37:49.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.862+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8b24020070 con 0x5561bd3084d0 2024-04-22T13:37:49.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.862+0000 7f8b3d3a0740 10 client.26384 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:49.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.862+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561bd0497d0 con 0x5561bd3084d0 2024-04-22T13:37:49.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.862+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b24011420 con 0x5561bd3084d0 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.864+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8b2402c020 con 0x5561bd3084d0 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.864+0000 7f8b3d3a0740 10 client.26384 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.864+0000 7f8b3d3a0740 10 client.26384 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.864+0000 7f8b3d3a0740 20 client.26384 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.864+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd321da0 0x5561bd324240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.865+0000 7f8b3d3a0740 4 client.26384 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:49.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.865+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561bd20c930 con 0x5561bd321da0 2024-04-22T13:37:49.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.865+0000 7f8b337fe640 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd321da0 0x5561bd324240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.867+0000 7f8b137fe640 10 client.26384 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:49.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.867+0000 7f8b337fe640 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd321da0 0x5561bd324240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac5e46fb1195b38d server_cookie=f835a649b39dc021 in_seq=0 out_seq=0 2024-04-22T13:37:49.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.868+0000 7f8b137fe640 1 -- 172.21.15.87:0/3968834368 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561bd20c930 con 0x5561bd321da0 2024-04-22T13:37:49.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.868+0000 7f8b137fe640 10 client.26384 handle_command_reply: tid=0 2024-04-22T13:37:49.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.869+0000 7f8b3d3a0740 1 client.26384 shutdown 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.870+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 msgr2=0x5561bd311200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.870+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009550 tx=0x7f8b24009ed0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 shutdown_connections 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd321da0 0x5561bd324240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd308e80 0x5561bd30e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/3968834368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd311200 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 >> 172.21.15.87:0/3968834368 conn(0x5561bd302ba0 msgr2=0x5561bd3048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b137fe640 0 client.26384 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:49.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.871+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 shutdown_connections 2024-04-22T13:37:49.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.873+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/3968834368 wait complete. 2024-04-22T13:37:49.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.874+0000 7f8b3d3a0740 20 client.26384 trim_cache size 0 max 16384 2024-04-22T13:37:49.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.921+0000 7f8b3d3a0740 1 Processor -- start 2024-04-22T13:37:49.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.922+0000 7f8b3d3a0740 1 -- start start 2024-04-22T13:37:49.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.922+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd31d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.923+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd310cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.923+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 0x5561bd311200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.923+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b20010e20 con 0x5561bd311a00 2024-04-22T13:37:49.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.924+0000 7f8b38d4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 0x5561bd311200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.924+0000 7f8b38d4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 0x5561bd311200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41286/0 (socket says 172.21.15.87:41286) 2024-04-22T13:37:49.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.924+0000 7f8b38d4f640 1 -- 172.21.15.87:0/604894425 learned_addr learned my addr 172.21.15.87:0/604894425 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.924+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2000e6f0 con 0x5561bd3084d0 2024-04-22T13:37:49.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.924+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2000c5f0 con 0x5561bd308e80 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b337fe640 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd310cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd31d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 -- 172.21.15.87:0/604894425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 msgr2=0x5561bd310cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd310cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 -- 172.21.15.87:0/604894425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 msgr2=0x5561bd311200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 0x5561bd311200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.927+0000 7f8b33fff640 1 -- 172.21.15.87:0/604894425 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561bd308c20 con 0x5561bd3084d0 2024-04-22T13:37:49.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.928+0000 7f8b33fff640 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd31d8d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8b24001f70 tx=0x7f8b240091b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a506b7dc242f641 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.937+0000 7f8b19ffb640 1 -- 172.21.15.87:0/604894425 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24015070 con 0x5561bd3084d0 2024-04-22T13:37:49.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.937+0000 7f8b19ffb640 1 -- 172.21.15.87:0/604894425 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b24033070 con 0x5561bd3084d0 2024-04-22T13:37:49.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.937+0000 7f8b19ffb640 1 -- 172.21.15.87:0/604894425 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24003e80 con 0x5561bd3084d0 2024-04-22T13:37:49.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.940+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 msgr2=0x5561bd31d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.940+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd31d8d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8b24001f70 tx=0x7f8b240091b0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 shutdown_connections 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd311a00 0x5561bd311200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd308e80 0x5561bd310cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/604894425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd3084d0 0x5561bd31d8d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 >> 172.21.15.87:0/604894425 conn(0x5561bd302ba0 msgr2=0x5561bd301c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 shutdown_connections 2024-04-22T13:37:49.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.941+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/604894425 wait complete. 2024-04-22T13:37:49.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.944+0000 7f8b3d3a0740 1 -- start start 2024-04-22T13:37:49.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.950+0000 7f8b3d3a0740 1 Processor -- start 2024-04-22T13:37:49.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd319de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2000cbb0 con 0x5561bd3084d0 2024-04-22T13:37:49.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2000e020 con 0x5561bd308e80 2024-04-22T13:37:49.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.951+0000 7f8b3d3a0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2000d280 con 0x5561bd311a00 2024-04-22T13:37:49.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.952+0000 7f8b3d3a0740 10 client.0 _ll_register_callbacks cb 0x7f8b24011a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:49.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.952+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.952+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:55992/0 (socket says 172.21.15.87:55992) 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.952+0000 7f8b337fe640 1 -- 172.21.15.87:0/826552971 learned_addr learned my addr 172.21.15.87:0/826552971 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.952+0000 7f8b38d4f640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 -- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 msgr2=0x5561bd30a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 -- 172.21.15.87:0/826552971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 msgr2=0x5561bd319de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd319de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 -- 172.21.15.87:0/826552971 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561bd30aec0 con 0x5561bd308e80 2024-04-22T13:37:49.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.953+0000 7f8b337fe640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0323b0 tx=0x7f8b2c02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=546b29b6bd541117 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:49.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.961+0000 7f8b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:49.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.961+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c060060 con 0x5561bd308e80 2024-04-22T13:37:49.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.961+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f8b2c003970 con 0x5561bd308e80 2024-04-22T13:37:49.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.961+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b24033070 con 0x5561bd308e80 2024-04-22T13:37:49.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.964+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8b2c039040 con 0x5561bd308e80 2024-04-22T13:37:49.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.964+0000 7f8b3d3a0740 10 client.26390 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:49.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.964+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561bd387d70 con 0x5561bd308e80 2024-04-22T13:37:49.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b2c003b70 con 0x5561bd308e80 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8b2c063020 con 0x5561bd308e80 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 10 client.26390 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 10 client.26390 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 20 client.26390 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd3884d0 0x5561bd38a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 4 client.26390 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:49.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.965+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8b24033070 con 0x5561bd3884d0 2024-04-22T13:37:49.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.966+0000 7f8b33fff640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd3884d0 0x5561bd38a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:49.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.967+0000 7f8b33fff640 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd3884d0 0x5561bd38a8f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=349262c19ada49a8 server_cookie=df984bfa10ea9bc1 in_seq=0 out_seq=0 2024-04-22T13:37:49.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.968+0000 7f8b137fe640 10 client.26390 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:49.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.968+0000 7f8b137fe640 1 -- 172.21.15.87:0/826552971 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8b24033070 con 0x5561bd3884d0 2024-04-22T13:37:49.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.968+0000 7f8b137fe640 10 client.26390 handle_command_reply: tid=0 2024-04-22T13:37:49.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.968+0000 7f8b3d3a0740 1 client.26390 shutdown 2024-04-22T13:37:49.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.975+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 msgr2=0x5561bd31a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.975+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0323b0 tx=0x7f8b2c02c640 comp rx=0 tx=0).stop 2024-04-22T13:37:49.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 shutdown_connections 2024-04-22T13:37:49.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5561bd3884d0 0x5561bd38a8f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561bd311a00 0x5561bd30a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561bd308e80 0x5561bd31a320 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 --2- 172.21.15.87:0/826552971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561bd3084d0 0x5561bd319de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 >> 172.21.15.87:0/826552971 conn(0x5561bd302ba0 msgr2=0x5561bd309260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b137fe640 0 client.26390 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 shutdown_connections 2024-04-22T13:37:49.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.982+0000 7f8b3d3a0740 1 -- 172.21.15.87:0/826552971 wait complete. 2024-04-22T13:37:49.979 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:49.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.983+0000 7f8b3d3a0740 20 client.26390 trim_cache size 0 max 16384 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.985+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 msgr2=0x7f8b1c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.985+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 0x7f8b1c0a6320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c000f80 tx=0x7f8b2c02e040 comp rx=0 tx=0).stop 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.985+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 msgr2=0x7f8b341abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.985+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b341abfb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800b3e0 tx=0x7f8b2800b8a0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 shutdown_connections 2024-04-22T13:37:49.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8b1c0a3e60 0x7f8b1c0a6320 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b3411b730 0x7f8b341abfb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b341107a0 0x7f8b341a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 --2- 172.21.15.87:0/564520024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b3410e7d0 0x7f8b341a76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:49.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.986+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 >> 172.21.15.87:0/564520024 conn(0x7f8b3406d060 msgr2=0x7f8b3411a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:49.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.987+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 shutdown_connections 2024-04-22T13:37:49.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:49.987+0000 7f8b3a7b2640 1 -- 172.21.15.87:0/564520024 wait complete. 2024-04-22T13:37:50.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:37:50.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:50 smithi087 ceph-mon[27999]: pgmap v183: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 830 MiB/s wr, 2.17k op/s 2024-04-22T13:37:50.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.504+0000 7fb825daf640 1 -- 172.21.15.87:0/2251638791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820101900 msgr2=0x7fb820105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.504+0000 7fb825daf640 1 --2- 172.21.15.87:0/2251638791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820101900 0x7fb820105940 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009860 tx=0x7fb80c02bbb0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.507+0000 7fb825daf640 1 -- 172.21.15.87:0/2251638791 shutdown_connections 2024-04-22T13:37:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.507+0000 7fb825daf640 1 --2- 172.21.15.87:0/2251638791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820106020 0x7fb82010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.507+0000 7fb825daf640 1 --2- 172.21.15.87:0/2251638791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820101900 0x7fb820105940 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.507+0000 7fb825daf640 1 --2- 172.21.15.87:0/2251638791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820100f50 0x7fb820101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.507+0000 7fb825daf640 1 -- 172.21.15.87:0/2251638791 >> 172.21.15.87:0/2251638791 conn(0x7fb8200fc7d0 msgr2=0x7fb8200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.510+0000 7fb825daf640 1 -- 172.21.15.87:0/2251638791 shutdown_connections 2024-04-22T13:37:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.511+0000 7fb825daf640 1 -- 172.21.15.87:0/2251638791 wait complete. 2024-04-22T13:37:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 Processor -- start 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 -- start start 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820100f50 0x7fb820116700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820110dd0 0x7fb820111280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb82010f000 con 0x7fb820106020 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb82010ee80 con 0x7fb820100f50 2024-04-22T13:37:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.512+0000 7fb825daf640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb82010f180 con 0x7fb820110dd0 2024-04-22T13:37:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.513+0000 7fb81ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.514+0000 7fb81ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41318/0 (socket says 172.21.15.87:41318) 2024-04-22T13:37:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.514+0000 7fb81ffff640 1 -- 172.21.15.87:0/3354188150 learned_addr learned my addr 172.21.15.87:0/3354188150 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.516+0000 7fb8255ae640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820110dd0 0x7fb820111280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb824dad640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820100f50 0x7fb820116700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb81ffff640 1 -- 172.21.15.87:0/3354188150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820110dd0 msgr2=0x7fb820111280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb81ffff640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820110dd0 0x7fb820111280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb81ffff640 1 -- 172.21.15.87:0/3354188150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820100f50 msgr2=0x7fb820116700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb81ffff640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820100f50 0x7fb820116700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.517+0000 7fb81ffff640 1 -- 172.21.15.87:0/3354188150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8201b3b10 con 0x7fb820106020 2024-04-22T13:37:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.520+0000 7fb81ffff640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb80c02c0f0 tx=0x7fb80c035a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.520+0000 7fb81dffb640 1 -- 172.21.15.87:0/3354188150 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c039040 con 0x7fb820106020 2024-04-22T13:37:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.520+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8201b3da0 con 0x7fb820106020 2024-04-22T13:37:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.521+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8201b4250 con 0x7fb820106020 2024-04-22T13:37:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.520+0000 7fb81dffb640 1 -- 172.21.15.87:0/3354188150 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb80c035c50 con 0x7fb820106020 2024-04-22T13:37:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.521+0000 7fb81dffb640 1 -- 172.21.15.87:0/3354188150 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c005650 con 0x7fb820106020 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.529+0000 7fb829c00740 1 Processor -- start 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.529+0000 7fb829c00740 1 -- start start 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.529+0000 7fb829c00740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11d4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb829c00740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb829c00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11ddc00 0x55acb11e0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb829c00740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55acb11391f0 con 0x55acb11d5080 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb829c00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55acb10d64b0 con 0x55acb11ddc00 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb829c00740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55acb11351b0 con 0x55acb11d46d0 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb81dffb640 1 -- 172.21.15.87:0/3354188150 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb80c044020 con 0x7fb820106020 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb824dad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb824dad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41332/0 (socket says 172.21.15.87:41332) 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.530+0000 7fb824dad640 1 -- 172.21.15.87:0/3777159315 learned_addr learned my addr 172.21.15.87:0/3777159315 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 -- 172.21.15.87:0/3777159315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 msgr2=0x55acb11d4ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11d4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 -- 172.21.15.87:0/3777159315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11ddc00 msgr2=0x55acb11e0050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11ddc00 0x55acb11e0050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 -- 172.21.15.87:0/3777159315 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acb10c81b0 con 0x55acb11d5080 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb824dad640 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb814009c20 tx=0x7fb8140117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9835195d2ed79c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.531+0000 7fb81dffb640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 0x7fb8100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.532+0000 7fb81dffb640 1 -- 172.21.15.87:0/3354188150 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fb80c0f1530 con 0x7fb820106020 2024-04-22T13:37:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.532+0000 7fb8255ae640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 0x7fb8100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.532+0000 7fb80affd640 1 -- 172.21.15.87:0/3777159315 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814012450 con 0x55acb11d5080 2024-04-22T13:37:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.533+0000 7fb80affd640 1 -- 172.21.15.87:0/3777159315 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb814012aa0 con 0x55acb11d5080 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 -- 172.21.15.87:0/3777159315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 msgr2=0x55acb11dd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb814009c20 tx=0x7fb8140117b0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 -- 172.21.15.87:0/3777159315 shutdown_connections 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11ddc00 0x55acb11e0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d5080 0x55acb11dd630 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 --2- 172.21.15.87:0/3777159315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11d4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.534+0000 7fb829c00740 1 -- 172.21.15.87:0/3777159315 >> 172.21.15.87:0/3777159315 conn(0x55acb11ceda0 msgr2=0x55acb11cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.535+0000 7fb8255ae640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 0x7fb8100a6260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb818005fd0 tx=0x7fb8180095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.535+0000 7fb829c00740 1 -- 172.21.15.87:0/3777159315 shutdown_connections 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.537+0000 7fb829c00740 1 -- 172.21.15.87:0/3777159315 wait complete. 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.537+0000 7fb829c00740 1 -- start start 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.538+0000 7fb829c00740 1 Processor -- start 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.538+0000 7fb829c00740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11da7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11ddc00 0x55acb11dad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55acb11e0bc0 con 0x55acb11ddc00 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55acb10f2500 con 0x55acb11d5080 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55acb11e0d40 con 0x55acb11d46d0 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb829c00740 10 client.0 _ll_register_callbacks cb 0x55acb10c0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb824dad640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb824dad640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:35196/0 (socket says 172.21.15.87:35196) 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.539+0000 7fb824dad640 1 -- 172.21.15.87:0/3562490844 learned_addr learned my addr 172.21.15.87:0/3562490844 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 -- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 msgr2=0x55acb11da7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11da7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 -- 172.21.15.87:0/3562490844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11ddc00 msgr2=0x55acb11dad00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11ddc00 0x55acb11dad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 -- 172.21.15.87:0/3562490844 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acb0e4b3f0 con 0x55acb11d46d0 2024-04-22T13:37:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.540+0000 7fb824dad640 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb8140099e0 tx=0x7fb81400ff00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f186ea3ddbb4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.552+0000 7fb7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.552+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814028020 con 0x55acb11d46d0 2024-04-22T13:37:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.552+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fb814005270 con 0x55acb11d46d0 2024-04-22T13:37:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.553+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb814012aa0 con 0x55acb11d46d0 2024-04-22T13:37:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.554+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb814033050 con 0x55acb11d46d0 2024-04-22T13:37:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.554+0000 7fb829c00740 10 client.26440 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.554+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55acb0f157d0 con 0x55acb11d46d0 2024-04-22T13:37:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.554+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb814012aa0 con 0x55acb11d46d0 2024-04-22T13:37:50.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.556+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb8140039a0 con 0x55acb11d46d0 2024-04-22T13:37:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 10 client.26440 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 10 client.26440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 20 client.26440 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb120fdb0 0x55acb12121a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 4 client.26440 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.560+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acb10d8930 con 0x55acb120fdb0 2024-04-22T13:37:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.561+0000 7fb81ffff640 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb120fdb0 0x55acb12121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.563+0000 7fb81ffff640 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb120fdb0 0x55acb12121a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5be8d376b5aefbf server_cookie=33fb2e028f274cd7 in_seq=0 out_seq=0 2024-04-22T13:37:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.563+0000 7fb7fffff640 10 client.26440 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.569+0000 7fb7fffff640 1 -- 172.21.15.87:0/3562490844 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55acb10d8930 con 0x55acb120fdb0 2024-04-22T13:37:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.569+0000 7fb7fffff640 10 client.26440 handle_command_reply: tid=0 2024-04-22T13:37:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.571+0000 7fb829c00740 1 client.26440 shutdown 2024-04-22T13:37:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 msgr2=0x55acb11dd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb8140099e0 tx=0x7fb81400ff00 comp rx=0 tx=0).stop 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 shutdown_connections 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb120fdb0 0x55acb12121a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11ddc00 0x55acb11dad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11da7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 --2- 172.21.15.87:0/3562490844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11d46d0 0x55acb11dd430 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb7fffff640 0 client.26440 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 >> 172.21.15.87:0/3562490844 conn(0x55acb11ceda0 msgr2=0x55acb11d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 shutdown_connections 2024-04-22T13:37:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.572+0000 7fb829c00740 1 -- 172.21.15.87:0/3562490844 wait complete. 2024-04-22T13:37:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.573+0000 7fb829c00740 20 client.26440 trim_cache size 0 max 16384 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.614+0000 7fb829c00740 1 Processor -- start 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.614+0000 7fb829c00740 1 -- start start 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.614+0000 7fb829c00740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb829c00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb829c00740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11dcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb829c00740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb804011000 con 0x55acb11d46d0 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb829c00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb80400e8d0 con 0x55acb11d5080 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb829c00740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb80400c720 con 0x55acb11ddc00 2024-04-22T13:37:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb824dad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb824dad640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41344/0 (socket says 172.21.15.87:41344) 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.615+0000 7fb824dad640 1 -- 172.21.15.87:0/1997615034 learned_addr learned my addr 172.21.15.87:0/1997615034 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb81ffff640 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb8255ae640 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11dcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb824dad640 1 -- 172.21.15.87:0/1997615034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 msgr2=0x55acb11dcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb824dad640 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11dcef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb824dad640 1 -- 172.21.15.87:0/1997615034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 msgr2=0x55acb11d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb824dad640 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.616+0000 7fb824dad640 1 -- 172.21.15.87:0/1997615034 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acb11d10a0 con 0x55acb11d46d0 2024-04-22T13:37:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.617+0000 7fb824dad640 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fb8140117f0 tx=0x7fb814009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc457fb57c749fde server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.626+0000 7fb80a7fc640 1 -- 172.21.15.87:0/1997615034 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814028070 con 0x55acb11d46d0 2024-04-22T13:37:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.626+0000 7fb80a7fc640 1 -- 172.21.15.87:0/1997615034 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb81400f500 con 0x55acb11d46d0 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.627+0000 7fb829c00740 1 -- 172.21.15.87:0/1997615034 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 msgr2=0x55acb11e9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.627+0000 7fb829c00740 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fb8140117f0 tx=0x7fb814009b50 comp rx=0 tx=0).stop 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.629+0000 7fb829c00740 1 -- 172.21.15.87:0/1997615034 shutdown_connections 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.629+0000 7fb829c00740 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11dcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.629+0000 7fb829c00740 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.630+0000 7fb829c00740 1 --2- 172.21.15.87:0/1997615034 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e9b60 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.630+0000 7fb829c00740 1 -- 172.21.15.87:0/1997615034 >> 172.21.15.87:0/1997615034 conn(0x55acb11ceda0 msgr2=0x55acb11d5d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:50 smithi132 ceph-mon[31531]: pgmap v183: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 830 MiB/s wr, 2.17k op/s 2024-04-22T13:37:50.714 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:50 smithi120 ceph-mon[31101]: pgmap v183: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 830 MiB/s wr, 2.17k op/s 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.637+0000 7fb829c00740 1 -- 172.21.15.87:0/1997615034 shutdown_connections 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.637+0000 7fb829c00740 1 -- 172.21.15.87:0/1997615034 wait complete. 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.638+0000 7fb829c00740 1 -- start start 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.638+0000 7fb829c00740 1 Processor -- start 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11e2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11e2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb80400cd90 con 0x55acb11d46d0 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb80400e200 con 0x55acb11d5080 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb80400d460 con 0x55acb11ddc00 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.642+0000 7fb829c00740 10 client.0 _ll_register_callbacks cb 0x7fb814005610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.650+0000 7fb81ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.650+0000 7fb8255ae640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11e2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.651+0000 7fb8255ae640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11e2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:35216/0 (socket says 172.21.15.87:35216) 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.651+0000 7fb8255ae640 1 -- 172.21.15.87:0/1881199147 learned_addr learned my addr 172.21.15.87:0/1881199147 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.652+0000 7fb824dad640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.657+0000 7fb81ffff640 1 -- 172.21.15.87:0/1881199147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 msgr2=0x55acb11e2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.657+0000 7fb81ffff640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11e2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.657+0000 7fb81ffff640 1 -- 172.21.15.87:0/1881199147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 msgr2=0x55acb11e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.657+0000 7fb81ffff640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.657+0000 7fb81ffff640 1 -- 172.21.15.87:0/1881199147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acb11d9770 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.664+0000 7fb81ffff640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11e2220 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0eee00 tx=0x7fb80c009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d2909818dd5610 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.664+0000 7fb7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.670+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c035e00 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.670+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fb80c0bb950 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.670+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55acb1146c10 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.671+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb80c0bdd50 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.671+0000 7fb829c00740 10 client.26399 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.671+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55acb11d4e20 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.672+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55acb1146c10 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.673+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb80c03c070 con 0x55acb11d5080 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.674+0000 7fb829c00740 10 client.26399 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.674+0000 7fb829c00740 10 client.26399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.674+0000 7fb829c00740 20 client.26399 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.675+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb1276bb0 0x55acb1278fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.675+0000 7fb829c00740 4 client.26399 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.675+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55acb1140a80 con 0x55acb1276bb0 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.683+0000 7fb824dad640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb1276bb0 0x55acb1278fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.709+0000 7fb824dad640 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb1276bb0 0x55acb1278fd0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeafbe746d467225 server_cookie=6e9bd4c4b6edc800 in_seq=0 out_seq=0 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.709+0000 7fb7fffff640 10 client.26399 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.715+0000 7fb7fffff640 1 -- 172.21.15.87:0/1881199147 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55acb1140a80 con 0x55acb1276bb0 2024-04-22T13:37:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.715+0000 7fb7fffff640 10 client.26399 handle_command_reply: tid=0 2024-04-22T13:37:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.726+0000 7fb829c00740 1 client.26399 shutdown 2024-04-22T13:37:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.727+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 msgr2=0x55acb11e2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.727+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11e2220 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0eee00 tx=0x7fb80c009550 comp rx=0 tx=0).stop 2024-04-22T13:37:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 shutdown_connections 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55acb1276bb0 0x55acb1278fd0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55acb11ddc00 0x55acb11e2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acb11d5080 0x55acb11e2220 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 --2- 172.21.15.87:0/1881199147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55acb11d46d0 0x55acb11e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 >> 172.21.15.87:0/1881199147 conn(0x55acb11ceda0 msgr2=0x55acb11cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.731+0000 7fb7fffff640 0 client.26399 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:50.731 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:37:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.732+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 shutdown_connections 2024-04-22T13:37:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.732+0000 7fb829c00740 1 -- 172.21.15.87:0/1881199147 wait complete. 2024-04-22T13:37:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.732+0000 7fb829c00740 20 client.26399 trim_cache size 0 max 16384 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.746+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 msgr2=0x7fb8100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 0x7fb8100a6260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb818005fd0 tx=0x7fb8180095c0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 msgr2=0x7fb820110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb80c02c0f0 tx=0x7fb80c035a60 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 shutdown_connections 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb8100a3da0 0x7fb8100a6260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb820110dd0 0x7fb820111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb820106020 0x7fb820110890 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 --2- 172.21.15.87:0/3354188150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb820100f50 0x7fb820116700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:50.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.747+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 >> 172.21.15.87:0/3354188150 conn(0x7fb8200fc7d0 msgr2=0x7fb8200fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:50.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.751+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 shutdown_connections 2024-04-22T13:37:50.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:50.752+0000 7fb825daf640 1 -- 172.21.15.87:0/3354188150 wait complete. 2024-04-22T13:37:51.850 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:37:51.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:37:51.850 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:37:52.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 -- 172.21.15.87:0/245969739 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 msgr2=0x7fcb1c11d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/245969739 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c11d740 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009900 tx=0x7fcb1002bcf0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 -- 172.21.15.87:0/245969739 shutdown_connections 2024-04-22T13:37:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/245969739 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c11d740 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/245969739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb1c108f50 0x7fcb1c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/245969739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c1109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.321+0000 7fcb1b59e640 1 -- 172.21.15.87:0/245969739 >> 172.21.15.87:0/245969739 conn(0x7fcb1c06ced0 msgr2=0x7fcb1c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.329+0000 7fcb1b59e640 1 -- 172.21.15.87:0/245969739 shutdown_connections 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.330+0000 7fcb1b59e640 1 -- 172.21.15.87:0/245969739 wait complete. 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.332+0000 7fcb1b59e640 1 Processor -- start 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.332+0000 7fcb1b59e640 1 -- start start 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb1c108f50 0x7fcb1c11d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c116540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c116a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c120f50 con 0x7fcb1c1099d0 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c120dd0 con 0x7fcb1c1105f0 2024-04-22T13:37:52.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1b59e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1210d0 con 0x7fcb1c108f50 2024-04-22T13:37:52.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1ad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c116a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1ad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c116a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56064/0 (socket says 172.21.15.87:56064) 2024-04-22T13:37:52.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.333+0000 7fcb1ad9d640 1 -- 172.21.15.87:0/663247579 learned_addr learned my addr 172.21.15.87:0/663247579 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.336+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c116540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.337+0000 7fcb19d9b640 1 -- 172.21.15.87:0/663247579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb1c108f50 msgr2=0x7fcb1c11d400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.337+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb1c108f50 0x7fcb1c11d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.337+0000 7fcb19d9b640 1 -- 172.21.15.87:0/663247579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 msgr2=0x7fcb1c116a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.337+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c116a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.337+0000 7fcb19d9b640 1 -- 172.21.15.87:0/663247579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c117180 con 0x7fcb1c1099d0 2024-04-22T13:37:52.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.338+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c116540 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400bd70 tx=0x7fcb1400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.342+0000 7fcb0b7fe640 1 -- 172.21.15.87:0/663247579 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14019070 con 0x7fcb1c1099d0 2024-04-22T13:37:52.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.342+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c11af90 con 0x7fcb1c1099d0 2024-04-22T13:37:52.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.342+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c11b4e0 con 0x7fcb1c1099d0 2024-04-22T13:37:52.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.342+0000 7fcb0b7fe640 1 -- 172.21.15.87:0/663247579 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb140148f0 con 0x7fcb1c1099d0 2024-04-22T13:37:52.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.342+0000 7fcb0b7fe640 1 -- 172.21.15.87:0/663247579 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14012920 con 0x7fcb1c1099d0 2024-04-22T13:37:52.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.351+0000 7fcb237da740 1 Processor -- start 2024-04-22T13:37:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.352+0000 7fcb237da740 1 -- start start 2024-04-22T13:37:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cd5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4cde9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4ce1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5569a4c39f00 con 0x5569a4cdefb0 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569a4bd7590 con 0x5569a4cd6430 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.353+0000 7fcb237da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569a4c35ec0 con 0x5569a4cd5a80 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.354+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4cde9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.356+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4cde9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56080/0 (socket says 172.21.15.87:56080) 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.356+0000 7fcb1a59c640 1 -- 172.21.15.87:0/3747096893 learned_addr learned my addr 172.21.15.87:0/3747096893 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cd5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 -- 172.21.15.87:0/3747096893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 msgr2=0x5569a4cde9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4cde9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 -- 172.21.15.87:0/3747096893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 msgr2=0x5569a4ce1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4ce1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.357+0000 7fcb1ad9d640 1 -- 172.21.15.87:0/3747096893 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569a4a147e0 con 0x5569a4cd5a80 2024-04-22T13:37:52.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.360+0000 7fcb0b7fe640 1 -- 172.21.15.87:0/663247579 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcb1401c070 con 0x7fcb1c1099d0 2024-04-22T13:37:52.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.360+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cd5e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005ab0 tx=0x7fcb100040f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cca3fd911238307 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.361+0000 7fcb0b7fe640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 0x7fcb040a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.361+0000 7fcb08ff9640 1 -- 172.21.15.87:0/3747096893 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10004320 con 0x5569a4cd5a80 2024-04-22T13:37:52.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.361+0000 7fcb08ff9640 1 -- 172.21.15.87:0/3747096893 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb100044c0 con 0x5569a4cd5a80 2024-04-22T13:37:52.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.361+0000 7fcb08ff9640 1 -- 172.21.15.87:0/3747096893 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10004920 con 0x5569a4cd5a80 2024-04-22T13:37:52.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.362+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 0x7fcb040a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.362+0000 7fcb237da740 1 -- 172.21.15.87:0/3747096893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 msgr2=0x5569a4cd5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.362+0000 7fcb237da740 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cd5e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005ab0 tx=0x7fcb100040f0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb0b7fe640 1 -- 172.21.15.87:0/663247579 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fcb140d1e40 con 0x7fcb1c1099d0 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb237da740 1 -- 172.21.15.87:0/3747096893 shutdown_connections 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb237da740 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4ce1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb237da740 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4cde9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb237da740 1 --2- 172.21.15.87:0/3747096893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cd5e60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb237da740 1 -- 172.21.15.87:0/3747096893 >> 172.21.15.87:0/3747096893 conn(0x5569a4cd0150 msgr2=0x5569a4cd0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.363+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 0x7fcb040a63e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5569a4cd11c0 tx=0x7fcb0c009110 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.367+0000 7fcb237da740 1 -- 172.21.15.87:0/3747096893 shutdown_connections 2024-04-22T13:37:52.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.367+0000 7fcb237da740 1 -- 172.21.15.87:0/3747096893 wait complete. 2024-04-22T13:37:52.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.368+0000 7fcb237da740 1 -- start start 2024-04-22T13:37:52.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.368+0000 7fcb237da740 1 Processor -- start 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd6430 0x5569a4cdc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cdefb0 0x5569a4cdca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5569a4ce1ec0 con 0x5569a4cd6430 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569a4bf3580 con 0x5569a4cdefb0 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569a4ce2040 con 0x5569a4cd5a80 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.369+0000 7fcb237da740 10 client.0 _ll_register_callbacks cb 0x5569a4bf9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.370+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.370+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:35264/0 (socket says 172.21.15.87:35264) 2024-04-22T13:37:52.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.370+0000 7fcb1a59c640 1 -- 172.21.15.87:0/4003890877 learned_addr learned my addr 172.21.15.87:0/4003890877 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 -- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cdefb0 msgr2=0x5569a4cdca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cdefb0 0x5569a4cdca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 -- 172.21.15.87:0/4003890877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd6430 msgr2=0x5569a4cdc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd6430 0x5569a4cdc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 -- 172.21.15.87:0/4003890877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569a4a74f70 con 0x5569a4cd5a80 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.371+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c015760 tx=0x7fcb0c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8134aded4e72f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.372+0000 7fcb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.372+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c023020 con 0x5569a4cd5a80 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.372+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5569a4c47920 con 0x5569a4cd5a80 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.372+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fcb0c013590 con 0x5569a4cd5a80 2024-04-22T13:37:52.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.373+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcb0c02e050 con 0x5569a4cd5a80 2024-04-22T13:37:52.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.373+0000 7fcb237da740 10 client.26449 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:52.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.374+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5569a490c170 con 0x5569a4cd5a80 2024-04-22T13:37:52.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.378+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5569a4c47920 con 0x5569a4cd5a80 2024-04-22T13:37:52.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fcb0c01e030 con 0x5569a4cd5a80 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 10 client.26449 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 20 client.26449 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4cefa20 0x5569a4cf1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 4 client.26449 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:52.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.379+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569a4c41790 con 0x5569a4cefa20 2024-04-22T13:37:52.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.380+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4cefa20 0x5569a4cf1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.383+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4cefa20 0x5569a4cf1ee0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3271147e725edd5 server_cookie=2e1a86bce69f62e in_seq=0 out_seq=0 2024-04-22T13:37:52.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.383+0000 7fcb01ffb640 10 client.26449 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb01ffb640 1 -- 172.21.15.87:0/4003890877 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569a4c41790 con 0x5569a4cefa20 2024-04-22T13:37:52.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb01ffb640 10 client.26449 handle_command_reply: tid=0 2024-04-22T13:37:52.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 client.26449 shutdown 2024-04-22T13:37:52.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 msgr2=0x5569a4cdbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c015760 tx=0x7fcb0c004750 comp rx=0 tx=0).stop 2024-04-22T13:37:52.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 shutdown_connections 2024-04-22T13:37:52.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4cefa20 0x5569a4cf1ee0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cdefb0 0x5569a4cdca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd6430 0x5569a4cdc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 --2- 172.21.15.87:0/4003890877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd5a80 0x5569a4cdbf90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 >> 172.21.15.87:0/4003890877 conn(0x5569a4cd0150 msgr2=0x5569a4cd1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.389+0000 7fcb01ffb640 0 client.26449 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.390+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 shutdown_connections 2024-04-22T13:37:52.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.390+0000 7fcb237da740 1 -- 172.21.15.87:0/4003890877 wait complete. 2024-04-22T13:37:52.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.393+0000 7fcb237da740 20 client.26449 trim_cache size 0 max 16384 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 Processor -- start 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 -- start start 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd6430 0x5569a4ceae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4cf77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc010d00 con 0x5569a4cdefb0 2024-04-22T13:37:52.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc00e5d0 con 0x5569a4cd5a80 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.445+0000 7fcb237da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc00c4d0 con 0x5569a4cd6430 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56102/0 (socket says 172.21.15.87:56102) 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 -- 172.21.15.87:0/272554043 learned_addr learned my addr 172.21.15.87:0/272554043 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4cf77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 -- 172.21.15.87:0/272554043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd6430 msgr2=0x5569a4ceae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd6430 0x5569a4ceae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 -- 172.21.15.87:0/272554043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 msgr2=0x5569a4cf77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4cf77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 -- 172.21.15.87:0/272554043 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569a4cee180 con 0x5569a4cd5a80 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1ad9d640 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4cf77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:37:52.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.446+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e610 tx=0x7fcb0c0096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8074b742de10a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.449+0000 7fcb03fff640 1 -- 172.21.15.87:0/272554043 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c023040 con 0x5569a4cd5a80 2024-04-22T13:37:52.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.449+0000 7fcb03fff640 1 -- 172.21.15.87:0/272554043 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb0c037070 con 0x5569a4cd5a80 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 -- 172.21.15.87:0/272554043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 msgr2=0x5569a4cde6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e610 tx=0x7fcb0c0096b0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 -- 172.21.15.87:0/272554043 shutdown_connections 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cdefb0 0x5569a4cf77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cd6430 0x5569a4ceae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 --2- 172.21.15.87:0/272554043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd5a80 0x5569a4cde6c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.450+0000 7fcb237da740 1 -- 172.21.15.87:0/272554043 >> 172.21.15.87:0/272554043 conn(0x5569a4cd0150 msgr2=0x5569a4cda860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.458+0000 7fcb237da740 1 -- 172.21.15.87:0/272554043 shutdown_connections 2024-04-22T13:37:52.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.458+0000 7fcb237da740 1 -- 172.21.15.87:0/272554043 wait complete. 2024-04-22T13:37:52.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.458+0000 7fcb237da740 1 -- start start 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 Processor -- start 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4ce2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cdefb0 0x5569a4d54050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc00ca90 con 0x5569a4cd5a80 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc00df00 con 0x5569a4cd6430 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc00d160 con 0x5569a4cdefb0 2024-04-22T13:37:52.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.459+0000 7fcb237da740 10 client.0 _ll_register_callbacks cb 0x7fcb0c00eea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:52.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41394/0 (socket says 172.21.15.87:41394) 2024-04-22T13:37:52.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 -- 172.21.15.87:0/1500118977 learned_addr learned my addr 172.21.15.87:0/1500118977 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 -- 172.21.15.87:0/1500118977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cdefb0 msgr2=0x5569a4d54050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cdefb0 0x5569a4d54050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 -- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 msgr2=0x5569a4ce2a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4ce2a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.460+0000 7fcb1a59c640 1 -- 172.21.15.87:0/1500118977 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569a4cd7b70 con 0x5569a4cd5a80 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.461+0000 7fcb1a59c640 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c01fac0 tx=0x7fcb0c01f760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c187116a10bc7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.461+0000 7fcb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.461+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c023040 con 0x5569a4cd5a80 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.461+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb0c037070 con 0x5569a4cd5a80 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.462+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fcb0c0137a0 con 0x5569a4cd5a80 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.462+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcb0c0139c0 con 0x5569a4cd5a80 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.462+0000 7fcb237da740 10 client.17313 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.462+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5569a4d55670 con 0x5569a4cd5a80 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb0c037070 con 0x5569a4cd5a80 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fcb0c01e030 con 0x5569a4cd5a80 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 10 client.17313 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 10 client.17313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 20 client.17313 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4d55c50 0x5569a4d58070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 4 client.17313 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:37:52.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.463+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5569a4bbdc50 con 0x5569a4d55c50 2024-04-22T13:37:52.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.464+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4d55c50 0x5569a4d58070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.465+0000 7fcb19d9b640 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4d55c50 0x5569a4d58070 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce1e62e5219b797 server_cookie=b5d1b45a5deb395a in_seq=0 out_seq=0 2024-04-22T13:37:52.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.465+0000 7fcb01ffb640 10 client.17313 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.469+0000 7fcb01ffb640 1 -- 172.21.15.87:0/1500118977 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5569a4bbdc50 con 0x5569a4d55c50 2024-04-22T13:37:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.469+0000 7fcb01ffb640 10 client.17313 handle_command_reply: tid=0 2024-04-22T13:37:52.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.469+0000 7fcb237da740 1 client.17313 shutdown 2024-04-22T13:37:52.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.471+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 msgr2=0x5569a4ce2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.471+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c01fac0 tx=0x7fcb0c01f760 comp rx=0 tx=0).stop 2024-04-22T13:37:52.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 shutdown_connections 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5569a4d55c50 0x5569a4d58070 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569a4cdefb0 0x5569a4d54050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569a4cd6430 0x5569a4ce2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 --2- 172.21.15.87:0/1500118977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5569a4cd5a80 0x5569a4ce2550 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 >> 172.21.15.87:0/1500118977 conn(0x5569a4cd0150 msgr2=0x5569a4cdbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb01ffb640 0 client.17313 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 shutdown_connections 2024-04-22T13:37:52.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.472+0000 7fcb237da740 1 -- 172.21.15.87:0/1500118977 wait complete. 2024-04-22T13:37:52.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.473+0000 7fcb237da740 20 client.17313 trim_cache size 0 max 16384 2024-04-22T13:37:52.471 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:37:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "48416d74-e6b6-4d56-bd4d-a1ae9f7659f8", 2024-04-22T13:37:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:37:52.472 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:52.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.481+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 msgr2=0x7fcb040a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.481+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 0x7fcb040a63e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5569a4cd11c0 tx=0x7fcb0c009110 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.481+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 msgr2=0x7fcb1c116540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.481+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c116540 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400bd70 tx=0x7fcb1400be60 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 shutdown_connections 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcb040a3f20 0x7fcb040a63e0 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5569a4cd11c0 tx=0x7fcb0c009110 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb1c1105f0 0x7fcb1c116a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcb1c1099d0 0x7fcb1c116540 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 --2- 172.21.15.87:0/663247579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb1c108f50 0x7fcb1c11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.482+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 >> 172.21.15.87:0/663247579 conn(0x7fcb1c06ced0 msgr2=0x7fcb1c070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.486+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 shutdown_connections 2024-04-22T13:37:52.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.487+0000 7fcb1b59e640 1 -- 172.21.15.87:0/663247579 wait complete. 2024-04-22T13:37:52.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:37:52.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:37:52.603 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:52 smithi087 ceph-mon[27999]: pgmap v184: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 777 MiB/s wr, 2.07k op/s 2024-04-22T13:37:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:52 smithi120 ceph-mon[31101]: pgmap v184: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 777 MiB/s wr, 2.07k op/s 2024-04-22T13:37:52.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:52 smithi132 ceph-mon[31531]: pgmap v184: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 777 MiB/s wr, 2.07k op/s 2024-04-22T13:37:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.843+0000 7fc4abc81640 1 -- 172.21.15.87:0/1131937696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 msgr2=0x7fc4a410ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.843+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1131937696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a410ec10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009ca0 tx=0x7fc49c02c140 comp rx=0 tx=0).stop 2024-04-22T13:37:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.847+0000 7fc4abc81640 1 -- 172.21.15.87:0/1131937696 shutdown_connections 2024-04-22T13:37:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.847+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1131937696 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4a411b730 0x7fc4a411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.847+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1131937696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a410ec10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.847+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1131937696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a41106d0 0x7fc4a410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.847+0000 7fc4abc81640 1 -- 172.21.15.87:0/1131937696 >> 172.21.15.87:0/1131937696 conn(0x7fc4a406d070 msgr2=0x7fc4a406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.855+0000 7fc4abc81640 1 -- 172.21.15.87:0/1131937696 shutdown_connections 2024-04-22T13:37:52.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.858+0000 7fc4abc81640 1 -- 172.21.15.87:0/1131937696 wait complete. 2024-04-22T13:37:52.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.860+0000 7fc4abc81640 1 Processor -- start 2024-04-22T13:37:52.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 -- start start 2024-04-22T13:37:52.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a4114bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4a41106d0 0x7fc4a41150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a41113e0 con 0x7fc4a41106d0 2024-04-22T13:37:52.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a4111260 con 0x7fc4a411b730 2024-04-22T13:37:52.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.865+0000 7fc4abc81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a4111560 con 0x7fc4a410e7b0 2024-04-22T13:37:52.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.867+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.867+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56116/0 (socket says 172.21.15.87:56116) 2024-04-22T13:37:52.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.867+0000 7fc4aa21e640 1 -- 172.21.15.87:0/1959168795 learned_addr learned my addr 172.21.15.87:0/1959168795 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.868+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a4114bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.869+0000 7fc4aa21e640 1 -- 172.21.15.87:0/1959168795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 msgr2=0x7fc4a4114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.869+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a4114bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.869+0000 7fc4aa21e640 1 -- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4a41106d0 msgr2=0x7fc4a41150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.869+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4a41106d0 0x7fc4a41150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.869+0000 7fc4aa21e640 1 -- 172.21.15.87:0/1959168795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a4119c10 con 0x7fc4a411b730 2024-04-22T13:37:52.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.870+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b3c0 tx=0x7fc4a000b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.872+0000 7fc49affd640 1 -- 172.21.15.87:0/1959168795 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0011830 con 0x7fc4a411b730 2024-04-22T13:37:52.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.873+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a4119f00 con 0x7fc4a411b730 2024-04-22T13:37:52.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.873+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a41bc950 con 0x7fc4a411b730 2024-04-22T13:37:52.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.873+0000 7fc49affd640 1 -- 172.21.15.87:0/1959168795 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a0016070 con 0x7fc4a411b730 2024-04-22T13:37:52.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.873+0000 7fc49affd640 1 -- 172.21.15.87:0/1959168795 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0010e50 con 0x7fc4a411b730 2024-04-22T13:37:52.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.879+0000 7fc49affd640 1 -- 172.21.15.87:0/1959168795 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4a001b4a0 con 0x7fc4a411b730 2024-04-22T13:37:52.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.881+0000 7fc49affd640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 0x7fc4900a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.881+0000 7fc49affd640 1 -- 172.21.15.87:0/1959168795 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fc4a00d0c00 con 0x7fc4a411b730 2024-04-22T13:37:52.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.891+0000 7fc4ae873740 1 Processor -- start 2024-04-22T13:37:52.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 -- start start 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53580 0x56312fe53960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe5c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56312fdb81e0 con 0x56312fe53ea0 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56312fd55390 con 0x56312fe53580 2024-04-22T13:37:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.893+0000 7fc4ae873740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56312fdb41a0 con 0x56312fe5ca80 2024-04-22T13:37:52.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.895+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.895+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:35300/0 (socket says 172.21.15.87:35300) 2024-04-22T13:37:52.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.895+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3452308133 learned_addr learned my addr 172.21.15.87:0/3452308133 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.896+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3452308133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53580 msgr2=0x56312fe53960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.896+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53580 0x56312fe53960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.896+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3452308133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 msgr2=0x56312fe5c540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T13:37:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.896+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe5c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.896+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3452308133 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56312fb947d0 con 0x56312fe5ca80 2024-04-22T13:37:52.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.897+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b390 tx=0x7fc4a009d500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd486710e3aee0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.897+0000 7fc48ffff640 1 -- 172.21.15.87:0/3452308133 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00b6070 con 0x56312fe5ca80 2024-04-22T13:37:52.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.898+0000 7fc48ffff640 1 -- 172.21.15.87:0/3452308133 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00b6210 con 0x56312fe5ca80 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.899+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 0x7fc4900a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 -- 172.21.15.87:0/3452308133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 msgr2=0x56312fe5ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b390 tx=0x7fc4a009d500 comp rx=0 tx=0).stop 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 -- 172.21.15.87:0/3452308133 shutdown_connections 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe5ee70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe5c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3452308133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53580 0x56312fe53960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.900+0000 7fc4ae873740 1 -- 172.21.15.87:0/3452308133 >> 172.21.15.87:0/3452308133 conn(0x56312fe4dcb0 msgr2=0x56312fe4e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.903+0000 7fc4ae873740 1 -- 172.21.15.87:0/3452308133 shutdown_connections 2024-04-22T13:37:52.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.912+0000 7fc4ae873740 1 -- 172.21.15.87:0/3452308133 wait complete. 2024-04-22T13:37:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.912+0000 7fc4ae873740 1 -- start start 2024-04-22T13:37:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.913+0000 7fc4ae873740 1 Processor -- start 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.913+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe53580 0x56312fe599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe59ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56312fe5f6f0 con 0x56312fe53ea0 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56312fd712f0 con 0x56312fe5ca80 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56312fe5f870 con 0x56312fe53580 2024-04-22T13:37:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.914+0000 7fc4ae873740 10 client.0 _ll_register_callbacks cb 0x56312fd7e580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.916+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.917+0000 7fc4aa21e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56128/0 (socket says 172.21.15.87:56128) 2024-04-22T13:37:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.917+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3952100231 learned_addr learned my addr 172.21.15.87:0/3952100231 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.920+0000 7fc4a921c640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe59ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 0x7fc4900a6590 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc494005fd0 tx=0x7fc494005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3952100231 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe53580 msgr2=0x56312fe599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe53580 0x56312fe599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3952100231 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 msgr2=0x56312fe59ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe59ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.921+0000 7fc4aa21e640 1 -- 172.21.15.87:0/3952100231 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56312faca3f0 con 0x56312fe5ca80 2024-04-22T13:37:52.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.925+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00a58a0 tx=0x7fc4a00a5930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d01a01bf769299 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:52.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.926+0000 7fc48d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:37:52.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.927+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00b6040 con 0x56312fe5ca80 2024-04-22T13:37:52.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.927+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a00b6210 con 0x56312fe5ca80 2024-04-22T13:37:52.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.927+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fc4a00bd0b0 con 0x56312fe5ca80 2024-04-22T13:37:52.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.928+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc4a00ad050 con 0x56312fe5ca80 2024-04-22T13:37:52.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.930+0000 7fc4ae873740 10 client.26417 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:52.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.930+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56312fa8a0f0 con 0x56312fe5ca80 2024-04-22T13:37:52.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.934+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00b6210 con 0x56312fe5ca80 2024-04-22T13:37:52.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.934+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc4a00a7c30 con 0x56312fe5ca80 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 10 client.26417 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 10 client.26417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 20 client.26417 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fe6c780 0x56312fe6eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 4 client.26417 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:37:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.935+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56312fdc5c00 con 0x56312fe6c780 2024-04-22T13:37:52.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.938+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fe6c780 0x56312fe6eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.942+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fe6c780 0x56312fe6eb90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afb094f6a547f428 server_cookie=fd61ac39b9614ccf in_seq=0 out_seq=0 2024-04-22T13:37:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.942+0000 7fc48d7fa640 10 client.26417 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.948+0000 7fc48d7fa640 1 -- 172.21.15.87:0/3952100231 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56312fdc5c00 con 0x56312fe6c780 2024-04-22T13:37:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.948+0000 7fc48d7fa640 10 client.26417 handle_command_reply: tid=0 2024-04-22T13:37:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.948+0000 7fc4ae873740 1 client.26417 shutdown 2024-04-22T13:37:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.949+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 msgr2=0x56312fe5a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:52.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.949+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00a58a0 tx=0x7fc4a00a5930 comp rx=0 tx=0).stop 2024-04-22T13:37:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 shutdown_connections 2024-04-22T13:37:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fe6c780 0x56312fe6eb90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe5a420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe59ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 --2- 172.21.15.87:0/3952100231 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe53580 0x56312fe599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 >> 172.21.15.87:0/3952100231 conn(0x56312fe4dcb0 msgr2=0x56312fe4fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc48d7fa640 0 client.26417 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 shutdown_connections 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.951+0000 7fc4ae873740 1 -- 172.21.15.87:0/3952100231 wait complete. 2024-04-22T13:37:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:52.952+0000 7fc4ae873740 20 client.26417 trim_cache size 0 max 16384 2024-04-22T13:37:53.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.021+0000 7fc4ae873740 1 Processor -- start 2024-04-22T13:37:53.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.032+0000 7fc4ae873740 1 -- start start 2024-04-22T13:37:53.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53ea0 0x56312fe5a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe4cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc488010fb0 con 0x56312fe53580 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc48800e880 con 0x56312fe53ea0 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4ae873740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc48800c6d0 con 0x56312fe5ca80 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.033+0000 7fc4a9a1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41416/0 (socket says 172.21.15.87:41416) 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2892612523 learned_addr learned my addr 172.21.15.87:0/2892612523 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe4cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a921c640 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53ea0 0x56312fe5a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2892612523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 msgr2=0x56312fe4cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:53.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe4cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2892612523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53ea0 msgr2=0x56312fe5a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:53.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53ea0 0x56312fe5a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.034+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2892612523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56312fe4dae0 con 0x56312fe53580 2024-04-22T13:37:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.039+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc4940058c0 tx=0x7fc494012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab28bdd852ef2531 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.041+0000 7fc48f7fe640 1 -- 172.21.15.87:0/2892612523 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49400fba0 con 0x56312fe53580 2024-04-22T13:37:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.041+0000 7fc48f7fe640 1 -- 172.21.15.87:0/2892612523 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc49402b070 con 0x56312fe53580 2024-04-22T13:37:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 -- 172.21.15.87:0/2892612523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 msgr2=0x56312fe686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc4940058c0 tx=0x7fc494012000 comp rx=0 tx=0).stop 2024-04-22T13:37:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 -- 172.21.15.87:0/2892612523 shutdown_connections 2024-04-22T13:37:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe5ca80 0x56312fe4cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe53ea0 0x56312fe5a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2892612523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53580 0x56312fe686c0 secure :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc4940058c0 tx=0x7fc494012000 comp rx=0 tx=0).stop 2024-04-22T13:37:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.042+0000 7fc4ae873740 1 -- 172.21.15.87:0/2892612523 >> 172.21.15.87:0/2892612523 conn(0x56312fe4dcb0 msgr2=0x56312fe588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.044+0000 7fc4ae873740 1 -- 172.21.15.87:0/2892612523 shutdown_connections 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.045+0000 7fc4ae873740 1 -- 172.21.15.87:0/2892612523 wait complete. 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 -- start start 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 Processor -- start 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe60ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 0x56312fed1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc48800cd40 con 0x56312fe53ea0 2024-04-22T13:37:53.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc48800e1b0 con 0x56312fe5ca80 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc48800d410 con 0x56312fe60ff0 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.046+0000 7fc4ae873740 10 client.0 _ll_register_callbacks cb 0x7fc4a00b4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.047+0000 7fc4a9a1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.047+0000 7fc4a9a1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41432/0 (socket says 172.21.15.87:41432) 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.047+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2394754406 learned_addr learned my addr 172.21.15.87:0/2394754406 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.047+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 0x56312fed1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.047+0000 7fc4a921c640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe60ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2394754406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 msgr2=0x56312fed1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 0x56312fed1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 msgr2=0x56312fe60ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe60ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a9a1d640 1 -- 172.21.15.87:0/2394754406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56312fed2210 con 0x56312fe53ea0 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4aa21e640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 0x56312fed1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:53.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.048+0000 7fc4a921c640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe60ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:37:53.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.049+0000 7fc4a9a1d640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc494000c00 tx=0x7fc49400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d223bfd7898422d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:37:53.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.058+0000 7fc48d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:37:53.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.058+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494034060 con 0x56312fe53ea0 2024-04-22T13:37:53.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.058+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56312fe54730 con 0x56312fe53ea0 2024-04-22T13:37:53.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.058+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7fc494011b40 con 0x56312fe53ea0 2024-04-22T13:37:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.066+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc494011d40 con 0x56312fe53ea0 2024-04-22T13:37:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.067+0000 7fc4ae873740 10 client.17322 fetch_fsmap learned FSMap version 23 2024-04-22T13:37:53.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.067+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56312fed3450 con 0x56312fe53ea0 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.073+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56312fe54730 con 0x56312fe53ea0 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.073+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc494037020 con 0x56312fe53ea0 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 10 client.17322 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 20 client.17322 populate_metadata read hostname 'smithi087' 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fed3a20 0x56312fed5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 4 client.17322 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:37:53.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.075+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56312fdbfa70 con 0x56312fed3a20 2024-04-22T13:37:53.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.077+0000 7fc4a921c640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fed3a20 0x56312fed5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:37:53.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.085+0000 7fc4a921c640 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fed3a20 0x56312fed5e40 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9b25c3c72f087 server_cookie=185f04d0aaeeb211 in_seq=0 out_seq=0 2024-04-22T13:37:53.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.085+0000 7fc48d7fa640 10 client.17322 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.088+0000 7fc48d7fa640 1 -- 172.21.15.87:0/2394754406 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x56312fdbfa70 con 0x56312fed3a20 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.088+0000 7fc48d7fa640 10 client.17322 handle_command_reply: tid=0 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.088+0000 7fc4ae873740 1 client.17322 shutdown 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.089+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 msgr2=0x56312fe60570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.089+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc494000c00 tx=0x7fc49400fe20 comp rx=0 tx=0).stop 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 shutdown_connections 2024-04-22T13:37:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56312fed3a20 0x56312fed5e40 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56312fe60ff0 0x56312fed1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56312fe5ca80 0x56312fe60ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 --2- 172.21.15.87:0/2394754406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56312fe53ea0 0x56312fe60570 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 >> 172.21.15.87:0/2394754406 conn(0x56312fe4dcb0 msgr2=0x56312fe4cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.091+0000 7fc48d7fa640 0 client.17322 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.092+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 shutdown_connections 2024-04-22T13:37:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.092+0000 7fc4ae873740 1 -- 172.21.15.87:0/2394754406 wait complete. 2024-04-22T13:37:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.093+0000 7fc4ae873740 20 client.17322 trim_cache size 0 max 16384 2024-04-22T13:37:53.096 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:37:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "scrub active (12 inodes in the stack)", 2024-04-22T13:37:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:37:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "48416d74-e6b6-4d56-bd4d-a1ae9f7659f8": { 2024-04-22T13:37:53.096 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:37:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "48416d74-e6b6-4d56-bd4d-a1ae9f7659f8", 2024-04-22T13:37:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:37:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:53.097 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:37:53.097 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:37:53.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.102+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 msgr2=0x7fc4900a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:53.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.102+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 0x7fc4900a6590 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc494005fd0 tx=0x7fc494005850 comp rx=0 tx=0).stop 2024-04-22T13:37:53.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.102+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 msgr2=0x7fc4a41194e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:37:53.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.102+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b3c0 tx=0x7fc4a000b880 comp rx=0 tx=0).stop 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 shutdown_connections 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4900a40d0 0x7fc4900a6590 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a411b730 0x7fc4a41194e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4a41106d0 0x7fc4a41150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 --2- 172.21.15.87:0/1959168795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4a410e7b0 0x7fc4a4114bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:37:53.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 >> 172.21.15.87:0/1959168795 conn(0x7fc4a406d070 msgr2=0x7fc4a4072a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:37:53.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.104+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 shutdown_connections 2024-04-22T13:37:53.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:37:53.105+0000 7fc4abc81640 1 -- 172.21.15.87:0/1959168795 wait complete. 2024-04-22T13:37:53.146 INFO:tasks.cephfs.filesystem:scrub status for tag:48416d74-e6b6-4d56-bd4d-a1ae9f7659f8 - {'path': '/', 'tag': '48416d74-e6b6-4d56-bd4d-a1ae9f7659f8', 'options': 'recursive,force'} 2024-04-22T13:37:53.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:53 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:37:53.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:53 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:53.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:53 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:37:53.570 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:53 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:37:53.570 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:53 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:53.570 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:53 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:37:53.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:53 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:37:53.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:53 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:53.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:53 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:37:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:54 smithi120 ceph-mon[31101]: pgmap v185: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 980 MiB/s wr, 2.83k op/s 2024-04-22T13:37:54.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:54 smithi132 ceph-mon[31531]: pgmap v185: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 980 MiB/s wr, 2.83k op/s 2024-04-22T13:37:54.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:54 smithi087 ceph-mon[27999]: pgmap v185: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 980 MiB/s wr, 2.83k op/s 2024-04-22T13:37:56.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:56 smithi120 ceph-mon[31101]: pgmap v186: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 804 MiB/s wr, 2.28k op/s 2024-04-22T13:37:56.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:56 smithi132 ceph-mon[31531]: pgmap v186: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 804 MiB/s wr, 2.28k op/s 2024-04-22T13:37:56.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:56 smithi087 ceph-mon[27999]: pgmap v186: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 804 MiB/s wr, 2.28k op/s 2024-04-22T13:37:58.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:58 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:58.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:58 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:58.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:58 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:37:59.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:37:59 smithi120 ceph-mon[31101]: pgmap v187: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1000 MiB/s wr, 2.41k op/s 2024-04-22T13:37:59.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:37:59 smithi132 ceph-mon[31531]: pgmap v187: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1000 MiB/s wr, 2.41k op/s 2024-04-22T13:37:59.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:37:59 smithi087 ceph-mon[27999]: pgmap v187: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1000 MiB/s wr, 2.41k op/s 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 -- 172.21.15.87:0/4203886164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 msgr2=0x7f9a001152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 --2- 172.21.15.87:0/4203886164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 0x7f9a001152d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009900 tx=0x7f99fc02bc60 comp rx=0 tx=0).stop 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 -- 172.21.15.87:0/4203886164 shutdown_connections 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 --2- 172.21.15.87:0/4203886164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 0x7f9a001152d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 --2- 172.21.15.87:0/4203886164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a001099b0 0x7f9a0010d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 --2- 172.21.15.87:0/4203886164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a00109000 0x7f9a001093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.294+0000 7f9a06273640 1 -- 172.21.15.87:0/4203886164 >> 172.21.15.87:0/4203886164 conn(0x7f9a00077b30 msgr2=0x7f9a00079f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.295+0000 7f9a06273640 1 -- 172.21.15.87:0/4203886164 shutdown_connections 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.296+0000 7f9a06273640 1 -- 172.21.15.87:0/4203886164 wait complete. 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 Processor -- start 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 -- start start 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 0x7f9a0006dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a00071e00 0x7f9a00072280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0006be40 con 0x7f9a00109000 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0006bcc0 con 0x7f9a00071e00 2024-04-22T13:38:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.297+0000 7f9a06273640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0006bfc0 con 0x7f9a0010d750 2024-04-22T13:38:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.298+0000 7f9a05271640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.298+0000 7f9a05271640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36380/0 (socket says 172.21.15.87:36380) 2024-04-22T13:38:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.298+0000 7f9a05271640 1 -- 172.21.15.87:0/281803636 learned_addr learned my addr 172.21.15.87:0/281803636 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 -- 172.21.15.87:0/281803636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 msgr2=0x7f9a0006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 0x7f9a0006dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 -- 172.21.15.87:0/281803636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a00071e00 msgr2=0x7f9a00072280 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a00071e00 0x7f9a00072280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 -- 172.21.15.87:0/281803636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a000728f0 con 0x7f9a00109000 2024-04-22T13:38:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.299+0000 7f9a05271640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f99f800ba50 tx=0x7f99f800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.322+0000 7f99f67fc640 1 -- 172.21.15.87:0/281803636 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8019070 con 0x7f9a00109000 2024-04-22T13:38:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.322+0000 7f99f67fc640 1 -- 172.21.15.87:0/281803636 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99f800cc10 con 0x7f9a00109000 2024-04-22T13:38:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.322+0000 7f99f67fc640 1 -- 172.21.15.87:0/281803636 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80075d0 con 0x7f9a00109000 2024-04-22T13:38:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.322+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a00072be0 con 0x7f9a00109000 2024-04-22T13:38:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.322+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a0011e4d0 con 0x7f9a00109000 2024-04-22T13:38:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.350+0000 7f9a0a19b740 1 Processor -- start 2024-04-22T13:38:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.351+0000 7f99f67fc640 1 -- 172.21.15.87:0/281803636 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f99f8007830 con 0x7f9a00109000 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.352+0000 7f99f67fc640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 0x7f99ec0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.352+0000 7f99f67fc640 1 -- 172.21.15.87:0/281803636 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f99f80cfb20 con 0x7f9a00109000 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 -- start start 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557d8f92d8d0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557d8f8cae80 con 0x557d8f9c8d50 2024-04-22T13:38:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557d8f929890 con 0x557d8f9d22d0 2024-04-22T13:38:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.353+0000 7f9a05a72640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 0x7f99ec0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:51164/0 (socket says 172.21.15.87:51164) 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 -- 172.21.15.87:0/4003585447 learned_addr learned my addr 172.21.15.87:0/4003585447 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a05a72640 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 -- 172.21.15.87:0/4003585447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 msgr2=0x557d8f9d46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 -- 172.21.15.87:0/4003585447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 msgr2=0x557d8f9c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.355+0000 7f9a04a70640 1 -- 172.21.15.87:0/4003585447 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d8f8ec320 con 0x557d8f9c8d50 2024-04-22T13:38:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.356+0000 7f9a04a70640 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99f000ba90 tx=0x7f99f000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=346e6c84a973e4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.360+0000 7f99eb7fe640 1 -- 172.21.15.87:0/4003585447 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f000dca0 con 0x557d8f9c8d50 2024-04-22T13:38:00.356 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.360+0000 7f99eb7fe640 1 -- 172.21.15.87:0/4003585447 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99f000de40 con 0x557d8f9c8d50 2024-04-22T13:38:00.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.361+0000 7f9a0a19b740 1 -- 172.21.15.87:0/4003585447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 msgr2=0x557d8f9c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:00.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.361+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99f000ba90 tx=0x7f99f000bf50 comp rx=0 tx=0).stop 2024-04-22T13:38:00.357 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.362+0000 7f9a0a19b740 1 -- 172.21.15.87:0/4003585447 shutdown_connections 2024-04-22T13:38:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.362+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.362+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9c91d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.362+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/4003585447 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.362+0000 7f9a0a19b740 1 -- 172.21.15.87:0/4003585447 >> 172.21.15.87:0/4003585447 conn(0x557d8f98fc90 msgr2=0x557d8f9c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.367+0000 7f9a05a72640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 0x7f99ec0a63e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99fc005c20 tx=0x7f99fc005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.368+0000 7f9a0a19b740 1 -- 172.21.15.87:0/4003585447 shutdown_connections 2024-04-22T13:38:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.368+0000 7f9a0a19b740 1 -- 172.21.15.87:0/4003585447 wait complete. 2024-04-22T13:38:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.368+0000 7f9a0a19b740 1 -- start start 2024-04-22T13:38:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.368+0000 7f9a0a19b740 1 Processor -- start 2024-04-22T13:38:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 0x557d8f9d1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557d8f9e6060 con 0x557d8f9c9ac0 2024-04-22T13:38:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557d8f8b5eb0 con 0x557d8f9d22d0 2024-04-22T13:38:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557d8f9e61e0 con 0x557d8f9c8d50 2024-04-22T13:38:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.369+0000 7f9a0a19b740 10 client.0 _ll_register_callbacks cb 0x557d8f8e2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.370+0000 7f9a05a72640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.372+0000 7f9a05a72640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:51176/0 (socket says 172.21.15.87:51176) 2024-04-22T13:38:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.372+0000 7f9a05a72640 1 -- 172.21.15.87:0/2752432571 learned_addr learned my addr 172.21.15.87:0/2752432571 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.372+0000 7f9a05271640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 0x557d8f9d1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.378+0000 7f9a04a70640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 -- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 msgr2=0x557d8f9cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 -- 172.21.15.87:0/2752432571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 msgr2=0x557d8f9cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 -- 172.21.15.87:0/2752432571 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d8f640210 con 0x557d8f9c8d50 2024-04-22T13:38:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.379+0000 7f9a05271640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 0x557d8f9d1bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f99f80cd390 tx=0x7f99f809bca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09ff5d8290dfeec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.383+0000 7f99e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:38:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.389+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80bc070 con 0x557d8f9c8d50 2024-04-22T13:38:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.390+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f99f80cd500 con 0x557d8f9c8d50 2024-04-22T13:38:00.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.391+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557d8f941880 con 0x557d8f9c8d50 2024-04-22T13:38:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.395+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f99f80cd700 con 0x557d8f9c8d50 2024-04-22T13:38:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.395+0000 7f9a0a19b740 10 client.26467 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.395+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d8f70c620 con 0x557d8f9c8d50 2024-04-22T13:38:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557d8f941880 con 0x557d8f9c8d50 2024-04-22T13:38:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f99f8007b20 con 0x557d8f9c8d50 2024-04-22T13:38:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 10 client.26467 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 10 client.26467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 20 client.26467 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9f3900 0x557d8f9f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 4 client.26467 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.400+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d8f8cd3d0 con 0x557d8f9f3900 2024-04-22T13:38:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.404+0000 7f9a04a70640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9f3900 0x557d8f9f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.420+0000 7f9a04a70640 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9f3900 0x557d8f9f5d20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6723115fc2089624 server_cookie=93f493efec68f6af in_seq=0 out_seq=0 2024-04-22T13:38:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.421+0000 7f99e8ff9640 10 client.26467 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.427+0000 7f99e8ff9640 1 -- 172.21.15.87:0/2752432571 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d8f8cd3d0 con 0x557d8f9f3900 2024-04-22T13:38:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.427+0000 7f99e8ff9640 10 client.26467 handle_command_reply: tid=0 2024-04-22T13:38:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.428+0000 7f9a0a19b740 1 client.26467 shutdown 2024-04-22T13:38:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.428+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 msgr2=0x557d8f9d1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.428+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 0x557d8f9d1bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f99f80cd390 tx=0x7f99f809bca0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 shutdown_connections 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9f3900 0x557d8f9f5d20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/2752432571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c8d50 0x557d8f9d1bf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.430+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 >> 172.21.15.87:0/2752432571 conn(0x557d8f98fc90 msgr2=0x557d8f9c5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:00.431 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.433+0000 7f99e8ff9640 0 client.26467 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.434+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 shutdown_connections 2024-04-22T13:38:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.436+0000 7f9a0a19b740 1 -- 172.21.15.87:0/2752432571 wait complete. 2024-04-22T13:38:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.437+0000 7f9a0a19b740 20 client.26467 trim_cache size 0 max 16384 2024-04-22T13:38:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.491+0000 7f9a0a19b740 1 Processor -- start 2024-04-22T13:38:00.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.496+0000 7f9a0a19b740 1 -- start start 2024-04-22T13:38:00.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9ef010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9be970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.494 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f99e4010fb0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99e400e880 con 0x557d8f9c8d50 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.497+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99e400c6d0 con 0x557d8f9d22d0 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.499+0000 7f9a05a72640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.499+0000 7f9a05a72640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:41058/0 (socket says 172.21.15.87:41058) 2024-04-22T13:38:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.499+0000 7f9a05a72640 1 -- 172.21.15.87:0/1257815730 learned_addr learned my addr 172.21.15.87:0/1257815730 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.502+0000 7f9a05271640 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.502+0000 7f9a04a70640 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.503+0000 7f9a05a72640 1 -- 172.21.15.87:0/1257815730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 msgr2=0x557d8f9ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.503+0000 7f9a05a72640 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9ef010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.503+0000 7f9a05a72640 1 -- 172.21.15.87:0/1257815730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 msgr2=0x557d8f9be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.503+0000 7f9a05a72640 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.503+0000 7f9a05a72640 1 -- 172.21.15.87:0/1257815730 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d8f9cf820 con 0x557d8f9d22d0 2024-04-22T13:38:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.506+0000 7f9a05a72640 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99fc02c270 tx=0x7f99fc040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438fb37e9064e9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.510+0000 7f99eaffd640 1 -- 172.21.15.87:0/1257815730 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc04e030 con 0x557d8f9d22d0 2024-04-22T13:38:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.510+0000 7f99eaffd640 1 -- 172.21.15.87:0/1257815730 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99fc043040 con 0x557d8f9d22d0 2024-04-22T13:38:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.510+0000 7f99eaffd640 1 -- 172.21.15.87:0/1257815730 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc03f6d0 con 0x557d8f9d22d0 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 -- 172.21.15.87:0/1257815730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 msgr2=0x557d8f9d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99fc02c270 tx=0x7f99fc040000 comp rx=0 tx=0).stop 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 -- 172.21.15.87:0/1257815730 shutdown_connections 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9d22d0 0x557d8f9d16b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c9ac0 0x557d8f9be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/1257815730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9c8d50 0x557d8f9ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 -- 172.21.15.87:0/1257815730 >> 172.21.15.87:0/1257815730 conn(0x557d8f98fc90 msgr2=0x557d8f9cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 -- 172.21.15.87:0/1257815730 shutdown_connections 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.511+0000 7f9a0a19b740 1 -- 172.21.15.87:0/1257815730 wait complete. 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.512+0000 7f9a0a19b740 1 -- start start 2024-04-22T13:38:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.512+0000 7f9a0a19b740 1 Processor -- start 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c8d50 0x557d8f9eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f99e400cd40 con 0x557d8f9c8d50 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99e400e1b0 con 0x557d8f9d22d0 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99e400d410 con 0x557d8f9c9ac0 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a0a19b740 10 client.0 _ll_register_callbacks cb 0x7f99f8097e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a04a70640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a04a70640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:41068/0 (socket says 172.21.15.87:41068) 2024-04-22T13:38:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.513+0000 7f9a04a70640 1 -- 172.21.15.87:0/3043886876 learned_addr learned my addr 172.21.15.87:0/3043886876 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a05a72640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a04a70640 1 -- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 msgr2=0x557d8f9d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a04a70640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a04a70640 1 -- 172.21.15.87:0/3043886876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c8d50 msgr2=0x557d8f9eb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a04a70640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c8d50 0x557d8f9eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.514+0000 7f9a04a70640 1 -- 172.21.15.87:0/3043886876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d8f9cc200 con 0x557d8f9c9ac0 2024-04-22T13:38:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.517+0000 7f9a04a70640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99f000d770 tx=0x7f99f0003cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3008504623a338b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.518+0000 7f99e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:38:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.518+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00037f0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.518+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99fc043040 con 0x557d8f9c9ac0 2024-04-22T13:38:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.518+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8411+0+0 (secure 0 0 0) 0x7f99f00273c0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.525+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f99f00275c0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.525+0000 7f9a0a19b740 10 client.26473 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.525+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d8f9d29b0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99fc043040 con 0x557d8f9c9ac0 2024-04-22T13:38:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f99f00052c0 con 0x557d8f9c9ac0 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 10 client.26473 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 10 client.26473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 20 client.26473 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9d31b0 0x557d8fa5be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 4 client.26473 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:38:00.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.528+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d8f93b2f0 con 0x557d8f9d31b0 2024-04-22T13:38:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.530+0000 7f9a05271640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9d31b0 0x557d8fa5be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:00.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.550+0000 7f9a05271640 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9d31b0 0x557d8fa5be70 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f663b2343d19ace server_cookie=c09541cf8bd58a83 in_seq=0 out_seq=0 2024-04-22T13:38:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:00.550+0000 7f99e8ff9640 10 client.26473 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:00.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:00 smithi120 ceph-mon[31101]: pgmap v188: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 785 MiB/s wr, 1.87k op/s 2024-04-22T13:38:00.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:00 smithi132 ceph-mon[31531]: pgmap v188: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 785 MiB/s wr, 1.87k op/s 2024-04-22T13:38:00.753 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:00 smithi087 ceph-mon[27999]: pgmap v188: 97 pgs: 97 active+clean; 45 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 785 MiB/s wr, 1.87k op/s 2024-04-22T13:38:01.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:01 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:38:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:01 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:38:01.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:01 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:38:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:02 smithi120 ceph-mon[31101]: pgmap v189: 97 pgs: 97 active+clean; 45 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 777 MiB/s wr, 1.95k op/s 2024-04-22T13:38:02.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:02 smithi132 ceph-mon[31531]: pgmap v189: 97 pgs: 97 active+clean; 45 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 777 MiB/s wr, 1.95k op/s 2024-04-22T13:38:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:02 smithi087 ceph-mon[27999]: pgmap v189: 97 pgs: 97 active+clean; 45 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 777 MiB/s wr, 1.95k op/s 2024-04-22T13:38:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:04 smithi120 ceph-mon[31101]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 957 MiB/s wr, 2.59k op/s 2024-04-22T13:38:04.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:04 smithi132 ceph-mon[31531]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 957 MiB/s wr, 2.59k op/s 2024-04-22T13:38:04.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:04 smithi087 ceph-mon[27999]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 957 MiB/s wr, 2.59k op/s 2024-04-22T13:38:06.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:06 smithi120 ceph-mon[31101]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 795 MiB/s wr, 2.08k op/s 2024-04-22T13:38:06.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:06 smithi132 ceph-mon[31531]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 795 MiB/s wr, 2.08k op/s 2024-04-22T13:38:06.741 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:06 smithi087 ceph-mon[27999]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 795 MiB/s wr, 2.08k op/s 2024-04-22T13:38:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f99e8ff9640 1 -- 172.21.15.87:0/3043886876 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d8f93b2f0 con 0x557d8f9d31b0 2024-04-22T13:38:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f99e8ff9640 10 client.26473 handle_command_reply: tid=0 2024-04-22T13:38:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 client.26473 shutdown 2024-04-22T13:38:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 msgr2=0x557d8f9d37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99f000d770 tx=0x7f99f0003cc0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 shutdown_connections 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557d8f9d31b0 0x557d8fa5be70 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557d8f9d22d0 0x557d8f9d3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557d8f9c9ac0 0x557d8f9d37a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 --2- 172.21.15.87:0/3043886876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557d8f9c8d50 0x557d8f9eb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 >> 172.21.15.87:0/3043886876 conn(0x557d8f98fc90 msgr2=0x557d8f9c2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.600+0000 7f99e8ff9640 0 client.26473 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:07.600 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.605+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 shutdown_connections 2024-04-22T13:38:07.600 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.605+0000 7f9a0a19b740 1 -- 172.21.15.87:0/3043886876 wait complete. 2024-04-22T13:38:07.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.605+0000 7f9a0a19b740 20 client.26473 trim_cache size 0 max 16384 2024-04-22T13:38:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:38:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:38:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:38:07.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:38:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.612+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 msgr2=0x7f99ec0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.612+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 0x7f99ec0a63e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99fc005c20 tx=0x7f99fc005fd0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.612+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 msgr2=0x7f9a0006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.612+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f99f800ba50 tx=0x7f99f800bf10 comp rx=0 tx=0).stop 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 shutdown_connections 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f99ec0a3f20 0x7f99ec0a63e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a00071e00 0x7f9a00072280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a0010d750 0x7f9a0006dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 --2- 172.21.15.87:0/281803636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a00109000 0x7f9a0006d590 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.616+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 >> 172.21.15.87:0/281803636 conn(0x7f9a00077b30 msgr2=0x7f9a0010b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.618+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 shutdown_connections 2024-04-22T13:38:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:07.618+0000 7f9a06273640 1 -- 172.21.15.87:0/281803636 wait complete. 2024-04-22T13:38:07.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:07 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:38:07.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:07 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:38:07.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:07 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:38:08.669 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:08 smithi120 ceph-mon[31101]: pgmap v192: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 990 MiB/s wr, 2.59k op/s 2024-04-22T13:38:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:08 smithi132 ceph-mon[31531]: pgmap v192: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 990 MiB/s wr, 2.59k op/s 2024-04-22T13:38:08.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:08 smithi087 ceph-mon[27999]: pgmap v192: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 990 MiB/s wr, 2.59k op/s 2024-04-22T13:38:09.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:09 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:38:09.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:09 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:38:09.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:09 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:38:09.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:09 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:38:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:09 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:38:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:09 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:38:09.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:09 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:38:09.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:09 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:38:09.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:09 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:38:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:10 smithi120 ceph-mon[31101]: pgmap v193: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 769 MiB/s wr, 2.30k op/s 2024-04-22T13:38:10.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:10 smithi132 ceph-mon[31531]: pgmap v193: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 769 MiB/s wr, 2.30k op/s 2024-04-22T13:38:10.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:10 smithi087 ceph-mon[27999]: pgmap v193: 97 pgs: 97 active+clean; 45 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 769 MiB/s wr, 2.30k op/s 2024-04-22T13:38:12.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:12 smithi120 ceph-mon[31101]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T13:38:12.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:12 smithi120 ceph-mon[31101]: pgmap v195: 97 pgs: 97 active+clean; 45 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 839 MiB/s wr, 2.74k op/s 2024-04-22T13:38:12.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:12 smithi132 ceph-mon[31531]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T13:38:12.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:12 smithi132 ceph-mon[31531]: pgmap v195: 97 pgs: 97 active+clean; 45 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 839 MiB/s wr, 2.74k op/s 2024-04-22T13:38:12.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:12 smithi087 ceph-mon[27999]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T13:38:12.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:12 smithi087 ceph-mon[27999]: pgmap v195: 97 pgs: 97 active+clean; 45 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 839 MiB/s wr, 2.74k op/s 2024-04-22T13:38:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:14 smithi120 ceph-mon[31101]: pgmap v196: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 672 MiB/s wr, 2.49k op/s 2024-04-22T13:38:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:14 smithi132 ceph-mon[31531]: pgmap v196: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 672 MiB/s wr, 2.49k op/s 2024-04-22T13:38:14.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:14 smithi087 ceph-mon[27999]: pgmap v196: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 672 MiB/s wr, 2.49k op/s 2024-04-22T13:38:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:16 smithi120 ceph-mon[31101]: pgmap v197: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 576 MiB/s wr, 2.19k op/s 2024-04-22T13:38:16.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:16 smithi132 ceph-mon[31531]: pgmap v197: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 576 MiB/s wr, 2.19k op/s 2024-04-22T13:38:16.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:16 smithi087 ceph-mon[27999]: pgmap v197: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 576 MiB/s wr, 2.19k op/s 2024-04-22T13:38:18.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:18 smithi120 ceph-mon[31101]: pgmap v198: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 278 MiB/s wr, 1.55k op/s 2024-04-22T13:38:18.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:18 smithi132 ceph-mon[31531]: pgmap v198: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 278 MiB/s wr, 1.55k op/s 2024-04-22T13:38:18.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:18 smithi087 ceph-mon[27999]: pgmap v198: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 278 MiB/s wr, 1.55k op/s 2024-04-22T13:38:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- 172.21.15.87:0/630801996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 msgr2=0x7f55281014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 --2- 172.21.15.87:0/630801996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f55281014d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5524009900 tx=0x7f552402bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- 172.21.15.87:0/630801996 shutdown_connections 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 --2- 172.21.15.87:0/630801996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 0x7f552810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 --2- 172.21.15.87:0/630801996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5528101aa0 0x7f5528105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 --2- 172.21.15.87:0/630801996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f55281014d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- 172.21.15.87:0/630801996 >> 172.21.15.87:0/630801996 conn(0x7f5528077fd0 msgr2=0x7f55280783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- 172.21.15.87:0/630801996 shutdown_connections 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- 172.21.15.87:0/630801996 wait complete. 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 Processor -- start 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.744+0000 7f553006d640 1 -- start start 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 0x7f5528198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 0x7f552819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f55281110d0 con 0x7f55281010f0 2024-04-22T13:38:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5528110f50 con 0x7f5528106170 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5528111250 con 0x7f552819cff0 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57200/0 (socket says 172.21.15.87:57200) 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 -- 172.21.15.87:0/3545116378 learned_addr learned my addr 172.21.15.87:0/3545116378 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552e60a640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 0x7f552819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552d608640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 0x7f5528198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 -- 172.21.15.87:0/3545116378 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 msgr2=0x7f552819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 0x7f552819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 -- 172.21.15.87:0/3545116378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 msgr2=0x7f5528198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 0x7f5528198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 -- 172.21.15.87:0/3545116378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552819dae0 con 0x7f55281010f0 2024-04-22T13:38:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552e60a640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 0x7f552819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f552de09640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5524005940 tx=0x7f55240045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f551effd640 1 -- 172.21.15.87:0/3545116378 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524043020 con 0x7f55281010f0 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f551effd640 1 -- 172.21.15.87:0/3545116378 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f552402caf0 con 0x7f55281010f0 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.745+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552819dd70 con 0x7f55281010f0 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.746+0000 7f551effd640 1 -- 172.21.15.87:0/3545116378 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524003af0 con 0x7f55281010f0 2024-04-22T13:38:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.746+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281a5590 con 0x7f55281010f0 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.748+0000 7f551effd640 1 -- 172.21.15.87:0/3545116378 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5524046020 con 0x7f55281010f0 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.748+0000 7f5532c63740 1 Processor -- start 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.748+0000 7f5532c63740 1 -- start start 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.748+0000 7f551effd640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 0x7f55140a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc875620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc875b60 0x55cfcc875fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc7da8d0 con 0x55cfcc87f0e0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc777e80 con 0x55cfcc8768d0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f5532c63740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc7d6890 con 0x55cfcc875b60 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 0x7f55140a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f551effd640 1 -- 172.21.15.87:0/3545116378 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f55240f28f0 con 0x7f55281010f0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57214/0 (socket says 172.21.15.87:57214) 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 -- 172.21.15.87:0/966886572 learned_addr learned my addr 172.21.15.87:0/966886572 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc875620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552d608640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc875b60 0x55cfcc875fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 -- 172.21.15.87:0/966886572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc875b60 msgr2=0x55cfcc875fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc875b60 0x55cfcc875fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 -- 172.21.15.87:0/966886572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 msgr2=0x55cfcc8814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 -- 172.21.15.87:0/966886572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfcc799320 con 0x55cfcc8768d0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552e60a640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 0x7f55140a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5520012be0 tx=0x7f5520009db0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.749+0000 7f552de09640 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc875620 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f55240f0190 tx=0x7f55240badb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6bce8317da6e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5513fff640 1 -- 172.21.15.87:0/966886572 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240a4e30 con 0x55cfcc8768d0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5513fff640 1 -- 172.21.15.87:0/966886572 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55240bf0e0 con 0x55cfcc8768d0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5513fff640 1 -- 172.21.15.87:0/966886572 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240bcdc0 con 0x55cfcc8768d0 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 -- 172.21.15.87:0/966886572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 msgr2=0x55cfcc875620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc875620 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f55240f0190 tx=0x7f55240badb0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 -- 172.21.15.87:0/966886572 shutdown_connections 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc875b60 0x55cfcc875fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 --2- 172.21.15.87:0/966886572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc875620 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.750+0000 7f5532c63740 1 -- 172.21.15.87:0/966886572 >> 172.21.15.87:0/966886572 conn(0x55cfcc83cc90 msgr2=0x55cfcc870850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- 172.21.15.87:0/966886572 shutdown_connections 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- 172.21.15.87:0/966886572 wait complete. 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- start start 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 Processor -- start 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc885fe0 con 0x55cfcc875b60 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc762eb0 con 0x55cfcc8768d0 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cfcc886160 con 0x55cfcc87f0e0 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.751+0000 7f5532c63740 10 client.0 _ll_register_callbacks cb 0x55cfcc78fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:20.755 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57226/0 (socket says 172.21.15.87:57226) 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 -- 172.21.15.87:0/2113305549 learned_addr learned my addr 172.21.15.87:0/2113305549 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552e60a640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552d608640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 -- 172.21.15.87:0/2113305549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 msgr2=0x55cfcc87c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 -- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 msgr2=0x55cfcc87be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 -- 172.21.15.87:0/2113305549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfcc4ed210 con 0x55cfcc875b60 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f552de09640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5524005ec0 tx=0x7f5524005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a071455e575a846 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f55117fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240be9e0 con 0x55cfcc875b60 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f55240053b0 con 0x55cfcc875b60 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55240bf0e0 con 0x55cfcc875b60 2024-04-22T13:38:20.756 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f55240a4e30 con 0x55cfcc875b60 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f5532c63740 10 client.17346 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.752+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cfcc5b9620 con 0x55cfcc875b60 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55240bf0e0 con 0x55cfcc875b60 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f55240a2800 con 0x55cfcc875b60 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 10 client.17346 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 10 client.17346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 20 client.17346 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc893080 0x55cfcc895470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 4 client.17346 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfcc7ee880 con 0x55cfcc893080 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f552d608640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc893080 0x55cfcc895470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f55117fa640 10 client.17346 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.753+0000 7f552d608640 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc893080 0x55cfcc895470 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfd54f785b57186b server_cookie=172b5df3639d1c4e in_seq=0 out_seq=0 2024-04-22T13:38:20.757 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f55117fa640 1 -- 172.21.15.87:0/2113305549 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfcc7ee880 con 0x55cfcc893080 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f55117fa640 10 client.17346 handle_command_reply: tid=0 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 client.17346 shutdown 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 msgr2=0x55cfcc87ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5524005ec0 tx=0x7f5524005750 comp rx=0 tx=0).stop 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 shutdown_connections 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc893080 0x55cfcc895470 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f55117fa640 0 client.17346 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 --2- 172.21.15.87:0/2113305549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc87ead0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 >> 172.21.15.87:0/2113305549 conn(0x55cfcc83cc90 msgr2=0x55cfcc8725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.754+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 shutdown_connections 2024-04-22T13:38:20.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.755+0000 7f5532c63740 1 -- 172.21.15.87:0/2113305549 wait complete. 2024-04-22T13:38:20.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.755+0000 7f5532c63740 20 client.17346 trim_cache size 0 max 16384 2024-04-22T13:38:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.769+0000 7f5532c63740 1 Processor -- start 2024-04-22T13:38:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.769+0000 7f5532c63740 1 -- start start 2024-04-22T13:38:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.769+0000 7f5532c63740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.769+0000 7f5532c63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.769+0000 7f5532c63740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f5532c63740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f55281ae060 con 0x55cfcc875b60 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f5532c63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55281ab930 con 0x55cfcc8768d0 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f5532c63740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f55281a97a0 con 0x55cfcc87f0e0 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57228/0 (socket says 172.21.15.87:57228) 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 -- 172.21.15.87:0/2108572573 learned_addr learned my addr 172.21.15.87:0/2108572573 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 -- 172.21.15.87:0/2108572573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 msgr2=0x55cfcc87ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 -- 172.21.15.87:0/2108572573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 msgr2=0x55cfcc87e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 -- 172.21.15.87:0/2108572573 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfcc875990 con 0x55cfcc875b60 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f552de09640 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f5524006010 tx=0x7f55240be880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa6577844db683b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f55137fe640 1 -- 172.21.15.87:0/2108572573 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240bac80 con 0x55cfcc875b60 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f55137fe640 1 -- 172.21.15.87:0/2108572573 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55240bf0e0 con 0x55cfcc875b60 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.770+0000 7f55137fe640 1 -- 172.21.15.87:0/2108572573 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240c7650 con 0x55cfcc875b60 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- 172.21.15.87:0/2108572573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 msgr2=0x55cfcc88ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f5524006010 tx=0x7f55240be880 comp rx=0 tx=0).stop 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- 172.21.15.87:0/2108572573 shutdown_connections 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc87ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc87e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 --2- 172.21.15.87:0/2108572573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88ede0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- 172.21.15.87:0/2108572573 >> 172.21.15.87:0/2108572573 conn(0x55cfcc83cc90 msgr2=0x55cfcc870380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- 172.21.15.87:0/2108572573 shutdown_connections 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- 172.21.15.87:0/2108572573 wait complete. 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 -- start start 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.771+0000 7f5532c63740 1 Processor -- start 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc88b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f55281a9df0 con 0x55cfcc875b60 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55281ab260 con 0x55cfcc8768d0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f55281aa4c0 con 0x55cfcc87f0e0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f5532c63740 10 client.0 _ll_register_callbacks cb 0x7f55200126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57240/0 (socket says 172.21.15.87:57240) 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 -- 172.21.15.87:0/1749771204 learned_addr learned my addr 172.21.15.87:0/1749771204 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552d608640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc88b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552e60a640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 -- 172.21.15.87:0/1749771204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 msgr2=0x55cfcc8802f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8802f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 -- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 msgr2=0x55cfcc88b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc88b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 -- 172.21.15.87:0/1749771204 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfcc8957e0 con 0x55cfcc875b60 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f552de09640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f55240bcc90 tx=0x7f55240bb2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185e963ff23a8926 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f55117fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.772+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240bac80 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f55240a9110 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55240bf0e0 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f55240c7650 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 10 client.17352 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cfcc8957e0 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55240bf0e0 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f55240ba8d0 con 0x55cfcc875b60 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 10 client.17352 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 10 client.17352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 20 client.17352 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc8f9db0 0x55cfcc8fc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 4 client.17352 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfcc77a3d0 con 0x55cfcc8f9db0 2024-04-22T13:38:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.773+0000 7f552d608640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc8f9db0 0x55cfcc8fc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.774+0000 7f552d608640 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc8f9db0 0x55cfcc8fc1a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=decd062f51718d42 server_cookie=c1d0598ea41125fc in_seq=0 out_seq=0 2024-04-22T13:38:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.774+0000 7f55117fa640 10 client.17352 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f55117fa640 1 -- 172.21.15.87:0/1749771204 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfcc77a3d0 con 0x55cfcc8f9db0 2024-04-22T13:38:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f55117fa640 10 client.17352 handle_command_reply: tid=0 2024-04-22T13:38:20.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 client.17352 shutdown 2024-04-22T13:38:20.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 msgr2=0x55cfcc88b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.788 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f55240bcc90 tx=0x7f55240bb2d0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 shutdown_connections 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cfcc8f9db0 0x55cfcc8fc1a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfcc87f0e0 0x55cfcc8802f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfcc8768d0 0x55cfcc88b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 --2- 172.21.15.87:0/1749771204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cfcc875b60 0x55cfcc88b2f0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 >> 172.21.15.87:0/1749771204 conn(0x55cfcc83cc90 msgr2=0x55cfcc86f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.792+0000 7f55117fa640 0 client.17352 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.793+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 shutdown_connections 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.793+0000 7f5532c63740 1 -- 172.21.15.87:0/1749771204 wait complete. 2024-04-22T13:38:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.793+0000 7f5532c63740 20 client.17352 trim_cache size 0 max 16384 2024-04-22T13:38:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:38:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:38:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:38:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:38:20.790 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 msgr2=0x7f55140a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 0x7f55140a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5520012be0 tx=0x7f5520009db0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 msgr2=0x7f5528198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5524005940 tx=0x7f55240045e0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 shutdown_connections 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f55140a3dc0 0x7f55140a6280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f552819cff0 0x7f552819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5528106170 0x7f5528198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 --2- 172.21.15.87:0/3545116378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55281010f0 0x7f5528198780 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 >> 172.21.15.87:0/3545116378 conn(0x7f5528077fd0 msgr2=0x7f5528069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 shutdown_connections 2024-04-22T13:38:20.791 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:20.794+0000 7f553006d640 1 -- 172.21.15.87:0/3545116378 wait complete. 2024-04-22T13:38:20.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:20 smithi120 ceph-mon[31101]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T13:38:20.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:20 smithi120 ceph-mon[31101]: pgmap v200: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 195 MiB/s wr, 1.46k op/s 2024-04-22T13:38:20.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:20 smithi132 ceph-mon[31531]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T13:38:20.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:20 smithi132 ceph-mon[31531]: pgmap v200: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 195 MiB/s wr, 1.46k op/s 2024-04-22T13:38:20.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:20 smithi087 ceph-mon[27999]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T13:38:20.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:20 smithi087 ceph-mon[27999]: pgmap v200: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 195 MiB/s wr, 1.46k op/s 2024-04-22T13:38:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:22 smithi120 ceph-mon[31101]: pgmap v201: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 151 MiB/s wr, 1.22k op/s 2024-04-22T13:38:22.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:22 smithi132 ceph-mon[31531]: pgmap v201: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 151 MiB/s wr, 1.22k op/s 2024-04-22T13:38:22.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:22 smithi087 ceph-mon[27999]: pgmap v201: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 151 MiB/s wr, 1.22k op/s 2024-04-22T13:38:23.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:38:23.147 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:38:23.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- 172.21.15.87:0/2023514376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c1031f0 msgr2=0x7f474c1035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 --2- 172.21.15.87:0/2023514376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c1031f0 0x7f474c1035d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f473c009900 tx=0x7f473c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- 172.21.15.87:0/2023514376 shutdown_connections 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 --2- 172.21.15.87:0/2023514376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c10c530 0x7f474c10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 --2- 172.21.15.87:0/2023514376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c103b10 0x7f474c10bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 --2- 172.21.15.87:0/2023514376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c1031f0 0x7f474c1035d0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- 172.21.15.87:0/2023514376 >> 172.21.15.87:0/2023514376 conn(0x7f474c0771d0 msgr2=0x7f474c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- 172.21.15.87:0/2023514376 shutdown_connections 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- 172.21.15.87:0/2023514376 wait complete. 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 Processor -- start 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 -- start start 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.257+0000 7f47513e0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c1031f0 0x7f474c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 0x7f474c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1121c0 con 0x7f474c103b10 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f474c112040 con 0x7f474c10c530 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f474c112340 con 0x7f474c1031f0 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57254/0 (socket says 172.21.15.87:57254) 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 -- 172.21.15.87:0/1047345796 learned_addr learned my addr 172.21.15.87:0/1047345796 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474affd640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c1031f0 0x7f474c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474b7fe640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 0x7f474c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 -- 172.21.15.87:0/1047345796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c1031f0 msgr2=0x7f474c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c1031f0 0x7f474c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 -- 172.21.15.87:0/1047345796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 msgr2=0x7f474c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 0x7f474c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 -- 172.21.15.87:0/1047345796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c19d700 con 0x7f474c103b10 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474b7fe640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 0x7f474c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f474a7fc640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f474000c890 tx=0x7f474000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f4733fff640 1 -- 172.21.15.87:0/1047345796 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740004220 con 0x7f474c103b10 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c19d9f0 con 0x7f474c103b10 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f4733fff640 1 -- 172.21.15.87:0/1047345796 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47400043c0 con 0x7f474c103b10 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f4733fff640 1 -- 172.21.15.87:0/1047345796 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740004a00 con 0x7f474c103b10 2024-04-22T13:38:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.258+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c1a5280 con 0x7f474c103b10 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4733fff640 1 -- 172.21.15.87:0/1047345796 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47400079b0 con 0x7f474c103b10 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 Processor -- start 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 -- start start 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9f580 0x55669dc9f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55669dc041e0 con 0x55669dca8a80 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55669dba1390 con 0x55669dc9fea0 2024-04-22T13:38:23.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.261+0000 7f4753fcd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55669dc001a0 con 0x55669dc9f580 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57270/0 (socket says 172.21.15.87:57270) 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474a7fc640 1 -- 172.21.15.87:0/2363688373 learned_addr learned my addr 172.21.15.87:0/2363688373 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f4733fff640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 0x7f47340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474b7fe640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9f580 0x55669dc9f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474b7fe640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 0x7f47340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 -- 172.21.15.87:0/2363688373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9f580 msgr2=0x55669dc9f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9f580 0x55669dc9f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 -- 172.21.15.87:0/2363688373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 msgr2=0x55669dcaae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f4733fff640 1 -- 172.21.15.87:0/1047345796 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f47400d0510 con 0x7f474c103b10 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 -- 172.21.15.87:0/2363688373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55669d9e07d0 con 0x55669dc9fea0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474a7fc640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474affd640 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca8540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf267804f08c11b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f474b7fe640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 0x7f47340a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4744005c40 tx=0x7f4744009280 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f47317fa640 1 -- 172.21.15.87:0/2363688373 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c004230 con 0x55669dc9fea0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f47317fa640 1 -- 172.21.15.87:0/2363688373 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c02cdc0 con 0x55669dc9fea0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.262+0000 7f47317fa640 1 -- 172.21.15.87:0/2363688373 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c004920 con 0x55669dc9fea0 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- 172.21.15.87:0/2363688373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 msgr2=0x55669dca8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca8540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c004040 comp rx=0 tx=0).stop 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- 172.21.15.87:0/2363688373 shutdown_connections 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dca8a80 0x55669dcaae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca8540 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2363688373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9f580 0x55669dc9f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- 172.21.15.87:0/2363688373 >> 172.21.15.87:0/2363688373 conn(0x55669dc99cb0 msgr2=0x55669dc9a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- 172.21.15.87:0/2363688373 shutdown_connections 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- 172.21.15.87:0/2363688373 wait complete. 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 -- start start 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.263+0000 7f4753fcd740 1 Processor -- start 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dca6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55669dcab810 con 0x55669dc9f580 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55669dbbd2f0 con 0x55669dc9fea0 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55669dcab990 con 0x55669dca8a80 2024-04-22T13:38:23.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 10 client.0 _ll_register_callbacks cb 0x55669dbca580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57276/0 (socket says 172.21.15.87:57276) 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 -- 172.21.15.87:0/2872603698 learned_addr learned my addr 172.21.15.87:0/2872603698 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474a7fc640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474b7fe640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dca6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 -- 172.21.15.87:0/2872603698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 msgr2=0x55669dca6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dca6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 -- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 msgr2=0x55669dca5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 -- 172.21.15.87:0/2872603698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55669d9163f0 con 0x55669dc9f580 2024-04-22T13:38:23.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474b7fe640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dca6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474a7fc640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f474affd640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f473c005560 tx=0x7f473c02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5617cdb31260da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f472eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c04e070 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f473c003a00 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.264+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473c02cdc0 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f473c05a050 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 10 client.17364 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55669d8d60f0 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c02cdc0 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f473c034b70 con 0x55669dc9f580 2024-04-22T13:38:23.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 10 client.17364 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 20 client.17364 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dcb8860 0x55669dcbac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 4 client.17364 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55669dc11c00 con 0x55669dcb8860 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f474a7fc640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dcb8860 0x55669dcbac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f472eff5640 10 client.17364 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.265+0000 7f474a7fc640 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dcb8860 0x55669dcbac50 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f611d0743ed5adb8 server_cookie=b0ade8ca9067ac74 in_seq=0 out_seq=0 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f472eff5640 1 -- 172.21.15.87:0/2872603698 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55669dc11c00 con 0x55669dcb8860 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f472eff5640 10 client.17364 handle_command_reply: tid=0 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 client.17364 shutdown 2024-04-22T13:38:23.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 msgr2=0x55669dca5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f473c005560 tx=0x7f473c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 shutdown_connections 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dcb8860 0x55669dcbac50 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dca6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f472eff5640 0 client.17364 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dca5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 --2- 172.21.15.87:0/2872603698 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca5a80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 >> 172.21.15.87:0/2872603698 conn(0x55669dc99cb0 msgr2=0x55669dc9ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 shutdown_connections 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 1 -- 172.21.15.87:0/2872603698 wait complete. 2024-04-22T13:38:23.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.266+0000 7f4753fcd740 20 client.17364 trim_cache size 0 max 16384 2024-04-22T13:38:23.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.283+0000 7f4753fcd740 1 Processor -- start 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 -- start start 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dcb46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dcc0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1b0330 con 0x55669dc9f580 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1adc00 con 0x55669dc9fea0 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4753fcd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1aba70 con 0x55669dca8a80 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57292/0 (socket says 172.21.15.87:57292) 2024-04-22T13:38:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 -- 172.21.15.87:0/3171181965 learned_addr learned my addr 172.21.15.87:0/3171181965 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 -- 172.21.15.87:0/3171181965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 msgr2=0x55669dcc0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474b7fe640 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dcc0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474a7fc640 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dcb46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dcc0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 -- 172.21.15.87:0/3171181965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 msgr2=0x55669dcb46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dcb46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 -- 172.21.15.87:0/3171181965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55669dca1510 con 0x55669dc9f580 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f474affd640 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f473c0092a0 tx=0x7f473c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58d54390cf6ef04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4730ff9640 1 -- 172.21.15.87:0/3171181965 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c04e070 con 0x55669dc9f580 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4730ff9640 1 -- 172.21.15.87:0/3171181965 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c03fe50 con 0x55669dc9f580 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.284+0000 7f4730ff9640 1 -- 172.21.15.87:0/3171181965 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c003cb0 con 0x55669dc9f580 2024-04-22T13:38:23.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- 172.21.15.87:0/3171181965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 msgr2=0x55669dca81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f473c0092a0 tx=0x7f473c005e80 comp rx=0 tx=0).stop 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- 172.21.15.87:0/3171181965 shutdown_connections 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dca8a80 0x55669dcc0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dc9fea0 0x55669dcb46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- 172.21.15.87:0/3171181965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dca81b0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- 172.21.15.87:0/3171181965 >> 172.21.15.87:0/3171181965 conn(0x55669dc99cb0 msgr2=0x55669dca0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- 172.21.15.87:0/3171181965 shutdown_connections 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- 172.21.15.87:0/3171181965 wait complete. 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- start start 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 Processor -- start 2024-04-22T13:38:23.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9fea0 0x55669dcacd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dca8a80 0x55669dcad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1ac0c0 con 0x55669dc9f580 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1ad530 con 0x55669dca8a80 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.285+0000 7f4753fcd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1ac790 con 0x55669dc9fea0 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f4753fcd740 10 client.0 _ll_register_callbacks cb 0x7f47400d3f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57296/0 (socket says 172.21.15.87:57296) 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 -- 172.21.15.87:0/1770447961 learned_addr learned my addr 172.21.15.87:0/1770447961 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474b7fe640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dca8a80 0x55669dcad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 -- 172.21.15.87:0/1770447961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9fea0 msgr2=0x55669dcacd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9fea0 0x55669dcacd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 -- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dca8a80 msgr2=0x55669dcad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dca8a80 0x55669dcad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 -- 172.21.15.87:0/1770447961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55669dd1dc90 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f474affd640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f473c03f450 tx=0x7f473c02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2265733bdb7e706 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f472eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c057030 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f473c03fba0 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473c03fe50 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f473c04e040 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f4753fcd740 10 client.17370 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55669dd1ef80 con 0x55669dc9f580 2024-04-22T13:38:23.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.286+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c03fe50 con 0x55669dc9f580 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f473c0347b0 con 0x55669dc9f580 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 10 client.17370 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 20 client.17370 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dd1f5c0 0x55669dd219e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 4 client.17370 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55669dc0ba70 con 0x55669dd1f5c0 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f474a7fc640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dd1f5c0 0x55669dd219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f472eff5640 10 client.17370 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f474a7fc640 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dd1f5c0 0x55669dd219e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3861e48a6473381 server_cookie=7cf28adabb2bfcb7 in_seq=0 out_seq=0 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f472eff5640 1 -- 172.21.15.87:0/1770447961 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55669dc0ba70 con 0x55669dd1f5c0 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f472eff5640 10 client.17370 handle_command_reply: tid=0 2024-04-22T13:38:23.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.287+0000 7f4753fcd740 1 client.17370 shutdown 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 msgr2=0x55669dcac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f473c03f450 tx=0x7f473c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 shutdown_connections 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55669dd1f5c0 0x55669dd219e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55669dca8a80 0x55669dcad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55669dc9fea0 0x55669dcacd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f472eff5640 0 client.17370 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 --2- 172.21.15.87:0/1770447961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55669dc9f580 0x55669dcac7e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 >> 172.21.15.87:0/1770447961 conn(0x55669dc99cb0 msgr2=0x55669dc9afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 shutdown_connections 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 1 -- 172.21.15.87:0/1770447961 wait complete. 2024-04-22T13:38:23.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f4753fcd740 20 client.17370 trim_cache size 0 max 16384 2024-04-22T13:38:23.293 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:23.293 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:38:23.293 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:38:23.293 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 msgr2=0x7f47340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 0x7f47340a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4744005c40 tx=0x7f4744009280 comp rx=0 tx=0).stop 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 msgr2=0x7f474c198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f474000c890 tx=0x7f474000cd50 comp rx=0 tx=0).stop 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 shutdown_connections 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f47340a3d20 0x7f47340a61e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f474c10c530 0x7f474c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f474c103b10 0x7f474c198ca0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 --2- 172.21.15.87:0/1047345796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f474c1031f0 0x7f474c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 >> 172.21.15.87:0/1047345796 conn(0x7f474c0771d0 msgr2=0x7f474c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 shutdown_connections 2024-04-22T13:38:23.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.288+0000 7f47513e0640 1 -- 172.21.15.87:0/1047345796 wait complete. 2024-04-22T13:38:23.296 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:38:23.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 -- 172.21.15.87:0/1302878023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101ad0 msgr2=0x7f8a6410dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 --2- 172.21.15.87:0/1302878023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101ad0 0x7f8a6410dcb0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009ca0 tx=0x7f8a5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 -- 172.21.15.87:0/1302878023 shutdown_connections 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 --2- 172.21.15.87:0/1302878023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101ad0 0x7f8a6410dcb0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 --2- 172.21.15.87:0/1302878023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101130 0x7f8a64101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.398+0000 7f8a69c55640 1 --2- 172.21.15.87:0/1302878023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 0x7f8a641072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- 172.21.15.87:0/1302878023 >> 172.21.15.87:0/1302878023 conn(0x7f8a64077e30 msgr2=0x7f8a64078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- 172.21.15.87:0/1302878023 shutdown_connections 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- 172.21.15.87:0/1302878023 wait complete. 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 Processor -- start 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- start start 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101ad0 0x7f8a64199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 0x7f8a641a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8a641110d0 con 0x7f8a64101130 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8a64110f50 con 0x7f8a64106f10 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.399+0000 7f8a69c55640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8a64111250 con 0x7f8a64101ad0 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57316/0 (socket says 172.21.15.87:57316) 2024-04-22T13:38:23.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 -- 172.21.15.87:0/3900068639 learned_addr learned my addr 172.21.15.87:0/3900068639 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a63fff640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 0x7f8a641a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a62ffd640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101ad0 0x7f8a64199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 -- 172.21.15.87:0/3900068639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101ad0 msgr2=0x7f8a64199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101ad0 0x7f8a64199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 -- 172.21.15.87:0/3900068639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 msgr2=0x7f8a641a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 0x7f8a641a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 -- 172.21.15.87:0/3900068639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a641a4af0 con 0x7f8a64101130 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a637fe640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400e9f0 tx=0x7f8a5400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5400ccc0 con 0x7f8a64101130 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a540108e0 con 0x7f8a64101130 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a641a4d20 con 0x7f8a64101130 2024-04-22T13:38:23.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54018880 con 0x7f8a64101130 2024-04-22T13:38:23.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.400+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a641a51e0 con 0x7f8a64101130 2024-04-22T13:38:23.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.401+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a64069230 con 0x7f8a64101130 2024-04-22T13:38:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.402+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a54010430 con 0x7f8a64101130 2024-04-22T13:38:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.403+0000 7f8a60ff9640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 0x7f8a4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.403+0000 7f8a62ffd640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 0x7f8a4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.403+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f8a540d0fc0 con 0x7f8a64101130 2024-04-22T13:38:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.404+0000 7f8a62ffd640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 0x7f8a4c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a6406b1f0 tx=0x7f8a5800a2d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.407+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a5409d640 con 0x7f8a64101130 2024-04-22T13:38:23.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.622+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a6406ac90 con 0x7f8a64101130 2024-04-22T13:38:23.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.623+0000 7f8a60ff9640 1 -- 172.21.15.87:0/3900068639 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f8a5409cf40 con 0x7f8a64101130 2024-04-22T13:38:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:38:23.619 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:38:23.619 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 msgr2=0x7f8a4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 0x7f8a4c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a6406b1f0 tx=0x7f8a5800a2d0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 msgr2=0x7f8a64199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400e9f0 tx=0x7f8a5400edf0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 shutdown_connections 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8a4c0a3d20 0x7f8a4c0a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a64106f10 0x7f8a641a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a64101ad0 0x7f8a64199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 --2- 172.21.15.87:0/3900068639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a64101130 0x7f8a64199530 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 >> 172.21.15.87:0/3900068639 conn(0x7f8a64077e30 msgr2=0x7f8a640ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 shutdown_connections 2024-04-22T13:38:23.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.625+0000 7f8a69c55640 1 -- 172.21.15.87:0/3900068639 wait complete. 2024-04-22T13:38:23.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:38:23.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 -- 172.21.15.87:0/2160241253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac101090 msgr2=0x7f75ac101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/2160241253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac101090 0x7f75ac101470 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a002bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:23.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 -- 172.21.15.87:0/2160241253 shutdown_connections 2024-04-22T13:38:23.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/2160241253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac106160 0x7f75ac10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/2160241253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101a40 0x7f75ac105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/2160241253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac101090 0x7f75ac101470 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 -- 172.21.15.87:0/2160241253 >> 172.21.15.87:0/2160241253 conn(0x7f75ac077f50 msgr2=0x7f75ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 -- 172.21.15.87:0/2160241253 shutdown_connections 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.737+0000 7f75b2ceb640 1 -- 172.21.15.87:0/2160241253 wait complete. 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 Processor -- start 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 -- start start 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101090 0x7f75ac1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac19a090 0x7f75ac1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1110a0 con 0x7f75ac106160 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac110f20 con 0x7f75ac101090 2024-04-22T13:38:23.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b2ceb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac111220 con 0x7f75ac19a090 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57334/0 (socket says 172.21.15.87:57334) 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 -- 172.21.15.87:0/984482432 learned_addr learned my addr 172.21.15.87:0/984482432 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b0a87640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101090 0x7f75ac1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75b1288640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac19a090 0x7f75ac1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 -- 172.21.15.87:0/984482432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac19a090 msgr2=0x7f75ac1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac19a090 0x7f75ac1a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 -- 172.21.15.87:0/984482432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101090 msgr2=0x7f75ac1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101090 0x7f75ac1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 -- 172.21.15.87:0/984482432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac1a4e10 con 0x7f75ac106160 2024-04-22T13:38:23.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.738+0000 7f75abfff640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7598005fd0 tx=0x7f759800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.739+0000 7f75a9ffb640 1 -- 172.21.15.87:0/984482432 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598017070 con 0x7f75ac106160 2024-04-22T13:38:23.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.739+0000 7f75a9ffb640 1 -- 172.21.15.87:0/984482432 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7598009c30 con 0x7f75ac106160 2024-04-22T13:38:23.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.739+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac1a5100 con 0x7f75ac106160 2024-04-22T13:38:23.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.739+0000 7f75a9ffb640 1 -- 172.21.15.87:0/984482432 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598003d30 con 0x7f75ac106160 2024-04-22T13:38:23.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.739+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac1a5650 con 0x7f75ac106160 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.741+0000 7f75a9ffb640 1 -- 172.21.15.87:0/984482432 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7598003ed0 con 0x7f75ac106160 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.741+0000 7f75b59b8740 1 Processor -- start 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.741+0000 7f75b59b8740 1 -- start start 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 0x556ecbe55e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556ecbdaf1f0 con 0x556ecbe4ae80 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ecbd4c4b0 con 0x556ecbe53a00 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b59b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556ecbdab1b0 con 0x556ecbe4a4d0 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75a9ffb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 0x7f759c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57340/0 (socket says 172.21.15.87:57340) 2024-04-22T13:38:23.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 -- 172.21.15.87:0/1199803202 learned_addr learned my addr 172.21.15.87:0/1199803202 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b1288640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75abfff640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 0x556ecbe55e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b1288640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 0x7f759c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75a9ffb640 1 -- 172.21.15.87:0/984482432 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f75980d0470 con 0x7f75ac106160 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 -- 172.21.15.87:0/1199803202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 msgr2=0x556ecbe4a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 -- 172.21.15.87:0/1199803202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 msgr2=0x556ecbe55e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 0x556ecbe55e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 -- 172.21.15.87:0/1199803202 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ecbd3e1b0 con 0x556ecbe4ae80 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75abfff640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 0x556ecbe55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22176031d0fbc913 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f75b1288640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 0x7f759c0a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009fd0 tx=0x7f75a4004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f7596ffd640 1 -- 172.21.15.87:0/1199803202 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0045020 con 0x556ecbe4ae80 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f7596ffd640 1 -- 172.21.15.87:0/1199803202 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a002c7e0 con 0x556ecbe4ae80 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.742+0000 7f7596ffd640 1 -- 172.21.15.87:0/1199803202 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00046c0 con 0x556ecbe4ae80 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- 172.21.15.87:0/1199803202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 msgr2=0x556ecbe53430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a002c6c0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- 172.21.15.87:0/1199803202 shutdown_connections 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe53a00 0x556ecbe55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4ae80 0x556ecbe53430 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1199803202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- 172.21.15.87:0/1199803202 >> 172.21.15.87:0/1199803202 conn(0x556ecbe44ba0 msgr2=0x556ecbe44f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- 172.21.15.87:0/1199803202 shutdown_connections 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- 172.21.15.87:0/1199803202 wait complete. 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.743+0000 7f75b59b8740 1 -- start start 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 Processor -- start 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4ae80 0x556ecbe50660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe50bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556ecbe56910 con 0x556ecbe53a00 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ecbd68500 con 0x556ecbe4a4d0 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556ecbe56a90 con 0x556ecbe4ae80 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b59b8740 10 client.0 _ll_register_callbacks cb 0x556ecbd36b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe50bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:39260/0 (socket says 172.21.15.87:39260) 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b0a87640 1 -- 172.21.15.87:0/1039834363 learned_addr learned my addr 172.21.15.87:0/1039834363 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75abfff640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4ae80 0x556ecbe50660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 -- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4ae80 msgr2=0x556ecbe50660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4ae80 0x556ecbe50660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 -- 172.21.15.87:0/1039834363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 msgr2=0x556ecbe532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b1288640 1 -- 172.21.15.87:0/1039834363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ecbac13f0 con 0x556ecbe53a00 2024-04-22T13:38:23.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.744+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b1288640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe50bc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f75a40092f0 tx=0x7f75a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15408033a63fc0d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004d20 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f75a400fb60 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a002c7e0 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f75a4011e80 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 10 client.17388 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556ecbb8b7d0 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a002c7e0 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f75a401d040 con 0x556ecbe53a00 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 10 client.17388 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 10 client.17388 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 20 client.17388 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbe63b70 0x556ecbe65fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 4 client.17388 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:23.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.745+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ecbd4e930 con 0x556ecbe63b70 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.746+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbe63b70 0x556ecbe65fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.746+0000 7f75b0a87640 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbe63b70 0x556ecbe65fc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=584023b554d19416 server_cookie=cd6baab4797d18e0 in_seq=0 out_seq=0 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.746+0000 7f758bfff640 10 client.17388 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f758bfff640 1 -- 172.21.15.87:0/1039834363 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ecbd4e930 con 0x556ecbe63b70 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f758bfff640 10 client.17388 handle_command_reply: tid=0 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 client.17388 shutdown 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 msgr2=0x556ecbe50bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe50bc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f75a40092f0 tx=0x7f75a4004a40 comp rx=0 tx=0).stop 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 shutdown_connections 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbe63b70 0x556ecbe65fc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe50bc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f758bfff640 0 client.17388 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4ae80 0x556ecbe50660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 --2- 172.21.15.87:0/1039834363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4a4d0 0x556ecbe532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 >> 172.21.15.87:0/1039834363 conn(0x556ecbe44ba0 msgr2=0x556ecbe468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 shutdown_connections 2024-04-22T13:38:23.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 1 -- 172.21.15.87:0/1039834363 wait complete. 2024-04-22T13:38:23.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.747+0000 7f75b59b8740 20 client.17388 trim_cache size 0 max 16384 2024-04-22T13:38:23.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 Processor -- start 2024-04-22T13:38:23.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 -- start start 2024-04-22T13:38:23.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe43a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe53a00 0x556ecbe52d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1b03c0 con 0x556ecbe4a4d0 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1adc90 con 0x556ecbe4ae80 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b59b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1abb00 con 0x556ecbe53a00 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b0a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57354/0 (socket says 172.21.15.87:57354) 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.760+0000 7f75b0a87640 1 -- 172.21.15.87:0/2040633867 learned_addr learned my addr 172.21.15.87:0/2040633867 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75abfff640 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe43a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 -- 172.21.15.87:0/2040633867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe53a00 msgr2=0x556ecbe52d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe53a00 0x556ecbe52d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 -- 172.21.15.87:0/2040633867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 msgr2=0x556ecbe43a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe43a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 -- 172.21.15.87:0/2040633867 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ecbe627c0 con 0x556ecbe4a4d0 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75abfff640 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b0a87640 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc645284eb3d4b97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75967fc640 1 -- 172.21.15.87:0/2040633867 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0045040 con 0x556ecbe4a4d0 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75967fc640 1 -- 172.21.15.87:0/2040633867 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a002c7e0 con 0x556ecbe4a4d0 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75967fc640 1 -- 172.21.15.87:0/2040633867 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00527e0 con 0x556ecbe4a4d0 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b59b8740 1 -- 172.21.15.87:0/2040633867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 msgr2=0x556ecbe5f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.761+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a0000bc0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- 172.21.15.87:0/2040633867 shutdown_connections 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe53a00 0x556ecbe52d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2040633867 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe4a4d0 0x556ecbe5f7d0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- 172.21.15.87:0/2040633867 >> 172.21.15.87:0/2040633867 conn(0x556ecbe44ba0 msgr2=0x556ecbe43370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- 172.21.15.87:0/2040633867 shutdown_connections 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- 172.21.15.87:0/2040633867 wait complete. 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- start start 2024-04-22T13:38:23.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 Processor -- start 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe4d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1ac150 con 0x556ecbe53a00 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1ad5c0 con 0x556ecbe4ae80 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1ac820 con 0x556ecbe4a4d0 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b59b8740 10 client.0 _ll_register_callbacks cb 0x7f75a4009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b1288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b1288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57360/0 (socket says 172.21.15.87:57360) 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.762+0000 7f75b1288640 1 -- 172.21.15.87:0/2673559253 learned_addr learned my addr 172.21.15.87:0/2673559253 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75abfff640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe4d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 -- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 msgr2=0x556ecbe4d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 -- 172.21.15.87:0/2673559253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 msgr2=0x556ecbe4d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe4d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 -- 172.21.15.87:0/2673559253 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ecbec9130 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75abfff640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe4d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b1288640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f75a40117c0 tx=0x7f75a400fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6fb002e95b2c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40147e0 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f75a4004d30 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a002c7e0 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f75a4014030 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 10 client.17394 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556ecbeca3f0 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a002c7e0 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f75a401d070 con 0x556ecbe53a00 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 10 client.17394 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 10 client.17394 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:38:23.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 20 client.17394 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbecab10 0x556ecbeccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 4 client.17394 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.763+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ecbdbcc10 con 0x556ecbecab10 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b0a87640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbecab10 0x556ecbeccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f758bfff640 10 client.17394 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b0a87640 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbecab10 0x556ecbeccf30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8395384de4f4416a server_cookie=ecca5e9580e99cbd in_seq=0 out_seq=0 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f758bfff640 1 -- 172.21.15.87:0/2673559253 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ecbdbcc10 con 0x556ecbecab10 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f758bfff640 10 client.17394 handle_command_reply: tid=0 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 client.17394 shutdown 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 msgr2=0x556ecbe4de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f75a40117c0 tx=0x7f75a400fe70 comp rx=0 tx=0).stop 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 shutdown_connections 2024-04-22T13:38:23.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x556ecbecab10 0x556ecbeccf30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556ecbe53a00 0x556ecbe4de00 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f758bfff640 0 client.17394 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ecbe4ae80 0x556ecbe4d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 --2- 172.21.15.87:0/2673559253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556ecbe4a4d0 0x556ecbe4d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.764+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 >> 172.21.15.87:0/2673559253 conn(0x556ecbe44ba0 msgr2=0x556ecbe43c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 shutdown_connections 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b59b8740 1 -- 172.21.15.87:0/2673559253 wait complete. 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b59b8740 20 client.17394 trim_cache size 0 max 16384 2024-04-22T13:38:23.767 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 msgr2=0x7f759c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 0x7f759c0a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009fd0 tx=0x7f75a4004ab0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 msgr2=0x7f75ac199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7598005fd0 tx=0x7f759800dde0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 shutdown_connections 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f759c0a3d20 0x7f759c0a61e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75ac19a090 0x7f75ac1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75ac106160 0x7f75ac199b00 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 --2- 172.21.15.87:0/984482432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ac101090 0x7f75ac1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 >> 172.21.15.87:0/984482432 conn(0x7f75ac077f50 msgr2=0x7f75ac0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 shutdown_connections 2024-04-22T13:38:23.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.765+0000 7f75b2ceb640 1 -- 172.21.15.87:0/984482432 wait complete. 2024-04-22T13:38:23.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 -- 172.21.15.87:0/910032771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 msgr2=0x7ff09806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/910032771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff09806ac60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff088009900 tx=0x7ff08802bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 -- 172.21.15.87:0/910032771 shutdown_connections 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/910032771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff098107270 0x7ff09810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/910032771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff09806b230 0x7ff098106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/910032771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff09806ac60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 -- 172.21.15.87:0/910032771 >> 172.21.15.87:0/910032771 conn(0x7ff098077eb0 msgr2=0x7ff0980782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 -- 172.21.15.87:0/910032771 shutdown_connections 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 -- 172.21.15.87:0/910032771 wait complete. 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.876+0000 7ff09e5c7640 1 Processor -- start 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 -- start start 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff098107270 0x7ff098198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff09819d020 0x7ff09819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981121e0 con 0x7ff09806a880 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff098112060 con 0x7ff098107270 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09e5c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff098112360 con 0x7ff09819d020 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57386/0 (socket says 172.21.15.87:57386) 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 -- 172.21.15.87:0/3400412957 learned_addr learned my addr 172.21.15.87:0/3400412957 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff0977fe640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff098107270 0x7ff098198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff09cb64640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff09819d020 0x7ff09819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 -- 172.21.15.87:0/3400412957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff09819d020 msgr2=0x7ff09819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff09819d020 0x7ff09819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 -- 172.21.15.87:0/3400412957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff098107270 msgr2=0x7ff098198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff098107270 0x7ff098198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.877+0000 7ff097fff640 1 -- 172.21.15.87:0/3400412957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09819db10 con 0x7ff09806a880 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff097fff640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff088005940 tx=0x7ff0880045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff0957fa640 1 -- 172.21.15.87:0/3400412957 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088043020 con 0x7ff09806a880 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09819dda0 con 0x7ff09806a880 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff0957fa640 1 -- 172.21.15.87:0/3400412957 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08802caf0 con 0x7ff09806a880 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff0957fa640 1 -- 172.21.15.87:0/3400412957 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088003af0 con 0x7ff09806a880 2024-04-22T13:38:23.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.878+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0981a55c0 con 0x7ff09806a880 2024-04-22T13:38:23.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0957fa640 1 -- 172.21.15.87:0/3400412957 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff088046020 con 0x7ff09806a880 2024-04-22T13:38:23.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 Processor -- start 2024-04-22T13:38:23.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 -- start start 2024-04-22T13:38:23.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x5589719078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971907e80 0x558971910430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55897186c1f0 con 0x558971910a00 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5589718094b0 con 0x5589719074d0 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5589718681b0 con 0x558971907e80 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0957fa640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 0x7ff0800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57392/0 (socket says 172.21.15.87:57392) 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.880+0000 7ff0977fe640 1 -- 172.21.15.87:0/1662073249 learned_addr learned my addr 172.21.15.87:0/1662073249 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff09cb64640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 0x7ff0800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff09cb64640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x5589719078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0957fa640 1 -- 172.21.15.87:0/3400412957 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ff0880f2890 con 0x7ff09806a880 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff097fff640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971907e80 0x558971910430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 -- 172.21.15.87:0/1662073249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971907e80 msgr2=0x558971910430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971907e80 0x558971910430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 -- 172.21.15.87:0/1662073249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 msgr2=0x5589719078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x5589719078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 -- 172.21.15.87:0/1662073249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589717fb1b0 con 0x558971910a00 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff09cb64640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x5589719078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff0977fe640 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c60 tx=0x7ff08c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38628948b8bcc73d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff09cb64640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 0x7ff0800a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff090005cc0 tx=0x7ff09000a340 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff07e7fc640 1 -- 172.21.15.87:0/1662073249 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c004230 con 0x558971910a00 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff07e7fc640 1 -- 172.21.15.87:0/1662073249 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c0043d0 con 0x558971910a00 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.881+0000 7ff07e7fc640 1 -- 172.21.15.87:0/1662073249 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c004d40 con 0x558971910a00 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- 172.21.15.87:0/1662073249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 msgr2=0x558971912e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c60 tx=0x7ff08c011810 comp rx=0 tx=0).stop 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- 172.21.15.87:0/1662073249 shutdown_connections 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971910a00 0x558971912e50 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971907e80 0x558971910430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/1662073249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x5589719078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- 172.21.15.87:0/1662073249 >> 172.21.15.87:0/1662073249 conn(0x558971901ba0 msgr2=0x558971901f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- 172.21.15.87:0/1662073249 shutdown_connections 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- 172.21.15.87:0/1662073249 wait complete. 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 -- start start 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.882+0000 7ff0a11ba740 1 Processor -- start 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x55897190db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558971913970 con 0x5589719074d0 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558971825500 con 0x558971907e80 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558971913af0 con 0x558971910a00 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 10 client.0 _ll_register_callbacks cb 0x5589717f3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57404/0 (socket says 172.21.15.87:57404) 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 -- 172.21.15.87:0/3196123225 learned_addr learned my addr 172.21.15.87:0/3196123225 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0977fe640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff09cb64640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x55897190db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 -- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 msgr2=0x55897190db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x55897190db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 -- 172.21.15.87:0/3196123225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 msgr2=0x55897190d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 -- 172.21.15.87:0/3196123225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897157e3f0 con 0x5589719074d0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff09cb64640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x55897190db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff097fff640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff0880bb350 tx=0x7ff0880be750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcfc4556f86e172 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0777fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0880bf080 con 0x5589719074d0 2024-04-22T13:38:23.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ff0880053b0 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.883+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff08c0043d0 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff0880be980 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 10 client.17406 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5589716487d0 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff08c0043d0 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff0880c7160 con 0x5589719074d0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 10 client.17406 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 10 client.17406 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 20 client.17406 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558971920ba0 0x558971923030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 4 client.17406 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55897180b930 con 0x558971920ba0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.884+0000 7ff0977fe640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558971920ba0 0x558971923030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0977fe640 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558971920ba0 0x558971923030 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91f8d8d208e483fa server_cookie=a7bfa83b907fccf5 in_seq=0 out_seq=0 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0777fe640 10 client.17406 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:23.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0777fe640 1 -- 172.21.15.87:0/3196123225 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55897180b930 con 0x558971920ba0 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0777fe640 10 client.17406 handle_command_reply: tid=0 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0a11ba740 1 client.17406 shutdown 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 msgr2=0x5589719102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff0880bb350 tx=0x7ff0880be750 comp rx=0 tx=0).stop 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 shutdown_connections 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.885+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x558971920ba0 0x558971923030 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x55897190db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0777fe640 0 client.17406 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/3196123225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x5589719102b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 >> 172.21.15.87:0/3196123225 conn(0x558971901ba0 msgr2=0x5589719038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 shutdown_connections 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 1 -- 172.21.15.87:0/3196123225 wait complete. 2024-04-22T13:38:23.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.886+0000 7ff0a11ba740 20 client.17406 trim_cache size 0 max 16384 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 Processor -- start 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 -- start start 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x5589719102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981b03d0 con 0x5589719074d0 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981adca0 con 0x558971907e80 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981abb10 con 0x558971910a00 2024-04-22T13:38:23.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57412/0 (socket says 172.21.15.87:57412) 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 -- 172.21.15.87:0/2006474961 learned_addr learned my addr 172.21.15.87:0/2006474961 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff0977fe640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff09cb64640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x5589719102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 -- 172.21.15.87:0/2006474961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 msgr2=0x5589719102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x5589719102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 -- 172.21.15.87:0/2006474961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 msgr2=0x55897190fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 -- 172.21.15.87:0/2006474961 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589718fbfe0 con 0x5589719074d0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff09cb64640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x5589719102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff097fff640 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff0880bade0 tx=0x7ff0880be880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800daad0025e1f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.899+0000 7ff07dffb640 1 -- 172.21.15.87:0/2006474961 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0880be4c0 con 0x5589719074d0 2024-04-22T13:38:23.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff07dffb640 1 -- 172.21.15.87:0/2006474961 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0880bf080 con 0x5589719074d0 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff07dffb640 1 -- 172.21.15.87:0/2006474961 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0880892b0 con 0x5589719074d0 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2006474961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 msgr2=0x55897191c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff0880bade0 tx=0x7ff0880be880 comp rx=0 tx=0).stop 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2006474961 shutdown_connections 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x5589719102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558971907e80 0x55897190fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2006474961 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5589719074d0 0x55897191c830 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2006474961 >> 172.21.15.87:0/2006474961 conn(0x558971901ba0 msgr2=0x558971908c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2006474961 shutdown_connections 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2006474961 wait complete. 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.900+0000 7ff0a11ba740 1 -- start start 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 Processor -- start 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x558971918dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x558971914860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981ac160 con 0x558971907e80 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981ad5d0 con 0x5589719074d0 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981ac830 con 0x558971910a00 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0a11ba740 10 client.0 _ll_register_callbacks cb 0x7ff088005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57422/0 (socket says 172.21.15.87:57422) 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 -- 172.21.15.87:0/2348132654 learned_addr learned my addr 172.21.15.87:0/2348132654 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff097fff640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x558971918dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff09cb64640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x558971914860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 -- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 msgr2=0x558971914860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x558971914860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 -- 172.21.15.87:0/2348132654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 msgr2=0x558971918dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x558971918dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 -- 172.21.15.87:0/2348132654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558971914da0 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff097fff640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x558971918dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0977fe640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c30 tx=0x7ff08c0048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9314f4043d2bbe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.901+0000 7ff0777fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c01e040 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ff08c004460 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0880bf080 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff08c009e10 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 10 client.17412 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558971986de0 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0880bf080 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff08c024040 con 0x558971907e80 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 10 client.17412 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:23.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 10 client.17412 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 20 client.17412 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5589719873c0 0x5589719897e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 4 client.17412 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558971879c10 con 0x5589719873c0 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.902+0000 7ff097fff640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5589719873c0 0x5589719897e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff097fff640 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5589719873c0 0x5589719897e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fbb0be8ab29c097d server_cookie=aa53c8c21bf710a in_seq=0 out_seq=0 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0777fe640 10 client.17412 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0777fe640 1 -- 172.21.15.87:0/2348132654 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558971879c10 con 0x5589719873c0 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0777fe640 10 client.17412 handle_command_reply: tid=0 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 client.17412 shutdown 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 msgr2=0x558971914320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c30 tx=0x7ff08c0048e0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 shutdown_connections 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5589719873c0 0x5589719897e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558971910a00 0x558971914860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0777fe640 0 client.17412 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558971907e80 0x558971914320 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 --2- 172.21.15.87:0/2348132654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5589719074d0 0x558971918dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 >> 172.21.15.87:0/2348132654 conn(0x558971901ba0 msgr2=0x558971902ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 shutdown_connections 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 1 -- 172.21.15.87:0/2348132654 wait complete. 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.903+0000 7ff0a11ba740 20 client.17412 trim_cache size 0 max 16384 2024-04-22T13:38:23.906 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 msgr2=0x7ff0800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 0x7ff0800a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff090005cc0 tx=0x7ff09000a340 comp rx=0 tx=0).stop 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 msgr2=0x7ff098198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff088005940 tx=0x7ff0880045e0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 shutdown_connections 2024-04-22T13:38:23.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0800a3d20 0x7ff0800a61e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff09819d020 0x7ff09819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff098107270 0x7ff098198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 --2- 172.21.15.87:0/3400412957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff09806a880 0x7ff098198760 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 >> 172.21.15.87:0/3400412957 conn(0x7ff098077eb0 msgr2=0x7ff098108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 shutdown_connections 2024-04-22T13:38:23.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:23.904+0000 7ff09e5c7640 1 -- 172.21.15.87:0/3400412957 wait complete. 2024-04-22T13:38:23.909 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:23 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3900068639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:23.910 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:38:23.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:23 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3900068639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 -- 172.21.15.87:0/3661422512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc0107270 msgr2=0x7f8cc010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 --2- 172.21.15.87:0/3661422512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc0107270 0x7f8cc010ee40 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009ca0 tx=0x7f8cb802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 -- 172.21.15.87:0/3661422512 shutdown_connections 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 --2- 172.21.15.87:0/3661422512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc0107270 0x7f8cc010ee40 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 --2- 172.21.15.87:0/3661422512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cc0102b50 0x7f8cc0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 --2- 172.21.15.87:0/3661422512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc01021a0 0x7f8cc0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 -- 172.21.15.87:0/3661422512 >> 172.21.15.87:0/3661422512 conn(0x7f8cc00fd9c0 msgr2=0x7f8cc00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 -- 172.21.15.87:0/3661422512 shutdown_connections 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 -- 172.21.15.87:0/3661422512 wait complete. 2024-04-22T13:38:24.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.014+0000 7f8cc5406640 1 Processor -- start 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 -- start start 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 0x7f8cc0198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cc019cfd0 0x7f8cc019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01121c0 con 0x7f8cc01021a0 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0112040 con 0x7f8cc0107270 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0112340 con 0x7f8cc019cfd0 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57432/0 (socket says 172.21.15.87:57432) 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 -- 172.21.15.87:0/1203126359 learned_addr learned my addr 172.21.15.87:0/1203126359 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 0x7f8cc0198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 -- 172.21.15.87:0/1203126359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cc019cfd0 msgr2=0x7f8cc019d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cc019cfd0 0x7f8cc019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 -- 172.21.15.87:0/1203126359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 msgr2=0x7f8cc0198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 0x7f8cc0198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 -- 172.21.15.87:0/1203126359 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc019dac0 con 0x7f8cc01021a0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 0x7f8cc0198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000ea60 tx=0x7f8cb000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8ca7fff640 1 -- 172.21.15.87:0/1203126359 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000cde0 con 0x7f8cc01021a0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8ca7fff640 1 -- 172.21.15.87:0/1203126359 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb00035f0 con 0x7f8cc01021a0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8ca7fff640 1 -- 172.21.15.87:0/1203126359 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0010690 con 0x7f8cc01021a0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.015+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc019ddb0 con 0x7f8cc01021a0 2024-04-22T13:38:24.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.016+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc01a55a0 con 0x7f8cc01021a0 2024-04-22T13:38:24.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8ca7fff640 1 -- 172.21.15.87:0/1203126359 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8cb00037a0 con 0x7f8cc01021a0 2024-04-22T13:38:24.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 Processor -- start 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 -- start start 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e86710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e8fe40 0x556b07e902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556b07deb1f0 con 0x556b07e86ce0 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b07d884b0 con 0x556b07e8fe40 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b07de71b0 con 0x556b07e86330 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8ca7fff640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 0x7f8ca80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 0x7f8ca80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57440/0 (socket says 172.21.15.87:57440) 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cbeffd640 1 -- 172.21.15.87:0/3983035161 learned_addr learned my addr 172.21.15.87:0/3983035161 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.018+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e86710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e8fe40 0x556b07e902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8ca7fff640 1 -- 172.21.15.87:0/1203126359 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f8cb00d0bc0 con 0x7f8cc01021a0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 -- 172.21.15.87:0/3983035161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 msgr2=0x556b07e86710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e86710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 -- 172.21.15.87:0/3983035161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e8fe40 msgr2=0x556b07e902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e8fe40 0x556b07e902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 -- 172.21.15.87:0/3983035161 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b07d7a1b0 con 0x556b07e86ce0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0009920 tx=0x7f8cb006f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce68b8c6c0d66117 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 0x7f8ca80a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8cb802c630 tx=0x7f8cb8009800 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8ca57fa640 1 -- 172.21.15.87:0/3983035161 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb009d2a0 con 0x556b07e86ce0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8ca57fa640 1 -- 172.21.15.87:0/3983035161 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb009cba0 con 0x556b07e86ce0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.019+0000 7f8ca57fa640 1 -- 172.21.15.87:0/3983035161 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb009af70 con 0x556b07e86ce0 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3983035161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 msgr2=0x556b07e8f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0009920 tx=0x7f8cb006f690 comp rx=0 tx=0).stop 2024-04-22T13:38:24.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3983035161 shutdown_connections 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e8fe40 0x556b07e902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e8f820 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3983035161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e86710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3983035161 >> 172.21.15.87:0/3983035161 conn(0x556b07e80ba0 msgr2=0x556b07e80f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3983035161 shutdown_connections 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3983035161 wait complete. 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- start start 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 Processor -- start 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 0x556b07e96230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e93b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556b07e96dc0 con 0x556b07e86ce0 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b07da4500 con 0x556b07e86330 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.020+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b07e96f40 con 0x556b07e8fe40 2024-04-22T13:38:24.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cc7ff5740 10 client.0 _ll_register_callbacks cb 0x556b07d72b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57442/0 (socket says 172.21.15.87:57442) 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 -- 172.21.15.87:0/3247283029 learned_addr learned my addr 172.21.15.87:0/3247283029 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e93b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 0x556b07e96230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 -- 172.21.15.87:0/3247283029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 msgr2=0x556b07e93b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e93b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 -- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 msgr2=0x556b07e96230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 0x556b07e96230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 -- 172.21.15.87:0/3247283029 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b07afd3f0 con 0x556b07e86ce0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 0x556b07e96230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400d9d0 tx=0x7f8cb400de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79aeb933154866cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8ca2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb401b440 con 0x556b07e86ce0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f8cb4011440 con 0x556b07e86ce0 2024-04-22T13:38:24.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cb009cba0 con 0x556b07e86ce0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8cb4011640 con 0x556b07e86ce0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cc7ff5740 10 client.17424 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556b07bc77d0 con 0x556b07e86ce0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.021+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb009cba0 con 0x556b07e86ce0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8cb4019070 con 0x556b07e86ce0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 10 client.17424 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 20 client.17424 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07ea4110 0x556b07ea6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 4 client.17424 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b07d8a930 con 0x556b07ea4110 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07ea4110 0x556b07ea6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8ca2ff5640 10 client.17424 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.022+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07ea4110 0x556b07ea6570 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585c99896d571f7b server_cookie=51a43b6223ddaaab in_seq=0 out_seq=0 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/3247283029 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b07d8a930 con 0x556b07ea4110 2024-04-22T13:38:24.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8ca2ff5640 10 client.17424 handle_command_reply: tid=0 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 client.17424 shutdown 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 msgr2=0x556b07e935c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400d9d0 tx=0x7f8cb400de90 comp rx=0 tx=0).stop 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 shutdown_connections 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07ea4110 0x556b07ea6570 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e93b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86ce0 0x556b07e935c0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8ca2ff5640 0 client.17424 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/3247283029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86330 0x556b07e96230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 >> 172.21.15.87:0/3247283029 conn(0x556b07e80ba0 msgr2=0x556b07e828f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 shutdown_connections 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/3247283029 wait complete. 2024-04-22T13:38:24.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.023+0000 7f8cc7ff5740 20 client.17424 trim_cache size 0 max 16384 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 Processor -- start 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 -- start start 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e96230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01ae270 con 0x556b07e86330 2024-04-22T13:38:24.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01abb40 con 0x556b07e86ce0 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.036+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01a99d0 con 0x556b07e8fe40 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57456/0 (socket says 172.21.15.87:57456) 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 -- 172.21.15.87:0/207814521 learned_addr learned my addr 172.21.15.87:0/207814521 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 -- 172.21.15.87:0/207814521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 msgr2=0x556b07e96230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e96230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 -- 172.21.15.87:0/207814521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 msgr2=0x556b07e95cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 -- 172.21.15.87:0/207814521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b07e91b10 con 0x556b07e86330 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00ce400 tx=0x7f8cb00c7cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b6bceb21824488 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8ca4ff9640 1 -- 172.21.15.87:0/207814521 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb009d2a0 con 0x556b07e86330 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8ca4ff9640 1 -- 172.21.15.87:0/207814521 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb009cba0 con 0x556b07e86330 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.037+0000 7f8ca4ff9640 1 -- 172.21.15.87:0/207814521 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0004000 con 0x556b07e86330 2024-04-22T13:38:24.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/207814521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 msgr2=0x556b07e9fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00ce400 tx=0x7f8cb00c7cb0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/207814521 shutdown_connections 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e8fe40 0x556b07e96230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/207814521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e86330 0x556b07e9fcc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/207814521 >> 172.21.15.87:0/207814521 conn(0x556b07e80ba0 msgr2=0x556b07e7fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/207814521 shutdown_connections 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/207814521 wait complete. 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- start start 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 Processor -- start 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e95210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01aa020 con 0x556b07e8fe40 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01ab470 con 0x556b07e86ce0 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01aa6d0 con 0x556b07e86330 2024-04-22T13:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cc7ff5740 10 client.0 _ll_register_callbacks cb 0x7f8cb40043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57462/0 (socket says 172.21.15.87:57462) 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbf7fe640 1 -- 172.21.15.87:0/588716133 learned_addr learned my addr 172.21.15.87:0/588716133 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e95210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.038+0000 7f8cbf7fe640 1 -- 172.21.15.87:0/588716133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 msgr2=0x556b07e95210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e95210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbf7fe640 1 -- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 msgr2=0x556b07e95750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbf7fe640 1 -- 172.21.15.87:0/588716133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b07f096b0 con 0x556b07e8fe40 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e95210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbe7fc640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cbf7fe640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009ca0 tx=0x7f8cb80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456b7ee193015f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:24.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8053070 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f8cb8005230 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cb009cba0 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8cb805e050 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 10 client.17430 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556b07f0a990 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb009cba0 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8cb80347b0 con 0x556b07e8fe40 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 10 client.17430 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 10 client.17430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 20 client.17430 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07f0b170 0x556b07f0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 4 client.17430 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.039+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b07df8c10 con 0x556b07f0b170 2024-04-22T13:38:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07f0b170 0x556b07f0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8ca2ff5640 10 client.17430 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cbeffd640 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07f0b170 0x556b07f0d590 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8dd6392b9f5cc4f server_cookie=3e49db59cc993935 in_seq=0 out_seq=0 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8ca2ff5640 1 -- 172.21.15.87:0/588716133 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b07df8c10 con 0x556b07f0b170 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8ca2ff5640 10 client.17430 handle_command_reply: tid=0 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 client.17430 shutdown 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 msgr2=0x556b07e95c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009ca0 tx=0x7f8cb80095c0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 shutdown_connections 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b07f0b170 0x556b07f0d590 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b07e8fe40 0x556b07e95c90 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b07e86ce0 0x556b07e95750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 --2- 172.21.15.87:0/588716133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b07e86330 0x556b07e95210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 >> 172.21.15.87:0/588716133 conn(0x556b07e80ba0 msgr2=0x556b07e8da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8ca2ff5640 0 client.17430 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:24.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 shutdown_connections 2024-04-22T13:38:24.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 1 -- 172.21.15.87:0/588716133 wait complete. 2024-04-22T13:38:24.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.040+0000 7f8cc7ff5740 20 client.17430 trim_cache size 0 max 16384 2024-04-22T13:38:24.043 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 msgr2=0x7f8ca80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 0x7f8ca80a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8cb802c630 tx=0x7f8cb8009800 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 msgr2=0x7f8cc0198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000ea60 tx=0x7f8cb000ef20 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 shutdown_connections 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8ca80a3d20 0x7f8ca80a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cc019cfd0 0x7f8cc019d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8cc0107270 0x7f8cc0198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 --2- 172.21.15.87:0/1203126359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8cc01021a0 0x7f8cc0198760 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 >> 172.21.15.87:0/1203126359 conn(0x7f8cc00fd9c0 msgr2=0x7f8cc00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 shutdown_connections 2024-04-22T13:38:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:24.041+0000 7f8cc5406640 1 -- 172.21.15.87:0/1203126359 wait complete. 2024-04-22T13:38:24.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:23 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3900068639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:24.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:24 smithi132 ceph-mon[31531]: pgmap v202: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 143 MiB/s wr, 1.46k op/s 2024-04-22T13:38:24.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:24 smithi087 ceph-mon[27999]: pgmap v202: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 143 MiB/s wr, 1.46k op/s 2024-04-22T13:38:25.048 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:38:25.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:38:25.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- 172.21.15.87:0/3147775225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358107260 msgr2=0x7fe35810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 --2- 172.21.15.87:0/3147775225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358107260 0x7fe35810ee30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe350009ca0 tx=0x7fe35002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- 172.21.15.87:0/3147775225 shutdown_connections 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 --2- 172.21.15.87:0/3147775225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358107260 0x7fe35810ee30 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 --2- 172.21.15.87:0/3147775225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102b40 0x7fe358106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 --2- 172.21.15.87:0/3147775225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358102190 0x7fe358102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- 172.21.15.87:0/3147775225 >> 172.21.15.87:0/3147775225 conn(0x7fe3580793a0 msgr2=0x7fe3580797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- 172.21.15.87:0/3147775225 shutdown_connections 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- 172.21.15.87:0/3147775225 wait complete. 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 Processor -- start 2024-04-22T13:38:25.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.167+0000 7fe35e72e640 1 -- start start 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102190 0x7fe35819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358107260 0x7fe35819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581121d0 con 0x7fe358198590 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe358112050 con 0x7fe358102190 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe358112350 con 0x7fe358107260 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57482/0 (socket says 172.21.15.87:57482) 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 -- 172.21.15.87:0/1771797110 learned_addr learned my addr 172.21.15.87:0/1771797110 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe3577fe640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358107260 0x7fe35819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe357fff640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102190 0x7fe35819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 -- 172.21.15.87:0/1771797110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358107260 msgr2=0x7fe35819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358107260 0x7fe35819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 -- 172.21.15.87:0/1771797110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102190 msgr2=0x7fe35819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102190 0x7fe35819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 -- 172.21.15.87:0/1771797110 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3581990b0 con 0x7fe358198590 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35cccb640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe350005e30 tx=0x7fe350034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe3557fa640 1 -- 172.21.15.87:0/1771797110 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350039040 con 0x7fe358198590 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe3557fa640 1 -- 172.21.15.87:0/1771797110 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe350044070 con 0x7fe358198590 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe358199340 con 0x7fe358198590 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe3557fa640 1 -- 172.21.15.87:0/1771797110 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350003e60 con 0x7fe358198590 2024-04-22T13:38:25.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.168+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3581a5530 con 0x7fe358198590 2024-04-22T13:38:25.173 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:24 smithi120 ceph-mon[31101]: pgmap v202: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 143 MiB/s wr, 1.46k op/s 2024-04-22T13:38:25.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3557fa640 1 -- 172.21.15.87:0/1771797110 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe3500324c0 con 0x7fe358198590 2024-04-22T13:38:25.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 Processor -- start 2024-04-22T13:38:25.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 -- start start 2024-04-22T13:38:25.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cdda94ff00 con 0x55cdda9ec170 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cdda8ed590 con 0x55cdda9eb7c0 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3613f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cdda94bec0 con 0x55cdda9f5270 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57492/0 (socket says 172.21.15.87:57492) 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe357fff640 1 -- 172.21.15.87:0/150193418 learned_addr learned my addr 172.21.15.87:0/150193418 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.171+0000 7fe3557fa640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 0x7fe3400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe3577fe640 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe3577fe640 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe35cccb640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 0x7fe3400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 -- 172.21.15.87:0/150193418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 msgr2=0x55cdda9f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 -- 172.21.15.87:0/150193418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 msgr2=0x55cdda9ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 -- 172.21.15.87:0/150193418 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdda72a7e0 con 0x55cdda9ec170 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe3557fa640 1 -- 172.21.15.87:0/1771797110 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fe3500f2950 con 0x7fe358198590 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe357fff640 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe348009bc0 tx=0x7fe34800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33a3205966f11b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe35cccb640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 0x7fe3400a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe3500f02a0 tx=0x7fe35008d3f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe33e7fc640 1 -- 172.21.15.87:0/150193418 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348016070 con 0x55cdda9ec170 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe33e7fc640 1 -- 172.21.15.87:0/150193418 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe348011920 con 0x55cdda9ec170 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.172+0000 7fe33e7fc640 1 -- 172.21.15.87:0/150193418 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348010790 con 0x55cdda9ec170 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- 172.21.15.87:0/150193418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 msgr2=0x55cdda9f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe348009bc0 tx=0x7fe34800e6c0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- 172.21.15.87:0/150193418 shutdown_connections 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f4ca0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 --2- 172.21.15.87:0/150193418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- 172.21.15.87:0/150193418 >> 172.21.15.87:0/150193418 conn(0x55cdda9e5fb0 msgr2=0x55cdda9e6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- 172.21.15.87:0/150193418 shutdown_connections 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- 172.21.15.87:0/150193418 wait complete. 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.173+0000 7fe3613f5740 1 -- start start 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 Processor -- start 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cdda9fc1b0 con 0x55cdda9ec170 2024-04-22T13:38:25.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cdda909580 con 0x55cdda9eb7c0 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cdda9fc330 con 0x55cdda9f5270 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3613f5740 10 client.0 _ll_register_callbacks cb 0x55cdda90f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57496/0 (socket says 172.21.15.87:57496) 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 -- 172.21.15.87:0/3415456851 learned_addr learned my addr 172.21.15.87:0/3415456851 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3577fe640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3577fe640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 -- 172.21.15.87:0/3415456851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 msgr2=0x55cdda9f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 -- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 msgr2=0x55cdda9f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9f8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 -- 172.21.15.87:0/3415456851 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdda78af70 con 0x55cdda9ec170 2024-04-22T13:38:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3577fe640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe357fff640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe348010490 tx=0x7fe34800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99837faeaa85ab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3377fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348016050 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.174+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fe348005320 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe348011920 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe3480097c0 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 10 client.17442 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cdda622170 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe348011920 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe34800f040 con 0x55cdda9ec170 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 10 client.17442 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 20 client.17442 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:25.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa0b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 4 client.17442 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdda95d920 con 0x55cddaa095c0 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.175+0000 7fe3577fe640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa0b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.176+0000 7fe3377fe640 10 client.17442 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.176+0000 7fe3577fe640 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa0b9b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cac85b1924c700 server_cookie=36e9df500ce138c0 in_seq=0 out_seq=0 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.176+0000 7fe3377fe640 1 -- 172.21.15.87:0/3415456851 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdda95d920 con 0x55cddaa095c0 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.176+0000 7fe3377fe640 10 client.17442 handle_command_reply: tid=0 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.176+0000 7fe3613f5740 1 client.17442 shutdown 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 msgr2=0x55cdda9f9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe348010490 tx=0x7fe34800eef0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 shutdown_connections 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa0b9b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9f9270 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3377fe640 0 client.17442 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 --2- 172.21.15.87:0/3415456851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 >> 172.21.15.87:0/3415456851 conn(0x55cdda9e5fb0 msgr2=0x55cdda9e7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 shutdown_connections 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 1 -- 172.21.15.87:0/3415456851 wait complete. 2024-04-22T13:38:25.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.177+0000 7fe3613f5740 20 client.17442 trim_cache size 0 max 16384 2024-04-22T13:38:25.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 Processor -- start 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 -- start start 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581ae2a0 con 0x55cdda9ec170 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581abb70 con 0x55cdda9eb7c0 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe3613f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581a99e0 con 0x55cdda9f5270 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.194+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57500/0 (socket says 172.21.15.87:57500) 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 -- 172.21.15.87:0/163794234 learned_addr learned my addr 172.21.15.87:0/163794234 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe3577fe640 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 -- 172.21.15.87:0/163794234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 msgr2=0x55cdda9f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe3577fe640 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 -- 172.21.15.87:0/163794234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 msgr2=0x55cdda9fb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 -- 172.21.15.87:0/163794234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdda9f7d00 con 0x55cdda9ec170 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe357fff640 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe348009980 tx=0x7fe348011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c432ee5cedb28 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe33dffb640 1 -- 172.21.15.87:0/163794234 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348032020 con 0x55cdda9ec170 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe33dffb640 1 -- 172.21.15.87:0/163794234 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe348003aa0 con 0x55cdda9ec170 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.195+0000 7fe33dffb640 1 -- 172.21.15.87:0/163794234 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348016070 con 0x55cdda9ec170 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- 172.21.15.87:0/163794234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 msgr2=0x55cddaa050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe348009980 tx=0x7fe348011870 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- 172.21.15.87:0/163794234 shutdown_connections 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9f5270 0x55cdda9f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cddaa050b0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- 172.21.15.87:0/163794234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- 172.21.15.87:0/163794234 >> 172.21.15.87:0/163794234 conn(0x55cdda9e5fb0 msgr2=0x55cdda9e5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- 172.21.15.87:0/163794234 shutdown_connections 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- 172.21.15.87:0/163794234 wait complete. 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- start start 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 Processor -- start 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9f5270 0x55cdda9ff010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581aa030 con 0x55cdda9ec170 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581ab4a0 con 0x55cdda9f5270 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.196+0000 7fe3613f5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3581aa700 con 0x55cdda9eb7c0 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3613f5740 10 client.0 _ll_register_callbacks cb 0x7fe34c00a910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57516/0 (socket says 172.21.15.87:57516) 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 -- 172.21.15.87:0/809137822 learned_addr learned my addr 172.21.15.87:0/809137822 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3577fe640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9f5270 0x55cdda9ff010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 -- 172.21.15.87:0/809137822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9eb7c0 msgr2=0x55cdda9fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3577fe640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fe590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 -- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9f5270 msgr2=0x55cdda9ff010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9f5270 0x55cdda9ff010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 -- 172.21.15.87:0/809137822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cddaa6f930 con 0x55cdda9ec170 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe357fff640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe348005550 tx=0x7fe348009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b9919c952a945a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3377fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348016070 con 0x55cdda9ec170 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fe3480055b0 con 0x55cdda9ec170 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cdda957790 con 0x55cdda9ec170 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe3480094e0 con 0x55cdda9ec170 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3613f5740 10 client.17448 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:25.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cddaa70c50 con 0x55cdda9ec170 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.197+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cdda957790 con 0x55cdda9ec170 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe348025420 con 0x55cdda9ec170 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 10 client.17448 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 20 client.17448 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa73390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 4 client.17448 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cdda8d3c50 con 0x55cddaa095c0 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3577fe640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa73390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3377fe640 10 client.17448 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.198+0000 7fe3577fe640 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa73390 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c4ff2f29870ff20 server_cookie=7128e103a0f791d4 in_seq=0 out_seq=0 2024-04-22T13:38:25.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3377fe640 1 -- 172.21.15.87:0/809137822 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cdda8d3c50 con 0x55cddaa095c0 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3377fe640 10 client.17448 handle_command_reply: tid=0 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 client.17448 shutdown 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 msgr2=0x55cdda9fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe348005550 tx=0x7fe348009f70 comp rx=0 tx=0).stop 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 shutdown_connections 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cddaa095c0 0x55cddaa73390 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdda9f5270 0x55cdda9ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3377fe640 0 client.17448 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cdda9ec170 0x55cdda9fead0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 --2- 172.21.15.87:0/809137822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cdda9eb7c0 0x55cdda9fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 >> 172.21.15.87:0/809137822 conn(0x55cdda9e5fb0 msgr2=0x55cdda9e4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 shutdown_connections 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 1 -- 172.21.15.87:0/809137822 wait complete. 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe3613f5740 20 client.17448 trim_cache size 0 max 16384 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "56ecde86-70c8-4f93-9e9c-118d4b1972c2", 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:38:25.202 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 msgr2=0x7fe3400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 0x7fe3400a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe3500f02a0 tx=0x7fe35008d3f0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 msgr2=0x7fe358198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.200+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe350005e30 tx=0x7fe350034760 comp rx=0 tx=0).stop 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 shutdown_connections 2024-04-22T13:38:25.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe3400a3d20 0x7fe3400a61e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe358198590 0x7fe358198a40 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe358107260 0x7fe35819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 --2- 172.21.15.87:0/1771797110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102190 0x7fe35819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 >> 172.21.15.87:0/1771797110 conn(0x7fe3580793a0 msgr2=0x7fe3580ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 shutdown_connections 2024-04-22T13:38:25.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.201+0000 7fe35e72e640 1 -- 172.21.15.87:0/1771797110 wait complete. 2024-04-22T13:38:25.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:38:25.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 -- 172.21.15.87:0/346860053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4101a60 msgr2=0x7f6ad4105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 --2- 172.21.15.87:0/346860053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4101a60 0x7f6ad4105aa0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009ca0 tx=0x7f6ac802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 -- 172.21.15.87:0/346860053 shutdown_connections 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 --2- 172.21.15.87:0/346860053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad4106180 0x7f6ad410dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 --2- 172.21.15.87:0/346860053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4101a60 0x7f6ad4105aa0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 --2- 172.21.15.87:0/346860053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad41010b0 0x7f6ad4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.313+0000 7f6ad924a640 1 -- 172.21.15.87:0/346860053 >> 172.21.15.87:0/346860053 conn(0x7f6ad40771d0 msgr2=0x7f6ad40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.314+0000 7f6ad924a640 1 -- 172.21.15.87:0/346860053 shutdown_connections 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.314+0000 7f6ad924a640 1 -- 172.21.15.87:0/346860053 wait complete. 2024-04-22T13:38:25.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.314+0000 7f6ad924a640 1 Processor -- start 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.314+0000 7f6ad924a640 1 -- start start 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 0x7f6ad4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 0x7f6ad419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad4112240 con 0x7f6ad4106180 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41120c0 con 0x7f6ad41010b0 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41123c0 con 0x7f6ad419d020 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57532/0 (socket says 172.21.15.87:57532) 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 -- 172.21.15.87:0/2750458651 learned_addr learned my addr 172.21.15.87:0/2750458651 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad359e640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 0x7f6ad419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 0x7f6ad4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 -- 172.21.15.87:0/2750458651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 msgr2=0x7f6ad419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 0x7f6ad419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 -- 172.21.15.87:0/2750458651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 msgr2=0x7f6ad4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 0x7f6ad4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 -- 172.21.15.87:0/2750458651 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad419db10 con 0x7f6ad4106180 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad359e640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 0x7f6ad419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 0x7f6ad4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad259c640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8004af0 tx=0x7f6ac80346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6abbfff640 1 -- 172.21.15.87:0/2750458651 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8041020 con 0x7f6ad4106180 2024-04-22T13:38:25.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6abbfff640 1 -- 172.21.15.87:0/2750458651 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac8032970 con 0x7f6ad4106180 2024-04-22T13:38:25.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad419dda0 con 0x7f6ad4106180 2024-04-22T13:38:25.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6abbfff640 1 -- 172.21.15.87:0/2750458651 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8039040 con 0x7f6ad4106180 2024-04-22T13:38:25.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.315+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad41a5500 con 0x7f6ad4106180 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6abbfff640 1 -- 172.21.15.87:0/2750458651 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ac8044020 con 0x7f6ad4106180 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 Processor -- start 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 -- start start 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fa7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6abbfff640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 0x7f6abc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fb0a80 0x564c30fb2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564c30f0c1e0 con 0x564c30fa7ea0 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c30ea9390 con 0x564c30fb0a80 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564c30f081a0 con 0x564c30fa7580 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57546/0 (socket says 172.21.15.87:57546) 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/801858513 learned_addr learned my addr 172.21.15.87:0/801858513 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.318+0000 7f6ad259c640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 0x7f6abc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6abbfff640 1 -- 172.21.15.87:0/2750458651 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f6ac80f25d0 con 0x7f6ad4106180 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad359e640 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fa7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad359e640 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fb0a80 0x564c30fb2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/801858513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 msgr2=0x564c30fa7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fa7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/801858513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fb0a80 msgr2=0x564c30fb2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fb0a80 0x564c30fb2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/801858513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c30ce87d0 con 0x564c30fa7ea0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009c50 tx=0x7f6acc00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f70555db2294014 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ad259c640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 0x7f6abc0a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80eff20 tx=0x7f6ac80b3b50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ab97fa640 1 -- 172.21.15.87:0/801858513 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc016070 con 0x564c30fa7ea0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ab97fa640 1 -- 172.21.15.87:0/801858513 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6acc011920 con 0x564c30fa7ea0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.319+0000 7f6ab97fa640 1 -- 172.21.15.87:0/801858513 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc010810 con 0x564c30fa7ea0 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- 172.21.15.87:0/801858513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 msgr2=0x564c30fb0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009c50 tx=0x7f6acc00e6c0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- 172.21.15.87:0/801858513 shutdown_connections 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fb0a80 0x564c30fb2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fb0540 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/801858513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fa7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- 172.21.15.87:0/801858513 >> 172.21.15.87:0/801858513 conn(0x564c30fa1cb0 msgr2=0x564c30fa2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- 172.21.15.87:0/801858513 shutdown_connections 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- 172.21.15.87:0/801858513 wait complete. 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- start start 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 Processor -- start 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30fae560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564c30fb3870 con 0x564c30fa7580 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c30ec52f0 con 0x564c30fa7ea0 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564c30fb39f0 con 0x564c30fb0a80 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6adbe3f740 10 client.0 _ll_register_callbacks cb 0x564c30ed2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57554/0 (socket says 172.21.15.87:57554) 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.320+0000 7f6ad359e640 1 -- 172.21.15.87:0/3165194047 learned_addr learned my addr 172.21.15.87:0/3165194047 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30fae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 -- 172.21.15.87:0/3165194047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 msgr2=0x564c30fae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30fae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 -- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 msgr2=0x564c30fae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 -- 172.21.15.87:0/3165194047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c30c1e3f0 con 0x564c30fa7580 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400cec0 tx=0x7f6ac40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86167c92c4414eee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ab6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400fbd0 con 0x564c30fa7580 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f6ac400fd70 con 0x564c30fa7580 2024-04-22T13:38:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6acc011920 con 0x564c30fa7580 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ac4026050 con 0x564c30fa7580 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6adbe3f740 10 client.17460 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c30bde0f0 con 0x564c30fa7580 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.321+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6acc011920 con 0x564c30fa7580 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6ac4013450 con 0x564c30fa7580 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 10 client.17460 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 20 client.17460 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c30fc0b00 0x564c30fc2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 4 client.17460 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c30f19c00 con 0x564c30fc0b00 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c30fc0b00 0x564c30fc2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c30fc0b00 0x564c30fc2f30 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b69077e0b31f1c7 server_cookie=53a1d018fa444e9c in_seq=0 out_seq=0 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.322+0000 7f6ab6ff5640 10 client.17460 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3165194047 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c30f19c00 con 0x564c30fc0b00 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6ab6ff5640 10 client.17460 handle_command_reply: tid=0 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 client.17460 shutdown 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 msgr2=0x564c30fadae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400cec0 tx=0x7f6ac40047d0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 shutdown_connections 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c30fc0b00 0x564c30fc2f30 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30fae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6ab6ff5640 0 client.17460 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3165194047 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7580 0x564c30fadae0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 >> 172.21.15.87:0/3165194047 conn(0x564c30fa1cb0 msgr2=0x564c30fa3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 shutdown_connections 2024-04-22T13:38:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3165194047 wait complete. 2024-04-22T13:38:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.323+0000 7f6adbe3f740 20 client.17460 trim_cache size 0 max 16384 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.340+0000 7f6adbe3f740 1 Processor -- start 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.340+0000 7f6adbe3f740 1 -- start start 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7580 0x564c30fb0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30facc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41b03a0 con 0x564c30fa7ea0 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41adc70 con 0x564c30fa7580 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41abae0 con 0x564c30fb0a80 2024-04-22T13:38:25.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57564/0 (socket says 172.21.15.87:57564) 2024-04-22T13:38:25.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/4221869810 learned_addr learned my addr 172.21.15.87:0/4221869810 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad359e640 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7580 0x564c30fb0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad359e640 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30facc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/4221869810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 msgr2=0x564c30facc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30facc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/4221869810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7580 msgr2=0x564c30fb0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7580 0x564c30fb0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 -- 172.21.15.87:0/4221869810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c30fb2e70 con 0x564c30fa7ea0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009c50 tx=0x7f6acc010470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e97ef025779374 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ab8ff9640 1 -- 172.21.15.87:0/4221869810 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc016040 con 0x564c30fa7ea0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ab8ff9640 1 -- 172.21.15.87:0/4221869810 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6acc0298e0 con 0x564c30fa7ea0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.341+0000 7f6ab8ff9640 1 -- 172.21.15.87:0/4221869810 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc004880 con 0x564c30fa7ea0 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- 172.21.15.87:0/4221869810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 msgr2=0x564c30fbc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009c50 tx=0x7f6acc010470 comp rx=0 tx=0).stop 2024-04-22T13:38:25.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- 172.21.15.87:0/4221869810 shutdown_connections 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fb0a80 0x564c30facc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fa7ea0 0x564c30fbc730 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/4221869810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7580 0x564c30fb0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- 172.21.15.87:0/4221869810 >> 172.21.15.87:0/4221869810 conn(0x564c30fa1cb0 msgr2=0x564c30fa04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- 172.21.15.87:0/4221869810 shutdown_connections 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- 172.21.15.87:0/4221869810 wait complete. 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- start start 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 Processor -- start 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fb8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fb4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41ac130 con 0x564c30fb0a80 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41ad5a0 con 0x564c30fa7ea0 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad41ac800 con 0x564c30fa7580 2024-04-22T13:38:25.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6adbe3f740 10 client.0 _ll_register_callbacks cb 0x7f6ac400c430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57576/0 (socket says 172.21.15.87:57576) 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.342+0000 7f6ad359e640 1 -- 172.21.15.87:0/3782355353 learned_addr learned my addr 172.21.15.87:0/3782355353 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fb8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fb4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 -- 172.21.15.87:0/3782355353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 msgr2=0x564c30fb8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fb8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 -- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 msgr2=0x564c30fb4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fb4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 -- 172.21.15.87:0/3782355353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c30fb4d30 con 0x564c30fb0a80 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ad359e640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400cec0 tx=0x7f6ac400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98892aa9c87dbb26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4013a60 con 0x564c30fb0a80 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f6ac4013c00 con 0x564c30fb0a80 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6acc0298e0 con 0x564c30fb0a80 2024-04-22T13:38:25.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ac403d050 con 0x564c30fb0a80 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 10 client.17466 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c31026e30 con 0x564c30fb0a80 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6acc0298e0 con 0x564c30fb0a80 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6ac40186b0 con 0x564c30fb0a80 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 10 client.17466 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 20 client.17466 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c31027500 0x564c310298f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 4 client.17466 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.343+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c30f13a70 con 0x564c31027500 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c31027500 0x564c310298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ab6ff5640 10 client.17466 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ad2d9d640 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c31027500 0x564c310298f0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c366c453122e91ea server_cookie=7e79e93af1986608 in_seq=0 out_seq=0 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ab6ff5640 1 -- 172.21.15.87:0/3782355353 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564c30f13a70 con 0x564c31027500 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ab6ff5640 10 client.17466 handle_command_reply: tid=0 2024-04-22T13:38:25.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 client.17466 shutdown 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 msgr2=0x564c30fb4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400cec0 tx=0x7f6ac400cca0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 shutdown_connections 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c31027500 0x564c310298f0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c30fb0a80 0x564c30fb4740 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6ab6ff5640 0 client.17466 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c30fa7ea0 0x564c30fb4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 --2- 172.21.15.87:0/3782355353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c30fa7580 0x564c30fb8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 >> 172.21.15.87:0/3782355353 conn(0x564c30fa1cb0 msgr2=0x564c30fae330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.344+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 shutdown_connections 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6adbe3f740 1 -- 172.21.15.87:0/3782355353 wait complete. 2024-04-22T13:38:25.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6adbe3f740 20 client.17466 trim_cache size 0 max 16384 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "56ecde86-70c8-4f93-9e9c-118d4b1972c2": { 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "56ecde86-70c8-4f93-9e9c-118d4b1972c2", 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:38:25.351 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 msgr2=0x7f6abc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 0x7f6abc0a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80eff20 tx=0x7f6ac80b3b50 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 msgr2=0x7f6ad4198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8004af0 tx=0x7f6ac80346e0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 shutdown_connections 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6abc0a3d20 0x7f6abc0a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ad419d020 0x7f6ad419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ad4106180 0x7f6ad4198ca0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 --2- 172.21.15.87:0/2750458651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ad41010b0 0x7f6ad4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 >> 172.21.15.87:0/2750458651 conn(0x7f6ad40771d0 msgr2=0x7f6ad40fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:25.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 shutdown_connections 2024-04-22T13:38:25.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:25.345+0000 7f6ad924a640 1 -- 172.21.15.87:0/2750458651 wait complete. 2024-04-22T13:38:25.354 INFO:tasks.cephfs.filesystem:scrub status for tag:56ecde86-70c8-4f93-9e9c-118d4b1972c2 - {'path': '/', 'tag': '56ecde86-70c8-4f93-9e9c-118d4b1972c2', 'options': 'recursive,force'} 2024-04-22T13:38:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:26 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:38:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:26 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:26 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:38:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:26 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:26 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:38:26.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:26 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:38:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:26 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:26 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:38:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:26 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:26 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:38:26.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:26 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:38:26.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:26 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:26 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:38:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:26 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:26 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:38:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:27 smithi120 ceph-mon[31101]: pgmap v203: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 166 MiB/s wr, 1.58k op/s 2024-04-22T13:38:27.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:27 smithi132 ceph-mon[31531]: pgmap v203: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 166 MiB/s wr, 1.58k op/s 2024-04-22T13:38:27.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:27 smithi087 ceph-mon[27999]: pgmap v203: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 166 MiB/s wr, 1.58k op/s 2024-04-22T13:38:28.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:28 smithi120 ceph-mon[31101]: pgmap v204: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 148 MiB/s wr, 1.38k op/s 2024-04-22T13:38:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:28 smithi132 ceph-mon[31531]: pgmap v204: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 148 MiB/s wr, 1.38k op/s 2024-04-22T13:38:28.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:28 smithi087 ceph-mon[27999]: pgmap v204: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 148 MiB/s wr, 1.38k op/s 2024-04-22T13:38:30.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:30 smithi120 ceph-mon[31101]: pgmap v205: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 138 MiB/s wr, 1.29k op/s 2024-04-22T13:38:30.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:30 smithi132 ceph-mon[31531]: pgmap v205: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 138 MiB/s wr, 1.29k op/s 2024-04-22T13:38:30.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:30 smithi087 ceph-mon[27999]: pgmap v205: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 138 MiB/s wr, 1.29k op/s 2024-04-22T13:38:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:32 smithi120 ceph-mon[31101]: pgmap v206: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 119 MiB/s wr, 1.11k op/s 2024-04-22T13:38:32.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:32 smithi132 ceph-mon[31531]: pgmap v206: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 119 MiB/s wr, 1.11k op/s 2024-04-22T13:38:32.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:32 smithi087 ceph-mon[27999]: pgmap v206: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 119 MiB/s wr, 1.11k op/s 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 -- 172.21.15.87:0/2262821790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528806a9f0 msgr2=0x7f528806add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 --2- 172.21.15.87:0/2262821790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528806a9f0 0x7f528806add0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5278009900 tx=0x7f527802bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 -- 172.21.15.87:0/2262821790 shutdown_connections 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 --2- 172.21.15.87:0/2262821790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5288107260 0x7f528810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 --2- 172.21.15.87:0/2262821790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528806b3a0 0x7f5288106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 --2- 172.21.15.87:0/2262821790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528806a9f0 0x7f528806add0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.890+0000 7f528e12c640 1 -- 172.21.15.87:0/2262821790 >> 172.21.15.87:0/2262821790 conn(0x7f5288076b80 msgr2=0x7f5288076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 -- 172.21.15.87:0/2262821790 shutdown_connections 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 -- 172.21.15.87:0/2262821790 wait complete. 2024-04-22T13:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 Processor -- start 2024-04-22T13:38:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 -- start start 2024-04-22T13:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 0x7f52881986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.891+0000 7f528e12c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 0x7f5288198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f52881121e0 con 0x7f528819cf60 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5288112060 con 0x7f528806a9f0 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5288112360 con 0x7f5288107260 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48898/0 (socket says 172.21.15.87:48898) 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 -- 172.21.15.87:0/267184130 learned_addr learned my addr 172.21.15.87:0/267184130 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5286ffd640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 0x7f5288198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f52877fe640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 0x7f52881986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 -- 172.21.15.87:0/267184130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 msgr2=0x7f5288198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 0x7f5288198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 -- 172.21.15.87:0/267184130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 msgr2=0x7f52881986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 0x7f52881986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 -- 172.21.15.87:0/267184130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528819da50 con 0x7f528819cf60 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5286ffd640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 0x7f5288198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f52877fe640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 0x7f52881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5287fff640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f528000eaa0 tx=0x7f528000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5284ff9640 1 -- 172.21.15.87:0/267184130 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528000cdb0 con 0x7f528819cf60 2024-04-22T13:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5284ff9640 1 -- 172.21.15.87:0/267184130 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5280005020 con 0x7f528819cf60 2024-04-22T13:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528819dd40 con 0x7f528819cf60 2024-04-22T13:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f5284ff9640 1 -- 172.21.15.87:0/267184130 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52800116d0 con 0x7f528819cf60 2024-04-22T13:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.892+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52881a5530 con 0x7f528819cf60 2024-04-22T13:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5284ff9640 1 -- 172.21.15.87:0/267184130 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f52800051d0 con 0x7f528819cf60 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 Processor -- start 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 -- start start 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5284ff9640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 0x7f52700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e80620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e8c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556b37de58d0 con 0x556b37e80b60 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b37d82e80 con 0x556b37e8a0e0 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5290df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b37de1890 con 0x556b37e818d0 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.895+0000 7f5286ffd640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 0x7f52700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5284ff9640 1 -- 172.21.15.87:0/267184130 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f52800d0970 con 0x7f528819cf60 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48902/0 (socket says 172.21.15.87:48902) 2024-04-22T13:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 -- 172.21.15.87:0/3111982703 learned_addr learned my addr 172.21.15.87:0/3111982703 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f52877fe640 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e80620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5287fff640 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e8c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 -- 172.21.15.87:0/3111982703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 msgr2=0x556b37e80620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e80620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 -- 172.21.15.87:0/3111982703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 msgr2=0x556b37e8c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e8c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 -- 172.21.15.87:0/3111982703 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b37da4320 con 0x556b37e80b60 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f527c013b40 tx=0x7f527c0103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e871bca565f96e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f5286ffd640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 0x7f52700a6300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f527c0108d0 tx=0x7f527c01b000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f526dffb640 1 -- 172.21.15.87:0/3111982703 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c021070 con 0x556b37e80b60 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f526dffb640 1 -- 172.21.15.87:0/3111982703 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c00b470 con 0x556b37e80b60 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.896+0000 7f526dffb640 1 -- 172.21.15.87:0/3111982703 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c005280 con 0x556b37e80b60 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- 172.21.15.87:0/3111982703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 msgr2=0x556b37e80fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f527c013b40 tx=0x7f527c0103d0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- 172.21.15.87:0/3111982703 shutdown_connections 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e8c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e80fe0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 --2- 172.21.15.87:0/3111982703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e80620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- 172.21.15.87:0/3111982703 >> 172.21.15.87:0/3111982703 conn(0x556b37e47c90 msgr2=0x556b37e7b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- 172.21.15.87:0/3111982703 shutdown_connections 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- 172.21.15.87:0/3111982703 wait complete. 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.897+0000 7f5290df0740 1 -- start start 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 Processor -- start 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e80b60 0x556b37e89b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e818d0 0x556b37e86ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556b37e8cdb0 con 0x556b37e8a0e0 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b37d6deb0 con 0x556b37e818d0 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b37e8cf30 con 0x556b37e80b60 2024-04-22T13:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5290df0740 10 client.0 _ll_register_callbacks cb 0x556b37d9ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48918/0 (socket says 172.21.15.87:48918) 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 -- 172.21.15.87:0/975977098 learned_addr learned my addr 172.21.15.87:0/975977098 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5286ffd640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e818d0 0x556b37e86ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f52877fe640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e80b60 0x556b37e89b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 -- 172.21.15.87:0/975977098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e80b60 msgr2=0x556b37e89b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e80b60 0x556b37e89b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 -- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e818d0 msgr2=0x556b37e86ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e818d0 0x556b37e86ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.898+0000 7f5287fff640 1 -- 172.21.15.87:0/975977098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b37af8210 con 0x556b37e8a0e0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5287fff640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f5280099320 tx=0x7f528009d180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fea5ff868dcac31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528001f040 con 0x556b37e8a0e0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f5280087150 con 0x556b37e8a0e0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556b37df9880 con 0x556b37e8a0e0 2024-04-22T13:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f52800a55c0 con 0x556b37e8a0e0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 10 client.17478 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556b37bc4620 con 0x556b37e8a0e0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556b37df9880 con 0x556b37e8a0e0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f528000f040 con 0x556b37e8a0e0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 10 client.17478 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 10 client.17478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 20 client.17478 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37e9a160 0x556b37e9c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 4 client.17478 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.899+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b37d853d0 con 0x556b37e9a160 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.900+0000 7f52877fe640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37e9a160 0x556b37e9c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.900+0000 7f52877fe640 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37e9a160 0x556b37e9c550 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de50ad4b8c02ed4 server_cookie=489373236e8bc3ce in_seq=0 out_seq=0 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.900+0000 7f5266ffd640 10 client.17478 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5266ffd640 1 -- 172.21.15.87:0/975977098 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b37d853d0 con 0x556b37e9a160 2024-04-22T13:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5266ffd640 10 client.17478 handle_command_reply: tid=0 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 client.17478 shutdown 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 msgr2=0x556b37e875f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f5280099320 tx=0x7f528009d180 comp rx=0 tx=0).stop 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 shutdown_connections 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37e9a160 0x556b37e9c550 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e8a0e0 0x556b37e875f0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e818d0 0x556b37e86ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5266ffd640 0 client.17478 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 --2- 172.21.15.87:0/975977098 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e80b60 0x556b37e89b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 >> 172.21.15.87:0/975977098 conn(0x556b37e47c90 msgr2=0x556b37e7d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 shutdown_connections 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 1 -- 172.21.15.87:0/975977098 wait complete. 2024-04-22T13:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.901+0000 7f5290df0740 20 client.17478 trim_cache size 0 max 16384 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 Processor -- start 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 -- start start 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e80b60 0x556b37e86e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e8a0e0 0x556b37e89600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f52881b0200 con 0x556b37e818d0 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52881adad0 con 0x556b37e80b60 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.916+0000 7f5290df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52881ab940 con 0x556b37e8a0e0 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48934/0 (socket says 172.21.15.87:48934) 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 -- 172.21.15.87:0/167617277 learned_addr learned my addr 172.21.15.87:0/167617277 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f52877fe640 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e80b60 0x556b37e86e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5287fff640 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e8a0e0 0x556b37e89600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 -- 172.21.15.87:0/167617277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e8a0e0 msgr2=0x556b37e89600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e8a0e0 0x556b37e89600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 -- 172.21.15.87:0/167617277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e80b60 msgr2=0x556b37e86e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e80b60 0x556b37e86e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 -- 172.21.15.87:0/167617277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b37e8ce00 con 0x556b37e818d0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f5286ffd640 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f527c005ee0 tx=0x7f527c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1069ec84f2a6bfef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f526d7fa640 1 -- 172.21.15.87:0/167617277 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c032070 con 0x556b37e818d0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f526d7fa640 1 -- 172.21.15.87:0/167617277 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c013740 con 0x556b37e818d0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.917+0000 7f526d7fa640 1 -- 172.21.15.87:0/167617277 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c021050 con 0x556b37e818d0 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- 172.21.15.87:0/167617277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 msgr2=0x556b37e95cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f527c005ee0 tx=0x7f527c013040 comp rx=0 tx=0).stop 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- 172.21.15.87:0/167617277 shutdown_connections 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e8a0e0 0x556b37e89600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e818d0 0x556b37e95cb0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 --2- 172.21.15.87:0/167617277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e80b60 0x556b37e86e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- 172.21.15.87:0/167617277 >> 172.21.15.87:0/167617277 conn(0x556b37e47c90 msgr2=0x556b37e86760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- 172.21.15.87:0/167617277 shutdown_connections 2024-04-22T13:38:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- 172.21.15.87:0/167617277 wait complete. 2024-04-22T13:38:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.918+0000 7f5290df0740 1 -- start start 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 Processor -- start 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e92700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e84150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f52881abf90 con 0x556b37e80b60 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52881ad400 con 0x556b37e8a0e0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52881ac660 con 0x556b37e818d0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5290df0740 10 client.0 _ll_register_callbacks cb 0x7f528000cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48936/0 (socket says 172.21.15.87:48936) 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 -- 172.21.15.87:0/1750226076 learned_addr learned my addr 172.21.15.87:0/1750226076 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5287fff640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e84150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5286ffd640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e92700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 -- 172.21.15.87:0/1750226076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 msgr2=0x556b37e92700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e92700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 -- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 msgr2=0x556b37e84150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e84150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 -- 172.21.15.87:0/1750226076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b37e846e0 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5287fff640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e84150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f52877fe640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f527802c170 tx=0x7f52780045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc0213a168d5830 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.919+0000 7f5266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527804d070 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f527802c860 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f527c013740 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5278058050 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 10 client.17484 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556b37e8ada0 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c013740 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5278005370 con 0x556b37e80b60 2024-04-22T13:38:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 10 client.17484 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 10 client.17484 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 20 client.17484 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37f006d0 0x556b37f02af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 4 client.17484 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556b37df32f0 con 0x556b37f006d0 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.920+0000 7f5286ffd640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37f006d0 0x556b37f02af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.921+0000 7f5266ffd640 10 client.17484 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.921+0000 7f5286ffd640 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37f006d0 0x556b37f02af0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=488ae27c0a0d438 server_cookie=acff07f082cb1ba9 in_seq=0 out_seq=0 2024-04-22T13:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5266ffd640 1 -- 172.21.15.87:0/1750226076 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556b37df32f0 con 0x556b37f006d0 2024-04-22T13:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5266ffd640 10 client.17484 handle_command_reply: tid=0 2024-04-22T13:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 client.17484 shutdown 2024-04-22T13:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 msgr2=0x556b37e921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f527802c170 tx=0x7f52780045b0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 shutdown_connections 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556b37f006d0 0x556b37f02af0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b37e8a0e0 0x556b37e84150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b37e818d0 0x556b37e92700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5266ffd640 0 client.17484 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 --2- 172.21.15.87:0/1750226076 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556b37e80b60 0x556b37e921c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 >> 172.21.15.87:0/1750226076 conn(0x556b37e47c90 msgr2=0x556b37e7a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 shutdown_connections 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 1 -- 172.21.15.87:0/1750226076 wait complete. 2024-04-22T13:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.933+0000 7f5290df0740 20 client.17484 trim_cache size 0 max 16384 2024-04-22T13:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 msgr2=0x7f52700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 0x7f52700a6300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f527c0108d0 tx=0x7f527c01b000 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 msgr2=0x7f528819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f528000eaa0 tx=0x7f528000ef60 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 shutdown_connections 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f52700a3e40 0x7f52700a6300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f528819cf60 0x7f528819d410 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5288107260 0x7f5288198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 --2- 172.21.15.87:0/267184130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f528806a9f0 0x7f52881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 >> 172.21.15.87:0/267184130 conn(0x7f5288076b80 msgr2=0x7f52880ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 shutdown_connections 2024-04-22T13:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:33.934+0000 7f528e12c640 1 -- 172.21.15.87:0/267184130 wait complete. 2024-04-22T13:38:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:34 smithi120 ceph-mon[31101]: pgmap v207: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 110 MiB/s wr, 1.03k op/s 2024-04-22T13:38:34.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:34 smithi132 ceph-mon[31531]: pgmap v207: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 110 MiB/s wr, 1.03k op/s 2024-04-22T13:38:34.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:34 smithi087 ceph-mon[27999]: pgmap v207: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 110 MiB/s wr, 1.03k op/s 2024-04-22T13:38:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:36 smithi120 ceph-mon[31101]: pgmap v208: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 MiB/s wr, 312 op/s 2024-04-22T13:38:36.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:36 smithi132 ceph-mon[31531]: pgmap v208: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 MiB/s wr, 312 op/s 2024-04-22T13:38:36.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:36 smithi087 ceph-mon[27999]: pgmap v208: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 33 MiB/s wr, 312 op/s 2024-04-22T13:38:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:38 smithi120 ceph-mon[31101]: pgmap v209: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 69 op/s 2024-04-22T13:38:38.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:38 smithi132 ceph-mon[31531]: pgmap v209: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 69 op/s 2024-04-22T13:38:38.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:38 smithi087 ceph-mon[27999]: pgmap v209: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 69 op/s 2024-04-22T13:38:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:40 smithi120 ceph-mon[31101]: pgmap v210: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2024-04-22T13:38:40.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:40 smithi132 ceph-mon[31531]: pgmap v210: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2024-04-22T13:38:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:40 smithi087 ceph-mon[27999]: pgmap v210: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2024-04-22T13:38:42.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:42 smithi120 ceph-mon[31101]: pgmap v211: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 37 op/s 2024-04-22T13:38:42.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:42 smithi132 ceph-mon[31531]: pgmap v211: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 37 op/s 2024-04-22T13:38:42.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:42 smithi087 ceph-mon[27999]: pgmap v211: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 37 op/s 2024-04-22T13:38:44.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:44 smithi120 ceph-mon[31101]: pgmap v212: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 211 op/s 2024-04-22T13:38:44.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:44 smithi132 ceph-mon[31531]: pgmap v212: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 211 op/s 2024-04-22T13:38:44.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:44 smithi087 ceph-mon[27999]: pgmap v212: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 MiB/s wr, 211 op/s 2024-04-22T13:38:46.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:46 smithi120 ceph-mon[31101]: pgmap v213: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 29 MiB/s wr, 283 op/s 2024-04-22T13:38:46.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:46 smithi132 ceph-mon[31531]: pgmap v213: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 29 MiB/s wr, 283 op/s 2024-04-22T13:38:46.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:46 smithi087 ceph-mon[27999]: pgmap v213: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 29 MiB/s wr, 283 op/s 2024-04-22T13:38:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 -- 172.21.15.87:0/344417956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 msgr2=0x7f07ac06ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 --2- 172.21.15.87:0/344417956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac06ad80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009900 tx=0x7f07a002bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 -- 172.21.15.87:0/344417956 shutdown_connections 2024-04-22T13:38:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 --2- 172.21.15.87:0/344417956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 0x7f07ac10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 --2- 172.21.15.87:0/344417956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac06b350 0x7f07ac106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 --2- 172.21.15.87:0/344417956 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac06ad80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 -- 172.21.15.87:0/344417956 >> 172.21.15.87:0/344417956 conn(0x7f07ac077c30 msgr2=0x7f07ac078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 -- 172.21.15.87:0/344417956 shutdown_connections 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.028+0000 7f07b31f4640 1 -- 172.21.15.87:0/344417956 wait complete. 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 Processor -- start 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 -- start start 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 0x7f07ac198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac19cfc0 0x7f07ac19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1121f0 con 0x7f07ac06a9a0 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac112070 con 0x7f07ac19cfc0 2024-04-22T13:38:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b31f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac112370 con 0x7f07ac107280 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42528/0 (socket says 172.21.15.87:42528) 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 -- 172.21.15.87:0/1581970949 learned_addr learned my addr 172.21.15.87:0/1581970949 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b1791640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac19cfc0 0x7f07ac19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07abfff640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 0x7f07ac198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 -- 172.21.15.87:0/1581970949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 msgr2=0x7f07ac198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 0x7f07ac198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 -- 172.21.15.87:0/1581970949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac19cfc0 msgr2=0x7f07ac19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac19cfc0 0x7f07ac19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.029+0000 7f07b0f90640 1 -- 172.21.15.87:0/1581970949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac19dab0 con 0x7f07ac06a9a0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07b0f90640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005940 tx=0x7f07a002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07a9ffb640 1 -- 172.21.15.87:0/1581970949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0043020 con 0x7f07ac06a9a0 2024-04-22T13:38:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ac19dd40 con 0x7f07ac06a9a0 2024-04-22T13:38:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07a9ffb640 1 -- 172.21.15.87:0/1581970949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07a002cbe0 con 0x7f07ac06a9a0 2024-04-22T13:38:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07a9ffb640 1 -- 172.21.15.87:0/1581970949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0003bf0 con 0x7f07ac06a9a0 2024-04-22T13:38:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.030+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ac1a5560 con 0x7f07ac06a9a0 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.032+0000 7f07a9ffb640 1 -- 172.21.15.87:0/1581970949 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f07a0046020 con 0x7f07ac06a9a0 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 Processor -- start 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07a9ffb640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 0x7f079c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 -- start start 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe58d0 0x558ceafe4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceaff24f0 0x558ceaff48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558ceaf498d0 con 0x558ceafe4b60 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b1791640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 0x7f079c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ceaee6e80 con 0x558ceafe58d0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b5de5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558ceaf45890 con 0x558ceaff24f0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07a9ffb640 1 -- 172.21.15.87:0/1581970949 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f07a00f2930 con 0x7f07ac06a9a0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42536/0 (socket says 172.21.15.87:42536) 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 -- 172.21.15.87:0/3174854381 learned_addr learned my addr 172.21.15.87:0/3174854381 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b0f90640 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe58d0 0x558ceafe4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b1791640 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceaff24f0 0x558ceaff48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 -- 172.21.15.87:0/3174854381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceaff24f0 msgr2=0x558ceaff48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceaff24f0 0x558ceaff48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 -- 172.21.15.87:0/3174854381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe58d0 msgr2=0x558ceafe4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07abfff640 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe58d0 0x558ceafe4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.033+0000 7f07b1791640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 0x7f079c0a6280 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f07a400a780 tx=0x7f07a400f790 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f07abfff640 1 -- 172.21.15.87:0/3174854381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ceaf08320 con 0x558ceafe4b60 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f07abfff640 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0798005fd0 tx=0x7f079800e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505ec97e3c831175 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f0796ffd640 1 -- 172.21.15.87:0/3174854381 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798016070 con 0x558ceafe4b60 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f0796ffd640 1 -- 172.21.15.87:0/3174854381 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07980054d0 con 0x558ceafe4b60 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f0796ffd640 1 -- 172.21.15.87:0/3174854381 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798011470 con 0x558ceafe4b60 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f07b5de5740 1 -- 172.21.15.87:0/3174854381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 msgr2=0x558ceafe4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.034+0000 7f07b5de5740 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0798005fd0 tx=0x7f079800e2f0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- 172.21.15.87:0/3174854381 shutdown_connections 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceaff24f0 0x558ceaff48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceafe4fe0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- 172.21.15.87:0/3174854381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe58d0 0x558ceafe4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- 172.21.15.87:0/3174854381 >> 172.21.15.87:0/3174854381 conn(0x558ceafabc90 msgr2=0x558ceafdf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- 172.21.15.87:0/3174854381 shutdown_connections 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- 172.21.15.87:0/3174854381 wait complete. 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- start start 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 Processor -- start 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558ceaff5220 con 0x558ceafe4b60 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ceaed1eb0 con 0x558ceaff24f0 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558ceaff53a0 con 0x558ceafe58d0 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.035+0000 7f07b5de5740 10 client.0 _ll_register_callbacks cb 0x558ceaefeb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42538/0 (socket says 172.21.15.87:42538) 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 -- 172.21.15.87:0/404973601 learned_addr learned my addr 172.21.15.87:0/404973601 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07abfff640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b1791640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 -- 172.21.15.87:0/404973601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 msgr2=0x558ceafef1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafef1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 -- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 msgr2=0x558ceafef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 -- 172.21.15.87:0/404973601 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ceac5c210 con 0x558ceafe4b60 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b0f90640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f07a0041000 tx=0x7f07a00be620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc4ee9037025d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f078bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00c2dd0 con 0x558ceafe4b60 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f07a00bb090 con 0x558ceafe4b60 2024-04-22T13:38:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07980054d0 con 0x558ceafe4b60 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f07a00c7690 con 0x558ceafe4b60 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b5de5740 10 client.17496 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.036+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558cead28620 con 0x558ceafe4b60 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07980054d0 con 0x558ceafe4b60 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f07a00dc050 con 0x558ceafe4b60 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 10 client.17496 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 10 client.17496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 20 client.17496 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb002590 0x558ceb0049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 4 client.17496 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ceaf5d880 con 0x558ceb002590 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07abfff640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb002590 0x558ceb0049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f078bfff640 10 client.17496 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.037+0000 7f07abfff640 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb002590 0x558ceb0049a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3d75e1fd180e21 server_cookie=e005086b54c6e7da in_seq=0 out_seq=0 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f078bfff640 1 -- 172.21.15.87:0/404973601 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ceaf5d880 con 0x558ceb002590 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f078bfff640 10 client.17496 handle_command_reply: tid=0 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 client.17496 shutdown 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 msgr2=0x558ceaff1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f07a0041000 tx=0x7f07a00be620 comp rx=0 tx=0).stop 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 shutdown_connections 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb002590 0x558ceb0049a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f078bfff640 0 client.17496 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 --2- 172.21.15.87:0/404973601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaff1e40 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 >> 172.21.15.87:0/404973601 conn(0x558ceafabc90 msgr2=0x558ceafe15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 shutdown_connections 2024-04-22T13:38:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.038+0000 7f07b5de5740 1 -- 172.21.15.87:0/404973601 wait complete. 2024-04-22T13:38:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.039+0000 7f07b5de5740 20 client.17496 trim_cache size 0 max 16384 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 Processor -- start 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 -- start start 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe4b60 0x558ceaffe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafda760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1ae040 con 0x558ceaff24f0 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1ab910 con 0x558ceafe4b60 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.053+0000 7f07b5de5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1a97a0 con 0x558ceafe58d0 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42546/0 (socket says 172.21.15.87:42546) 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 -- 172.21.15.87:0/947509820 learned_addr learned my addr 172.21.15.87:0/947509820 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07abfff640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafda760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 -- 172.21.15.87:0/947509820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 msgr2=0x558ceafda760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafda760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 -- 172.21.15.87:0/947509820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe4b60 msgr2=0x558ceaffe180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b0f90640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe4b60 0x558ceaffe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe4b60 0x558ceaffe180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 -- 172.21.15.87:0/947509820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ceafdaf30 con 0x558ceaff24f0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07abfff640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafda760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07b1791640 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f07a401d700 tx=0x7f07a401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11d2a3e1f0fb961 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07967fc640 1 -- 172.21.15.87:0/947509820 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4018070 con 0x558ceaff24f0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07967fc640 1 -- 172.21.15.87:0/947509820 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07a4010840 con 0x558ceaff24f0 2024-04-22T13:38:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.054+0000 7f07967fc640 1 -- 172.21.15.87:0/947509820 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40055c0 con 0x558ceaff24f0 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- 172.21.15.87:0/947509820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 msgr2=0x558ceaff1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f07a401d700 tx=0x7f07a401a000 comp rx=0 tx=0).stop 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- 172.21.15.87:0/947509820 shutdown_connections 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceaff24f0 0x558ceaff1900 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafda760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 --2- 172.21.15.87:0/947509820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceafe4b60 0x558ceaffe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- 172.21.15.87:0/947509820 >> 172.21.15.87:0/947509820 conn(0x558ceafabc90 msgr2=0x558ceafedaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- 172.21.15.87:0/947509820 shutdown_connections 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- 172.21.15.87:0/947509820 wait complete. 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 -- start start 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.055+0000 7f07b5de5740 1 Processor -- start 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafec530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1a9df0 con 0x558ceafe4b60 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1ab240 con 0x558ceaff24f0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac1aa4a0 con 0x558ceafe58d0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 10 client.0 _ll_register_callbacks cb 0x7f07a0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42558/0 (socket says 172.21.15.87:42558) 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 -- 172.21.15.87:0/671834450 learned_addr learned my addr 172.21.15.87:0/671834450 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b1791640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07abfff640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 -- 172.21.15.87:0/671834450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 msgr2=0x558ceafec530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafec530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 -- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 msgr2=0x558ceafecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 -- 172.21.15.87:0/671834450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ceaff2eb0 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b0f90640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f07a00f2c50 tx=0x7f07a0005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62430ac5b5e5daa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f078bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0088060 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f07a00bee80 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.056+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07a4010840 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f07a00bacc0 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 10 client.17502 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558ceaff2eb0 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07a4010840 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f07a00dc070 con 0x558ceafe4b60 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 10 client.17502 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 10 client.17502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 20 client.17502 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb069060 0x558ceb06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 4 client.17502 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ceaee93d0 con 0x558ceb069060 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.057+0000 7f07abfff640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb069060 0x558ceb06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.058+0000 7f078bfff640 10 client.17502 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.058+0000 7f07abfff640 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb069060 0x558ceb06b450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ae59d8065f8f7f2 server_cookie=5278e43333b96aef in_seq=0 out_seq=0 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.066+0000 7f078bfff640 1 -- 172.21.15.87:0/671834450 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ceaee93d0 con 0x558ceb069060 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.066+0000 7f078bfff640 10 client.17502 handle_command_reply: tid=0 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.066+0000 7f07b5de5740 1 client.17502 shutdown 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 msgr2=0x558ceaffab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f07a00f2c50 tx=0x7f07a0005700 comp rx=0 tx=0).stop 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 shutdown_connections 2024-04-22T13:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558ceb069060 0x558ceb06b450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaff24f0 0x558ceafecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f078bfff640 0 client.17502 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ceafe58d0 0x558ceafec530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 --2- 172.21.15.87:0/671834450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558ceafe4b60 0x558ceaffab70 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 >> 172.21.15.87:0/671834450 conn(0x558ceafabc90 msgr2=0x558ceafde040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 shutdown_connections 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 1 -- 172.21.15.87:0/671834450 wait complete. 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b5de5740 20 client.17502 trim_cache size 0 max 16384 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 msgr2=0x7f079c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 0x7f079c0a6280 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f07a400a780 tx=0x7f07a400f790 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 msgr2=0x7f07ac198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005940 tx=0x7f07a002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.067+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 shutdown_connections 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f079c0a3dc0 0x7f079c0a6280 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac19cfc0 0x7f07ac19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ac107280 0x7f07ac198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 --2- 172.21.15.87:0/1581970949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac06a9a0 0x7f07ac198750 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 >> 172.21.15.87:0/1581970949 conn(0x7f07ac077c30 msgr2=0x7f07ac0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 shutdown_connections 2024-04-22T13:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:38:47.068+0000 7f07b31f4640 1 -- 172.21.15.87:0/1581970949 wait complete. 2024-04-22T13:38:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:48 smithi120 ceph-mon[31101]: pgmap v214: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 53 MiB/s wr, 535 op/s 2024-04-22T13:38:48.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:48 smithi132 ceph-mon[31531]: pgmap v214: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 53 MiB/s wr, 535 op/s 2024-04-22T13:38:48.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:48 smithi087 ceph-mon[27999]: pgmap v214: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 53 MiB/s wr, 535 op/s 2024-04-22T13:38:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:50 smithi120 ceph-mon[31101]: pgmap v215: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 58 MiB/s wr, 586 op/s 2024-04-22T13:38:50.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:50 smithi132 ceph-mon[31531]: pgmap v215: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 58 MiB/s wr, 586 op/s 2024-04-22T13:38:50.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:50 smithi087 ceph-mon[27999]: pgmap v215: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 58 MiB/s wr, 586 op/s 2024-04-22T13:38:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:52 smithi120 ceph-mon[31101]: pgmap v216: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:52.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:52 smithi132 ceph-mon[31531]: pgmap v216: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:52.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:52 smithi087 ceph-mon[27999]: pgmap v216: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:54 smithi120 ceph-mon[31101]: pgmap v217: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:54 smithi132 ceph-mon[31531]: pgmap v217: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:54.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:54 smithi087 ceph-mon[27999]: pgmap v217: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 639 op/s 2024-04-22T13:38:55.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:38:55.354 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:38:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 -- 172.21.15.87:0/3701574393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9830102b60 msgr2=0x7f9830106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 --2- 172.21.15.87:0/3701574393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9830102b60 0x7f9830106ba0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9818009900 tx=0x7f981802bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 -- 172.21.15.87:0/3701574393 shutdown_connections 2024-04-22T13:38:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 --2- 172.21.15.87:0/3701574393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 0x7f983010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 --2- 172.21.15.87:0/3701574393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9830102b60 0x7f9830106ba0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 --2- 172.21.15.87:0/3701574393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98301021b0 0x7f9830102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 -- 172.21.15.87:0/3701574393 >> 172.21.15.87:0/3701574393 conn(0x7f98300fda10 msgr2=0x7f98300ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 -- 172.21.15.87:0/3701574393 shutdown_connections 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.472+0000 7f983605c640 1 -- 172.21.15.87:0/3701574393 wait complete. 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 Processor -- start 2024-04-22T13:38:55.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 -- start start 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 0x7f9830198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 0x7f983019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f98301121a0 con 0x7f98301021b0 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9830112020 con 0x7f983019cfa0 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f983605c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9830112320 con 0x7f9830107280 2024-04-22T13:38:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49626/0 (socket says 172.21.15.87:49626) 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 -- 172.21.15.87:0/3307105499 learned_addr learned my addr 172.21.15.87:0/3307105499 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f9827fff640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 0x7f9830198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982ffff640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 0x7f983019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 -- 172.21.15.87:0/3307105499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 msgr2=0x7f9830198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 0x7f9830198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 -- 172.21.15.87:0/3307105499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 msgr2=0x7f983019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 0x7f983019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982f7fe640 1 -- 172.21.15.87:0/3307105499 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983019da90 con 0x7f98301021b0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.473+0000 7f982ffff640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 0x7f983019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f982000d950 tx=0x7f982000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f982d7fa640 1 -- 172.21.15.87:0/3307105499 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820017070 con 0x7f98301021b0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f982d7fa640 1 -- 172.21.15.87:0/3307105499 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9820009c30 con 0x7f98301021b0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983019dd80 con 0x7f98301021b0 2024-04-22T13:38:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f982d7fa640 1 -- 172.21.15.87:0/3307105499 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982000f620 con 0x7f98301021b0 2024-04-22T13:38:55.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.474+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98301a5540 con 0x7f98301021b0 2024-04-22T13:38:55.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.476+0000 7f982d7fa640 1 -- 172.21.15.87:0/3307105499 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9820010460 con 0x7f98301021b0 2024-04-22T13:38:55.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.476+0000 7f9838c4d740 1 Processor -- start 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.476+0000 7f982d7fa640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 0x7f981c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.476+0000 7f9838c4d740 1 -- start start 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e464960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982ffff640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 0x7f981c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d91e3c91e0 con 0x55d91e464ea0 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d91e366390 con 0x55d91e46da80 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9838c4d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d91e3c51a0 con 0x55d91e464580 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982d7fa640 1 -- 172.21.15.87:0/3307105499 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f98200d0400 con 0x7f98301021b0 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49628/0 (socket says 172.21.15.87:49628) 2024-04-22T13:38:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 -- 172.21.15.87:0/2070717935 learned_addr learned my addr 172.21.15.87:0/2070717935 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982f7fe640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e464960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982ffff640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 -- 172.21.15.87:0/2070717935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 msgr2=0x55d91e464960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e464960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 -- 172.21.15.87:0/2070717935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 msgr2=0x55d91e46fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 -- 172.21.15.87:0/2070717935 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91e1a57d0 con 0x55d91e464ea0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982ffff640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982f7fe640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e464960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f982ffff640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 0x7f981c0a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9828005cc0 tx=0x7f982800e720 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9827fff640 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9818009900 tx=0x7f981802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d362c59fa5b127ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.477+0000 7f9825ffb640 1 -- 172.21.15.87:0/2070717935 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818045020 con 0x55d91e464ea0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9825ffb640 1 -- 172.21.15.87:0/2070717935 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f981802c7e0 con 0x55d91e464ea0 2024-04-22T13:38:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9825ffb640 1 -- 172.21.15.87:0/2070717935 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180046c0 con 0x55d91e464ea0 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 -- 172.21.15.87:0/2070717935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 msgr2=0x55d91e46d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9818009900 tx=0x7f981802c6c0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 -- 172.21.15.87:0/2070717935 shutdown_connections 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46d540 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2070717935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e464960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 -- 172.21.15.87:0/2070717935 >> 172.21.15.87:0/2070717935 conn(0x55d91e45ecb0 msgr2=0x55d91e45f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 -- 172.21.15.87:0/2070717935 shutdown_connections 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.478+0000 7f9838c4d740 1 -- 172.21.15.87:0/2070717935 wait complete. 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 -- start start 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 Processor -- start 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e46a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d91e474c20 con 0x55d91e464ea0 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d91e3822f0 con 0x55d91e46da80 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d91e474da0 con 0x55d91e464580 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9838c4d740 10 client.0 _ll_register_callbacks cb 0x55d91e38f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49632/0 (socket says 172.21.15.87:49632) 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 -- 172.21.15.87:0/4165040738 learned_addr learned my addr 172.21.15.87:0/4165040738 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f982ffff640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f982f7fe640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e46a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 -- 172.21.15.87:0/4165040738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 msgr2=0x55d91e46a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e46a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 -- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 msgr2=0x55d91e46b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f9827fff640 1 -- 172.21.15.87:0/4165040738 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d91e0db3f0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.479+0000 7f982ffff640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9827fff640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9818005750 tx=0x7f9818005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9399fdb3f758d4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180046c0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f9818004860 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f981802c7e0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f981805b050 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 10 client.17514 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d91e09b0f0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f981802c7e0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9818003cb0 con 0x55d91e464ea0 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 10 client.17514 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 20 client.17514 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e481b50 0x55d91e483f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 4 client.17514 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.480+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d91e3d6c00 con 0x55d91e481b50 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.481+0000 7f982f7fe640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e481b50 0x55d91e483f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.481+0000 7f982f7fe640 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e481b50 0x55d91e483f40 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcdf335c4fcffd2a server_cookie=e2011042dfa474ae in_seq=0 out_seq=0 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.481+0000 7f980effd640 10 client.17514 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f980effd640 1 -- 172.21.15.87:0/4165040738 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d91e3d6c00 con 0x55d91e481b50 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f980effd640 10 client.17514 handle_command_reply: tid=0 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 client.17514 shutdown 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 msgr2=0x55d91e46af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9818005750 tx=0x7f9818005300 comp rx=0 tx=0).stop 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 shutdown_connections 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e481b50 0x55d91e483f40 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e46da80 0x55d91e46b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f980effd640 0 client.17514 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:55.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464ea0 0x55d91e46af20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 --2- 172.21.15.87:0/4165040738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464580 0x55d91e46a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 >> 172.21.15.87:0/4165040738 conn(0x55d91e45ecb0 msgr2=0x55d91e460a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 shutdown_connections 2024-04-22T13:38:55.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 1 -- 172.21.15.87:0/4165040738 wait complete. 2024-04-22T13:38:55.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.482+0000 7f9838c4d740 20 client.17514 trim_cache size 0 max 16384 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 Processor -- start 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 -- start start 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 0x55d91e47dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e46da80 0x55d91e460810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f98301ae0e0 con 0x55d91e464580 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98301ab9b0 con 0x55d91e464ea0 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.499+0000 7f9838c4d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98301a9820 con 0x55d91e46da80 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49636/0 (socket says 172.21.15.87:49636) 2024-04-22T13:38:55.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 -- 172.21.15.87:0/3703690123 learned_addr learned my addr 172.21.15.87:0/3703690123 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 -- 172.21.15.87:0/3703690123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e46da80 msgr2=0x55d91e460810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e46da80 0x55d91e460810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 -- 172.21.15.87:0/3703690123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 msgr2=0x55d91e47dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f9827fff640 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 0x55d91e47dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 0x55d91e47dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 -- 172.21.15.87:0/3703690123 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91e45eae0 con 0x55d91e464580 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f9827fff640 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 0x55d91e47dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f982f7fe640 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9820098db0 tx=0x7f9820005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cb7c1ba5553c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f98257fa640 1 -- 172.21.15.87:0/3703690123 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820020040 con 0x55d91e464580 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f98257fa640 1 -- 172.21.15.87:0/3703690123 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98200a5050 con 0x55d91e464580 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.500+0000 7f98257fa640 1 -- 172.21.15.87:0/3703690123 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982005d4c0 con 0x55d91e464580 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- 172.21.15.87:0/3703690123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 msgr2=0x55d91e46d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9820098db0 tx=0x7f9820005bd0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- 172.21.15.87:0/3703690123 shutdown_connections 2024-04-22T13:38:55.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e46da80 0x55d91e460810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464ea0 0x55d91e47dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- 172.21.15.87:0/3703690123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e464580 0x55d91e46d110 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- 172.21.15.87:0/3703690123 >> 172.21.15.87:0/3703690123 conn(0x55d91e45ecb0 msgr2=0x55d91e460100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- 172.21.15.87:0/3703690123 shutdown_connections 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- 172.21.15.87:0/3703690123 wait complete. 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- start start 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 Processor -- start 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464580 0x55d91e467800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464ea0 0x55d91e467d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.501+0000 7f9838c4d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f98301a9e70 con 0x55d91e46da80 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98301ab2e0 con 0x55d91e464580 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98301aa540 con 0x55d91e464ea0 2024-04-22T13:38:55.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 10 client.0 _ll_register_callbacks cb 0x7f981802c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49642/0 (socket says 172.21.15.87:49642) 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 -- 172.21.15.87:0/2981391710 learned_addr learned my addr 172.21.15.87:0/2981391710 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982f7fe640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464580 0x55d91e467800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9827fff640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464ea0 0x55d91e467d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 -- 172.21.15.87:0/2981391710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464ea0 msgr2=0x55d91e467d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464ea0 0x55d91e467d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 -- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464580 msgr2=0x55d91e467800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464580 0x55d91e467800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 -- 172.21.15.87:0/2981391710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d91e4e6ee0 con 0x55d91e46da80 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f982ffff640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9828010e00 tx=0x7f9828004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b03f6a8165e460b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f980effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:55.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828023070 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f982801f620 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98200a5050 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f982802f050 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 10 client.17520 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.502+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d91e4e81a0 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98200a5050 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f98280278a0 con 0x55d91e46da80 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 10 client.17520 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 20 client.17520 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e4e8920 0x55d91e4ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 4 client.17520 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d91e3d0a70 con 0x55d91e4e8920 2024-04-22T13:38:55.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f982f7fe640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e4e8920 0x55d91e4ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f980effd640 10 client.17520 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f982f7fe640 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e4e8920 0x55d91e4ead40 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc31f2c875d796e server_cookie=bd5e159ce24465d1 in_seq=0 out_seq=0 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f980effd640 1 -- 172.21.15.87:0/2981391710 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d91e3d0a70 con 0x55d91e4e8920 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f980effd640 10 client.17520 handle_command_reply: tid=0 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.503+0000 7f9838c4d740 1 client.17520 shutdown 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 msgr2=0x55d91e468280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9828010e00 tx=0x7f9828004a40 comp rx=0 tx=0).stop 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 shutdown_connections 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d91e4e8920 0x55d91e4ead40 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d91e46da80 0x55d91e468280 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d91e464ea0 0x55d91e467d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f980effd640 0 client.17520 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:55.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 --2- 172.21.15.87:0/2981391710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d91e464580 0x55d91e467800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 >> 172.21.15.87:0/2981391710 conn(0x55d91e45ecb0 msgr2=0x55d91e460990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 shutdown_connections 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 1 -- 172.21.15.87:0/2981391710 wait complete. 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f9838c4d740 20 client.17520 trim_cache size 0 max 16384 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:38:55.508 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 msgr2=0x7f981c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 0x7f981c0a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9828005cc0 tx=0x7f982800e720 comp rx=0 tx=0).stop 2024-04-22T13:38:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 msgr2=0x7f98301986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f982000d950 tx=0x7f982000de10 comp rx=0 tx=0).stop 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 shutdown_connections 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f981c0a3d20 0x7f981c0a61e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983019cfa0 0x7f983019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9830107280 0x7f9830198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 --2- 172.21.15.87:0/3307105499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98301021b0 0x7f98301986e0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 >> 172.21.15.87:0/3307105499 conn(0x7f98300fda10 msgr2=0x7f98300ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 shutdown_connections 2024-04-22T13:38:55.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.504+0000 7f983605c640 1 -- 172.21.15.87:0/3307105499 wait complete. 2024-04-22T13:38:55.511 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:38:55.511 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 -- 172.21.15.87:0/367938265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 msgr2=0x7f08a010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 --2- 172.21.15.87:0/367938265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a010ee30 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f0898009ca0 tx=0x7f089802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 -- 172.21.15.87:0/367938265 shutdown_connections 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 --2- 172.21.15.87:0/367938265 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a010ee30 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 --2- 172.21.15.87:0/367938265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102b40 0x7f08a0106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 --2- 172.21.15.87:0/367938265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a0102190 0x7f08a0102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 -- 172.21.15.87:0/367938265 >> 172.21.15.87:0/367938265 conn(0x7f08a00fd9f0 msgr2=0x7f08a00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 -- 172.21.15.87:0/367938265 shutdown_connections 2024-04-22T13:38:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 -- 172.21.15.87:0/367938265 wait complete. 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.614+0000 7f08a865c640 1 Processor -- start 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 -- start start 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 0x7f08a01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 0x7f08a019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f08a01121b0 con 0x7f08a0107260 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08a0112030 con 0x7f08a019cf90 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a865c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f08a0112330 con 0x7f08a0102190 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49656/0 (socket says 172.21.15.87:49656) 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 -- 172.21.15.87:0/1540074981 learned_addr learned my addr 172.21.15.87:0/1540074981 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a63f8640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 0x7f08a01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a6bf9640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 0x7f08a019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 -- 172.21.15.87:0/1540074981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 msgr2=0x7f08a01986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 0x7f08a01986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 -- 172.21.15.87:0/1540074981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 msgr2=0x7f08a019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 0x7f08a019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 -- 172.21.15.87:0/1540074981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a019da80 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a63f8640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 0x7f08a01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a6bf9640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 0x7f08a019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08a5bf7640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f089c00d770 tx=0x7f089c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c017070 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.615+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f089c009c30 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.616+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a019dd70 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.616+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c00f620 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.616+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a01a5560 con 0x7f08a0107260 2024-04-22T13:38:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.616+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a0199a20 con 0x7f08a0107260 2024-04-22T13:38:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.618+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f089c01a070 con 0x7f08a0107260 2024-04-22T13:38:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.618+0000 7f08937fe640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 0x7f088c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.618+0000 7f08a63f8640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 0x7f088c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.618+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f089c0d02e0 con 0x7f08a0107260 2024-04-22T13:38:55.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.619+0000 7f08a63f8640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 0x7f088c0a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08940058c0 tx=0x7f0894005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.622+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f089c009de0 con 0x7f08a0107260 2024-04-22T13:38:55.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.837+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a0102b40 con 0x7f08a0107260 2024-04-22T13:38:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.838+0000 7f08937fe640 1 -- 172.21.15.87:0/1540074981 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f089c09c9c0 con 0x7f08a0107260 2024-04-22T13:38:55.834 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:38:55.834 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:38:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 msgr2=0x7f088c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 0x7f088c0a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08940058c0 tx=0x7f0894005850 comp rx=0 tx=0).stop 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 msgr2=0x7f08a0198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f089c00d770 tx=0x7f089c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 shutdown_connections 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f088c0a3d20 0x7f088c0a61e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a019cf90 0x7f08a019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08a0107260 0x7f08a0198c10 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 --2- 172.21.15.87:0/1540074981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08a0102190 0x7f08a01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 >> 172.21.15.87:0/1540074981 conn(0x7f08a00fd9f0 msgr2=0x7f08a00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 shutdown_connections 2024-04-22T13:38:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.840+0000 7f08a865c640 1 -- 172.21.15.87:0/1540074981 wait complete. 2024-04-22T13:38:55.847 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 -- 172.21.15.87:0/2401317996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 msgr2=0x7fb36810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 --2- 172.21.15.87:0/2401317996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36810e9b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb360009900 tx=0x7fb36002bd60 comp rx=0 tx=0).stop 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 -- 172.21.15.87:0/2401317996 shutdown_connections 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 --2- 172.21.15.87:0/2401317996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36810e9b0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 --2- 172.21.15.87:0/2401317996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb368103af0 0x7fb36810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 --2- 172.21.15.87:0/2401317996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3681031d0 0x7fb3681035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.952+0000 7fb36d975640 1 -- 172.21.15.87:0/2401317996 >> 172.21.15.87:0/2401317996 conn(0x7fb368077f50 msgr2=0x7fb368078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- 172.21.15.87:0/2401317996 shutdown_connections 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- 172.21.15.87:0/2401317996 wait complete. 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 Processor -- start 2024-04-22T13:38:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- start start 2024-04-22T13:38:55.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3681031d0 0x7fb3681986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 0x7fb368198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681121a0 con 0x7fb36810c510 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb368112020 con 0x7fb368103af0 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.953+0000 7fb36d975640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb368112320 con 0x7fb3681031d0 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49686/0 (socket says 172.21.15.87:49686) 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 -- 172.21.15.87:0/598599705 learned_addr learned my addr 172.21.15.87:0/598599705 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb366ffd640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3681031d0 0x7fb3681986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3667fc640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 0x7fb368198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 -- 172.21.15.87:0/598599705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3681031d0 msgr2=0x7fb3681986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3681031d0 0x7fb3681986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 -- 172.21.15.87:0/598599705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 msgr2=0x7fb368198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 0x7fb368198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 -- 172.21.15.87:0/598599705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36819d6d0 con 0x7fb36810c510 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3667fc640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 0x7fb368198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb3677fe640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb360009230 tx=0x7fb36002c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb36c973640 1 -- 172.21.15.87:0/598599705 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360043020 con 0x7fb36810c510 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb36c973640 1 -- 172.21.15.87:0/598599705 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb36002dc30 con 0x7fb36810c510 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36819d960 con 0x7fb36810c510 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb36c973640 1 -- 172.21.15.87:0/598599705 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360003970 con 0x7fb36810c510 2024-04-22T13:38:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.954+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3681a5250 con 0x7fb36810c510 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.956+0000 7fb36c973640 1 -- 172.21.15.87:0/598599705 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb360046020 con 0x7fb36810c510 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 Processor -- start 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 -- start start 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d25430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d27e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb36c973640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 0x7fb3500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5654d0c811f0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654d0c1e4b0 con 0x5654d0d1ce80 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3701fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5654d0c7d1b0 con 0x5654d0d25a00 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49690/0 (socket says 172.21.15.87:49690) 2024-04-22T13:38:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 -- 172.21.15.87:0/2990233907 learned_addr learned my addr 172.21.15.87:0/2990233907 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3677fe640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 0x7fb3500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb36c973640 1 -- 172.21.15.87:0/598599705 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fb3600f2860 con 0x7fb36810c510 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb366ffd640 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d25430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 -- 172.21.15.87:0/2990233907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 msgr2=0x5654d0d27e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d27e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 -- 172.21.15.87:0/2990233907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 msgr2=0x5654d0d25430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d25430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 -- 172.21.15.87:0/2990233907 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d0c101b0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009cf0 tx=0x7fb35c0137d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab678d578978ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.957+0000 7fb3677fe640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 0x7fb3500a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb3600f05f0 tx=0x7fb3600c1040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb34dffb640 1 -- 172.21.15.87:0/2990233907 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c013070 con 0x5654d0d1c4d0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb34dffb640 1 -- 172.21.15.87:0/2990233907 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c017030 con 0x5654d0d1c4d0 2024-04-22T13:38:55.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb34dffb640 1 -- 172.21.15.87:0/2990233907 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c010630 con 0x5654d0d1c4d0 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 -- 172.21.15.87:0/2990233907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 msgr2=0x5654d0d1c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009cf0 tx=0x7fb35c0137d0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 -- 172.21.15.87:0/2990233907 shutdown_connections 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d27e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d25430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2990233907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d1c8b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 -- 172.21.15.87:0/2990233907 >> 172.21.15.87:0/2990233907 conn(0x5654d0d16ba0 msgr2=0x5654d0d16f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 -- 172.21.15.87:0/2990233907 shutdown_connections 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.958+0000 7fb3701fd740 1 -- 172.21.15.87:0/2990233907 wait complete. 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 -- start start 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 Processor -- start 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d22bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5654d0d28970 con 0x5654d0d1c4d0 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654d0c3a500 con 0x5654d0d1ce80 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5654d0d28af0 con 0x5654d0d25a00 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3701fd740 10 client.0 _ll_register_callbacks cb 0x5654d0c08b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49698/0 (socket says 172.21.15.87:49698) 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 -- 172.21.15.87:0/2843937385 learned_addr learned my addr 172.21.15.87:0/2843937385 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb366ffd640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 -- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 msgr2=0x5654d0d22bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d22bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 -- 172.21.15.87:0/2843937385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 msgr2=0x5654d0d22650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d22650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.959+0000 7fb3667fc640 1 -- 172.21.15.87:0/2843937385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d09933f0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3667fc640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009a90 tx=0x7fb35c009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175d6ac58ea44735 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0046c0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fb35c004860 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb35c017030 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb35c004a60 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 10 client.17538 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5654d0a5d7d0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c017030 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb35c025020 con 0x5654d0d1c4d0 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 10 client.17538 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 10 client.17538 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:38:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 20 client.17538 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d35b40 0x5654d0d37f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 4 client.17538 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.960+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654d0c20930 con 0x5654d0d35b40 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.961+0000 7fb366ffd640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d35b40 0x5654d0d37f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.961+0000 7fb346ffd640 10 client.17538 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.961+0000 7fb366ffd640 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d35b40 0x5654d0d37f30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67c7273d68a782f9 server_cookie=e2d2e565fc5d3c49 in_seq=0 out_seq=0 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb346ffd640 1 -- 172.21.15.87:0/2843937385 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654d0c20930 con 0x5654d0d35b40 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb346ffd640 10 client.17538 handle_command_reply: tid=0 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 client.17538 shutdown 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 msgr2=0x5654d0d252c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009a90 tx=0x7fb35c009ac0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 shutdown_connections 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d35b40 0x5654d0d37f30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d22bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d22650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb346ffd640 0 client.17538 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 --2- 172.21.15.87:0/2843937385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d252c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 >> 172.21.15.87:0/2843937385 conn(0x5654d0d16ba0 msgr2=0x5654d0d188f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 shutdown_connections 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 1 -- 172.21.15.87:0/2843937385 wait complete. 2024-04-22T13:38:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.962+0000 7fb3701fd740 20 client.17538 trim_cache size 0 max 16384 2024-04-22T13:38:55.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 Processor -- start 2024-04-22T13:38:55.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 -- start start 2024-04-22T13:38:55.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1c4d0 0x5654d0d31860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d1ce80 0x5654d0d15a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681b03c0 con 0x5654d0d25a00 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681adc90 con 0x5654d0d1c4d0 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3701fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681abb00 con 0x5654d0d1ce80 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.975+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49704/0 (socket says 172.21.15.87:49704) 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 -- 172.21.15.87:0/1683126200 learned_addr learned my addr 172.21.15.87:0/1683126200 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb366ffd640 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d1ce80 0x5654d0d15a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 -- 172.21.15.87:0/1683126200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d1ce80 msgr2=0x5654d0d15a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d1ce80 0x5654d0d15a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 -- 172.21.15.87:0/1683126200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1c4d0 msgr2=0x5654d0d31860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1c4d0 0x5654d0d31860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 -- 172.21.15.87:0/1683126200 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654d0d28850 con 0x5654d0d25a00 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb3667fc640 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00fc10 tx=0x7fb35c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b7924aa0f027e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb34d7fa640 1 -- 172.21.15.87:0/1683126200 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c009890 con 0x5654d0d25a00 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb34d7fa640 1 -- 172.21.15.87:0/1683126200 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c017040 con 0x5654d0d25a00 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.976+0000 7fb34d7fa640 1 -- 172.21.15.87:0/1683126200 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c004660 con 0x5654d0d25a00 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- 172.21.15.87:0/1683126200 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 msgr2=0x5654d0d24d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00fc10 tx=0x7fb35c009ca0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- 172.21.15.87:0/1683126200 shutdown_connections 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d25a00 0x5654d0d24d80 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d1ce80 0x5654d0d15a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- 172.21.15.87:0/1683126200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1c4d0 0x5654d0d31860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- 172.21.15.87:0/1683126200 >> 172.21.15.87:0/1683126200 conn(0x5654d0d16ba0 msgr2=0x5654d0d15370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- 172.21.15.87:0/1683126200 shutdown_connections 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- 172.21.15.87:0/1683126200 wait complete. 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- start start 2024-04-22T13:38:55.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 Processor -- start 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d29460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d99f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681ac150 con 0x5654d0d1c4d0 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681ad5c0 con 0x5654d0d1ce80 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681ac820 con 0x5654d0d25a00 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3701fd740 10 client.0 _ll_register_callbacks cb 0x7fb35c0053e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3667fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49716/0 (socket says 172.21.15.87:49716) 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.977+0000 7fb3667fc640 1 -- 172.21.15.87:0/814080923 learned_addr learned my addr 172.21.15.87:0/814080923 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d99f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb366ffd640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d29460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 -- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 msgr2=0x5654d0d99f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d99f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 -- 172.21.15.87:0/814080923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 msgr2=0x5654d0d29460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d29460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 -- 172.21.15.87:0/814080923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654d0d9a440 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3667fc640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009730 tx=0x7fb35c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef266f0fcbf51212 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c004660 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fb35c004800 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb35c017040 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb35c00f440 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3701fd740 10 client.17544 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5654d0d9b6f0 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb35c017040 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.978+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb35c025070 con 0x5654d0d1c4d0 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 10 client.17544 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 10 client.17544 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 20 client.17544 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d9bd00 0x5654d0d9e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 4 client.17544 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654d0c8ec10 con 0x5654d0d9bd00 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb366ffd640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d9bd00 0x5654d0d9e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb346ffd640 10 client.17544 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb366ffd640 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d9bd00 0x5654d0d9e0f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d3c494becefc5ab server_cookie=f224d9f9da35bac8 in_seq=0 out_seq=0 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb346ffd640 1 -- 172.21.15.87:0/814080923 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654d0c8ec10 con 0x5654d0d9bd00 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb346ffd640 10 client.17544 handle_command_reply: tid=0 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 1 client.17544 shutdown 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 msgr2=0x5654d0d28f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.979+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009730 tx=0x7fb35c000bc0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 shutdown_connections 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5654d0d9bd00 0x5654d0d9e0f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654d0d25a00 0x5654d0d99f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654d0d1ce80 0x5654d0d29460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb346ffd640 0 client.17544 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 --2- 172.21.15.87:0/814080923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654d0d1c4d0 0x5654d0d28f20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 >> 172.21.15.87:0/814080923 conn(0x5654d0d16ba0 msgr2=0x5654d0d15c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 shutdown_connections 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 1 -- 172.21.15.87:0/814080923 wait complete. 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb3701fd740 20 client.17544 trim_cache size 0 max 16384 2024-04-22T13:38:55.982 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 msgr2=0x7fb3500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 0x7fb3500a61e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb3600f05f0 tx=0x7fb3600c1040 comp rx=0 tx=0).stop 2024-04-22T13:38:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 msgr2=0x7fb36819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb360009230 tx=0x7fb36002c840 comp rx=0 tx=0).stop 2024-04-22T13:38:55.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 shutdown_connections 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb3500a3d20 0x7fb3500a61e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb36810c510 0x7fb36819cfa0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb368103af0 0x7fb368198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 --2- 172.21.15.87:0/598599705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3681031d0 0x7fb3681986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 >> 172.21.15.87:0/598599705 conn(0x7fb368077f50 msgr2=0x7fb3680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 shutdown_connections 2024-04-22T13:38:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:55.980+0000 7fb36d975640 1 -- 172.21.15.87:0/598599705 wait complete. 2024-04-22T13:38:55.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:38:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 -- 172.21.15.87:0/2770614615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0824101a50 msgr2=0x7f0824105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 --2- 172.21.15.87:0/2770614615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0824101a50 0x7f0824105a90 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0814009900 tx=0x7f081402bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 -- 172.21.15.87:0/2770614615 shutdown_connections 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 --2- 172.21.15.87:0/2770614615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0824106170 0x7f082410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 --2- 172.21.15.87:0/2770614615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0824101a50 0x7f0824105a90 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 --2- 172.21.15.87:0/2770614615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08241010a0 0x7f0824101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 -- 172.21.15.87:0/2770614615 >> 172.21.15.87:0/2770614615 conn(0x7f0824077eb0 msgr2=0x7f08240782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 -- 172.21.15.87:0/2770614615 shutdown_connections 2024-04-22T13:38:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 -- 172.21.15.87:0/2770614615 wait complete. 2024-04-22T13:38:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.091+0000 7f0829248640 1 Processor -- start 2024-04-22T13:38:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 -- start start 2024-04-22T13:38:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0824106170 0x7f0824199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f082419a0d0 0x7f08241a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f08241110b0 con 0x7f08241010a0 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0824110f30 con 0x7f082419a0d0 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0829248640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0824111230 con 0x7f0824106170 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49742/0 (socket says 172.21.15.87:49742) 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 -- 172.21.15.87:0/2091915315 learned_addr learned my addr 172.21.15.87:0/2091915315 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f082259c640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0824106170 0x7f0824199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 -- 172.21.15.87:0/2091915315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0824106170 msgr2=0x7f0824199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f082359e640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f082419a0d0 0x7f08241a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0824106170 0x7f0824199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 -- 172.21.15.87:0/2091915315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f082419a0d0 msgr2=0x7f08241a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f082419a0d0 0x7f08241a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 -- 172.21.15.87:0/2091915315 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08241a4e80 con 0x7f08241010a0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.092+0000 7f080bfff640 1 -- 172.21.15.87:0/2091915315 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081800cd80 con 0x7f08241010a0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.093+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08241a5170 con 0x7f08241010a0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.093+0000 7f080bfff640 1 -- 172.21.15.87:0/2091915315 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08180054d0 con 0x7f08241010a0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.093+0000 7f080bfff640 1 -- 172.21.15.87:0/2091915315 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818010690 con 0x7f08241010a0 2024-04-22T13:38:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.093+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08241a5690 con 0x7f08241010a0 2024-04-22T13:38:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f080bfff640 1 -- 172.21.15.87:0/2091915315 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0818004e90 con 0x7f08241010a0 2024-04-22T13:38:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 Processor -- start 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 -- start start 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915d4d0 0x559c7915d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c79168e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559c790c21f0 con 0x559c7915de80 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559c7905f4b0 con 0x559c79166a00 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f082be31740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559c790be1b0 con 0x559c7915d4d0 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f080bfff640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 0x7f080c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49756/0 (socket says 172.21.15.87:49756) 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.095+0000 7f0822d9d640 1 -- 172.21.15.87:0/3010915783 learned_addr learned my addr 172.21.15.87:0/3010915783 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f082259c640 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c79168e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f082359e640 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915d4d0 0x559c7915d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f082359e640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 0x7f080c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f080bfff640 1 -- 172.21.15.87:0/2091915315 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f08180d0d20 con 0x7f08241010a0 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 -- 172.21.15.87:0/3010915783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915d4d0 msgr2=0x559c7915d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915d4d0 0x559c7915d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 -- 172.21.15.87:0/3010915783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 msgr2=0x559c79168e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c79168e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 -- 172.21.15.87:0/3010915783 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c790511b0 con 0x559c7915de80 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f081809ca00 tx=0x7f0818004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a6c457c0af2bff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f082359e640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 0x7f080c0a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fd0 tx=0x7f081c004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f08097fa640 1 -- 172.21.15.87:0/3010915783 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081809d400 con 0x559c7915de80 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f08097fa640 1 -- 172.21.15.87:0/3010915783 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081809cd00 con 0x559c7915de80 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.096+0000 7f08097fa640 1 -- 172.21.15.87:0/3010915783 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818016d90 con 0x559c7915de80 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- 172.21.15.87:0/3010915783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 msgr2=0x559c79166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f081809ca00 tx=0x7f0818004a90 comp rx=0 tx=0).stop 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- 172.21.15.87:0/3010915783 shutdown_connections 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c79168e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915de80 0x559c79166430 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- 172.21.15.87:0/3010915783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915d4d0 0x559c7915d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- 172.21.15.87:0/3010915783 >> 172.21.15.87:0/3010915783 conn(0x559c79157ba0 msgr2=0x559c79157f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- 172.21.15.87:0/3010915783 shutdown_connections 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- 172.21.15.87:0/3010915783 wait complete. 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- start start 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 Processor -- start 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79163680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c79163c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.097+0000 7f082be31740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559c791698b0 con 0x559c7915d4d0 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559c7907b500 con 0x559c7915de80 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559c79169a30 con 0x559c79166a00 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 10 client.0 _ll_register_callbacks cb 0x559c79049b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49770/0 (socket says 172.21.15.87:49770) 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 -- 172.21.15.87:0/2681266229 learned_addr learned my addr 172.21.15.87:0/2681266229 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082359e640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c79163c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082259c640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79163680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 -- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 msgr2=0x559c79163c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c79163c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 -- 172.21.15.87:0/2681266229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 msgr2=0x559c79163680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79163680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 -- 172.21.15.87:0/2681266229 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c78dd43f0 con 0x559c7915d4d0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082259c640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79163680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f08180996d0 tx=0x7f08180990b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbc728ea71ac817 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0806ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081809b000 con 0x559c7915d4d0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f08180d2030 con 0x559c7915d4d0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081809cd00 con 0x559c7915d4d0 2024-04-22T13:38:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f081809d400 con 0x559c7915d4d0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 10 client.17556 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559c78e9e7d0 con 0x559c7915d4d0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.098+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081809cd00 con 0x559c7915d4d0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f08180a5570 con 0x559c7915d4d0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 10 client.17556 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 10 client.17556 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 20 client.17556 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791769c0 0x559c79178db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 4 client.17556 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c79061930 con 0x559c791769c0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.099+0000 7f082259c640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791769c0 0x559c79178db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082259c640 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791769c0 0x559c79178db0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a6cc1cd6a9c4d1c9 server_cookie=8b396d1a217031de in_seq=0 out_seq=0 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f0806ff5640 10 client.17556 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f0806ff5640 1 -- 172.21.15.87:0/2681266229 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c79061930 con 0x559c791769c0 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f0806ff5640 10 client.17556 handle_command_reply: tid=0 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 client.17556 shutdown 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 msgr2=0x559c791662f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f08180996d0 tx=0x7f08180990b0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 shutdown_connections 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791769c0 0x559c79178db0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c79163c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f0806ff5640 0 client.17556 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79163680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 --2- 172.21.15.87:0/2681266229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791662f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 >> 172.21.15.87:0/2681266229 conn(0x559c79157ba0 msgr2=0x559c791598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 shutdown_connections 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.100+0000 7f082be31740 1 -- 172.21.15.87:0/2681266229 wait complete. 2024-04-22T13:38:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.101+0000 7f082be31740 20 client.17556 trim_cache size 0 max 16384 2024-04-22T13:38:56.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.113+0000 7f082be31740 1 Processor -- start 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.113+0000 7f082be31740 1 -- start start 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79165db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c791662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f08241b03c0 con 0x559c7915d4d0 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08241adc90 con 0x559c7915de80 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082be31740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f08241abb00 con 0x559c79166a00 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49780/0 (socket says 172.21.15.87:49780) 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 -- 172.21.15.87:0/2835912816 learned_addr learned my addr 172.21.15.87:0/2835912816 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f082359e640 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c791662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 -- 172.21.15.87:0/2835912816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 msgr2=0x559c791662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c791662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 -- 172.21.15.87:0/2835912816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 msgr2=0x559c79165db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79165db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 -- 172.21.15.87:0/2835912816 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c7915d0b0 con 0x559c7915d4d0 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0822d9d640 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f081809ca00 tx=0x7f08180052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9043ca49641e7545 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0808ff9640 1 -- 172.21.15.87:0/2835912816 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081809cad0 con 0x559c7915d4d0 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0808ff9640 1 -- 172.21.15.87:0/2835912816 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08180d2070 con 0x559c7915d4d0 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.114+0000 7f0808ff9640 1 -- 172.21.15.87:0/2835912816 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08180a5970 con 0x559c7915d4d0 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- 172.21.15.87:0/2835912816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 msgr2=0x559c7917c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f081809ca00 tx=0x7f08180052b0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- 172.21.15.87:0/2835912816 shutdown_connections 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c79166a00 0x559c791662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c7915de80 0x559c79165db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- 172.21.15.87:0/2835912816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c7917c1c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- 172.21.15.87:0/2835912816 >> 172.21.15.87:0/2835912816 conn(0x559c79157ba0 msgr2=0x559c79163450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- 172.21.15.87:0/2835912816 shutdown_connections 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- 172.21.15.87:0/2835912816 wait complete. 2024-04-22T13:38:56.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- start start 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 Processor -- start 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915de80 0x559c791d99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c791606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f08241ac150 con 0x559c7915d4d0 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08241ad5c0 con 0x559c79166a00 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.115+0000 7f082be31740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f08241ac820 con 0x559c7915de80 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 10 client.0 _ll_register_callbacks cb 0x7f0818009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49782/0 (socket says 172.21.15.87:49782) 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 -- 172.21.15.87:0/3257887421 learned_addr learned my addr 172.21.15.87:0/3257887421 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082259c640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915de80 0x559c791d99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 -- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915de80 msgr2=0x559c791d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915de80 0x559c791d99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 -- 172.21.15.87:0/3257887421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 msgr2=0x559c791606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c791606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 -- 172.21.15.87:0/3257887421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c79160de0 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0822d9d640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f08180b8040 tx=0x7f0818098980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a02c23d4e6ff92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081809cad0 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f08180b6040 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08180d2070 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0818005680 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 10 client.17562 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559c79160de0 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08180d2070 con 0x559c7915d4d0 2024-04-22T13:38:56.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f081809cf60 con 0x559c7915d4d0 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 10 client.17562 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.116+0000 7f082be31740 10 client.17562 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 20 client.17562 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791e0f40 0x559c791e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 4 client.17562 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c790cfc10 con 0x559c791e0f40 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082259c640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791e0f40 0x559c791e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082259c640 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791e0f40 0x559c791e3330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=94a131e4f322ff32 server_cookie=82fb950419fc710e in_seq=0 out_seq=0 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f0806ff5640 10 client.17562 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f0806ff5640 1 -- 172.21.15.87:0/3257887421 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c790cfc10 con 0x559c791e0f40 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f0806ff5640 10 client.17562 handle_command_reply: tid=0 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 client.17562 shutdown 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 msgr2=0x559c791d9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f08180b8040 tx=0x7f0818098980 comp rx=0 tx=0).stop 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 shutdown_connections 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x559c791e0f40 0x559c791e3330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559c79166a00 0x559c791606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f0806ff5640 0 client.17562 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c7915de80 0x559c791d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 --2- 172.21.15.87:0/3257887421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559c7915d4d0 0x559c791d9470 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.117+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 >> 172.21.15.87:0/3257887421 conn(0x559c79157ba0 msgr2=0x559c79158fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 shutdown_connections 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f082be31740 1 -- 172.21.15.87:0/3257887421 wait complete. 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f082be31740 20 client.17562 trim_cache size 0 max 16384 2024-04-22T13:38:56.120 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 msgr2=0x7f080c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 0x7f080c0a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fd0 tx=0x7f081c004a40 comp rx=0 tx=0).stop 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 msgr2=0x7f0824199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800edf0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 shutdown_connections 2024-04-22T13:38:56.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f080c0a3d20 0x7f080c0a61e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f082419a0d0 0x7f08241a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0824106170 0x7f0824199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 --2- 172.21.15.87:0/2091915315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f08241010a0 0x7f0824199600 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 >> 172.21.15.87:0/2091915315 conn(0x7f0824077eb0 msgr2=0x7f08240ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 shutdown_connections 2024-04-22T13:38:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.118+0000 7f0829248640 1 -- 172.21.15.87:0/2091915315 wait complete. 2024-04-22T13:38:56.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1540074981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:56.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:38:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1540074981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:56.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1540074981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 -- 172.21.15.87:0/770494369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4100f50 msgr2=0x7f1ab4101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 --2- 172.21.15.87:0/770494369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4100f50 0x7f1ab4101330 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009900 tx=0x7f1ab002bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 -- 172.21.15.87:0/770494369 shutdown_connections 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 --2- 172.21.15.87:0/770494369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4106020 0x7f1ab410dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 --2- 172.21.15.87:0/770494369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4101900 0x7f1ab4105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 --2- 172.21.15.87:0/770494369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4100f50 0x7f1ab4101330 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 -- 172.21.15.87:0/770494369 >> 172.21.15.87:0/770494369 conn(0x7f1ab40fc7d0 msgr2=0x7f1ab40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 -- 172.21.15.87:0/770494369 shutdown_connections 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.230+0000 7f1abb39f640 1 -- 172.21.15.87:0/770494369 wait complete. 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 Processor -- start 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 -- start start 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4100f50 0x7f1ab4199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4199f10 0x7f1ab41a4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab4110f70 con 0x7f1ab4106020 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab4110df0 con 0x7f1ab4100f50 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abb39f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41110f0 con 0x7f1ab4199f10 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49800/0 (socket says 172.21.15.87:49800) 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/2967072082 learned_addr learned my addr 172.21.15.87:0/2967072082 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1aba39d640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4100f50 0x7f1ab4199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1abab9e640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4199f10 0x7f1ab41a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/2967072082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4199f10 msgr2=0x7f1ab41a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4199f10 0x7f1ab41a4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/2967072082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4100f50 msgr2=0x7f1ab4199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4100f50 0x7f1ab4199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/2967072082 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab41a4c90 con 0x7f1ab4106020 2024-04-22T13:38:56.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.231+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c970 tx=0x7f1aa800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.232+0000 7f1aa77fe640 1 -- 172.21.15.87:0/2967072082 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8014070 con 0x7f1ab4106020 2024-04-22T13:38:56.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.232+0000 7f1aa77fe640 1 -- 172.21.15.87:0/2967072082 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa8004440 con 0x7f1ab4106020 2024-04-22T13:38:56.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.232+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab41a4f80 con 0x7f1ab4106020 2024-04-22T13:38:56.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.232+0000 7f1aa77fe640 1 -- 172.21.15.87:0/2967072082 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8003e50 con 0x7f1ab4106020 2024-04-22T13:38:56.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.232+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab41a5440 con 0x7f1ab4106020 2024-04-22T13:38:56.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.234+0000 7f1aa77fe640 1 -- 172.21.15.87:0/2967072082 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1aa8017070 con 0x7f1ab4106020 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.234+0000 7f1abf1f1740 1 Processor -- start 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 -- start start 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1aa77fe640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 0x7f1aa00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x558055563990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55805556d060 0x55805556d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5580554c81f0 con 0x558055563f60 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580554654b0 con 0x5580555635b0 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5580554c41b0 con 0x55805556d060 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abab9e640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 0x7f1aa00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1aa77fe640 1 -- 172.21.15.87:0/2967072082 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f1aa80d0340 con 0x7f1ab4106020 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49802/0 (socket says 172.21.15.87:49802) 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/428590613 learned_addr learned my addr 172.21.15.87:0/428590613 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1aba39d640 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x558055563990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/428590613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55805556d060 msgr2=0x55805556d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55805556d060 0x55805556d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/428590613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 msgr2=0x558055563990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x558055563990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/428590613 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580554571b0 con 0x558055563f60 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1abab9e640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 0x7f1aa00a6380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0092f0 tx=0x7f1aac0097f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.235+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c940 tx=0x7f1aa809be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f83844ea8ba76e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1aa4bfe640 1 -- 172.21.15.87:0/428590613 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa809ca20 con 0x558055563f60 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1aa4bfe640 1 -- 172.21.15.87:0/428590613 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa809c320 con 0x558055563f60 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1aa4bfe640 1 -- 172.21.15.87:0/428590613 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8098990 con 0x558055563f60 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 -- 172.21.15.87:0/428590613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 msgr2=0x55805556ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c940 tx=0x7f1aa809be60 comp rx=0 tx=0).stop 2024-04-22T13:38:56.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 -- 172.21.15.87:0/428590613 shutdown_connections 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55805556d060 0x55805556d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x55805556ca90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/428590613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x558055563990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 -- 172.21.15.87:0/428590613 >> 172.21.15.87:0/428590613 conn(0x55805555dda0 msgr2=0x55805555e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 -- 172.21.15.87:0/428590613 shutdown_connections 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.236+0000 7f1abf1f1740 1 -- 172.21.15.87:0/428590613 wait complete. 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 -- start start 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 Processor -- start 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580555635b0 0x5580555734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055570d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558055573fa0 con 0x558055563f60 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558055481500 con 0x55805556d060 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558055574120 con 0x5580555635b0 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abf1f1740 10 client.0 _ll_register_callbacks cb 0x55805544fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49814/0 (socket says 172.21.15.87:49814) 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/1437909053 learned_addr learned my addr 172.21.15.87:0/1437909053 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abab9e640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055570d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1aba39d640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580555635b0 0x5580555734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/1437909053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580555635b0 msgr2=0x5580555734d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580555635b0 0x5580555734d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 msgr2=0x558055570d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055570d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1ab9b9c640 1 -- 172.21.15.87:0/1437909053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580551da3f0 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.237+0000 7f1abab9e640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055570d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80871a0 tx=0x7f1aa80a06d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d415fb9a8e29be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8087470 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f1aa809b770 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa809c320 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1aa80a4f90 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 10 client.17574 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5580552a47d0 con 0x558055563f60 2024-04-22T13:38:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa809c320 con 0x558055563f60 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1aa8098070 con 0x558055563f60 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 10 client.17574 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 20 client.17574 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558055581120 0x558055583510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 4 client.17574 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558055467930 con 0x558055581120 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.238+0000 7f1aba39d640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558055581120 0x558055583510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.239+0000 7f1a9dffb640 10 client.17574 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.239+0000 7f1aba39d640 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558055581120 0x558055583510 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559bd830feaeef93 server_cookie=6075cdea2a6deed4 in_seq=0 out_seq=0 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1a9dffb640 1 -- 172.21.15.87:0/1437909053 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558055467930 con 0x558055581120 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1a9dffb640 10 client.17574 handle_command_reply: tid=0 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 client.17574 shutdown 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 msgr2=0x5580555707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80871a0 tx=0x7f1aa80a06d0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 shutdown_connections 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558055581120 0x558055583510 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055570d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1a9dffb640 0 client.17574 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558055563f60 0x5580555707c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/1437909053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580555635b0 0x5580555734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 >> 172.21.15.87:0/1437909053 conn(0x55805555dda0 msgr2=0x55805555faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 shutdown_connections 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 1 -- 172.21.15.87:0/1437909053 wait complete. 2024-04-22T13:38:56.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.240+0000 7f1abf1f1740 20 client.17574 trim_cache size 0 max 16384 2024-04-22T13:38:56.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 Processor -- start 2024-04-22T13:38:56.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 -- start start 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x55805556ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x55805555cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41adf80 con 0x55805556d060 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41ab850 con 0x5580555635b0 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41a96c0 con 0x558055563f60 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.253+0000 7f1abab9e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49820/0 (socket says 172.21.15.87:49820) 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 -- 172.21.15.87:0/3652323376 learned_addr learned my addr 172.21.15.87:0/3652323376 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1aba39d640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x55805556ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x55805555cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 -- 172.21.15.87:0/3652323376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 msgr2=0x55805555cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x55805555cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 -- 172.21.15.87:0/3652323376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 msgr2=0x55805556ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x55805556ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 -- 172.21.15.87:0/3652323376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558055589350 con 0x55805556d060 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1aba39d640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x55805556ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1abab9e640 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f1aac013750 tx=0x7f1aac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f324a38e884cfe3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1a9ffff640 1 -- 172.21.15.87:0/3652323376 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00f8e0 con 0x55805556d060 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1a9ffff640 1 -- 172.21.15.87:0/3652323376 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aac017070 con 0x55805556d060 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.254+0000 7f1a9ffff640 1 -- 172.21.15.87:0/3652323376 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac020390 con 0x55805556d060 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- 172.21.15.87:0/3652323376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 msgr2=0x558055573000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f1aac013750 tx=0x7f1aac015000 comp rx=0 tx=0).stop 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- 172.21.15.87:0/3652323376 shutdown_connections 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55805556d060 0x558055573000 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x55805555cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/3652323376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580555635b0 0x55805556ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- 172.21.15.87:0/3652323376 >> 172.21.15.87:0/3652323376 conn(0x55805555dda0 msgr2=0x55805555c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- 172.21.15.87:0/3652323376 shutdown_connections 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- 172.21.15.87:0/3652323376 wait complete. 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- start start 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 Processor -- start 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x5580555844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055584a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41a9d10 con 0x5580555635b0 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41ab180 con 0x55805556d060 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab41aa3e0 con 0x558055563f60 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1abf1f1740 10 client.0 _ll_register_callbacks cb 0x7f1ab0005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1aba39d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1aba39d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49836/0 (socket says 172.21.15.87:49836) 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1aba39d640 1 -- 172.21.15.87:0/2562476999 learned_addr learned my addr 172.21.15.87:0/2562476999 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:56.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.255+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x5580555844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abab9e640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055584a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 -- 172.21.15.87:0/2562476999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 msgr2=0x5580555844c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x5580555844c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 -- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 msgr2=0x558055584a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055584a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 -- 172.21.15.87:0/2562476999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580555e6300 con 0x5580555635b0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x5580555844c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abab9e640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055584a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1aba39d640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f1ab002c170 tx=0x7f1ab0004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf73256fe39d0c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab002c7e0 con 0x5580555635b0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f1ab002c980 con 0x5580555635b0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aac017070 con 0x5580555635b0 2024-04-22T13:38:56.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1ab0058050 con 0x5580555635b0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 10 client.17580 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5580555e7600 con 0x5580555635b0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aac017070 con 0x5580555635b0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1ab004a500 con 0x5580555635b0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 10 client.17580 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 20 client.17580 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5580555e7dd0 0x5580555ea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 4 client.17580 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.256+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580554d5c10 con 0x5580555e7dd0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5580555e7dd0 0x5580555ea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1ab9b9c640 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5580555e7dd0 0x5580555ea1f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=360ecfa8348bfaf5 server_cookie=e1af2e5b615fb0ef in_seq=0 out_seq=0 2024-04-22T13:38:56.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1a9dffb640 10 client.17580 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1a9dffb640 1 -- 172.21.15.87:0/2562476999 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580554d5c10 con 0x5580555e7dd0 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1a9dffb640 10 client.17580 handle_command_reply: tid=0 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 client.17580 shutdown 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 msgr2=0x558055583f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f1ab002c170 tx=0x7f1ab0004550 comp rx=0 tx=0).stop 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 shutdown_connections 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5580555e7dd0 0x5580555ea1f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55805556d060 0x558055584a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558055563f60 0x5580555844c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1a9dffb640 0 client.17580 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 --2- 172.21.15.87:0/2562476999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5580555635b0 0x558055583f80 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 >> 172.21.15.87:0/2562476999 conn(0x55805555dda0 msgr2=0x55805555c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.257+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 shutdown_connections 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abf1f1740 1 -- 172.21.15.87:0/2562476999 wait complete. 2024-04-22T13:38:56.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abf1f1740 20 client.17580 trim_cache size 0 max 16384 2024-04-22T13:38:56.261 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:38:56.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 msgr2=0x7f1aa00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 0x7f1aa00a6380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0092f0 tx=0x7f1aac0097f0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 msgr2=0x7f1ab41999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c970 tx=0x7f1aa800ce30 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 shutdown_connections 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1aa00a3ec0 0x7f1aa00a6380 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ab4199f10 0x7f1ab41a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ab4106020 0x7f1ab41999d0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 --2- 172.21.15.87:0/2967072082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ab4100f50 0x7f1ab4199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 >> 172.21.15.87:0/2967072082 conn(0x7f1ab40fc7d0 msgr2=0x7f1ab40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 shutdown_connections 2024-04-22T13:38:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:56.258+0000 7f1abb39f640 1 -- 172.21.15.87:0/2967072082 wait complete. 2024-04-22T13:38:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:56 smithi120 ceph-mon[31101]: pgmap v218: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 46 MiB/s wr, 480 op/s 2024-04-22T13:38:57.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:56 smithi132 ceph-mon[31531]: pgmap v218: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 46 MiB/s wr, 480 op/s 2024-04-22T13:38:57.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:56 smithi087 ceph-mon[27999]: pgmap v218: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 46 MiB/s wr, 480 op/s 2024-04-22T13:38:57.266 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:38:57.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:38:57.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 -- 172.21.15.87:0/2587305308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc006a840 msgr2=0x7ffbc006ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 --2- 172.21.15.87:0/2587305308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc006a840 0x7ffbc006ac20 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009900 tx=0x7ffbb002bc30 comp rx=0 tx=0).stop 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 -- 172.21.15.87:0/2587305308 shutdown_connections 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 --2- 172.21.15.87:0/2587305308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc0107270 0x7ffbc010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 --2- 172.21.15.87:0/2587305308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc006b1f0 0x7ffbc0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 --2- 172.21.15.87:0/2587305308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc006a840 0x7ffbc006ac20 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 -- 172.21.15.87:0/2587305308 >> 172.21.15.87:0/2587305308 conn(0x7ffbc0077eb0 msgr2=0x7ffbc00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 -- 172.21.15.87:0/2587305308 shutdown_connections 2024-04-22T13:38:57.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 -- 172.21.15.87:0/2587305308 wait complete. 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.386+0000 7ffbc6043640 1 Processor -- start 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 -- start start 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc006a840 0x7ffbc0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc019cf50 0x7ffbc019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01121b0 con 0x7ffbc0107270 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0112030 con 0x7ffbc006a840 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbc6043640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc0112330 con 0x7ffbc019cf50 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49848/0 (socket says 172.21.15.87:49848) 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 -- 172.21.15.87:0/1720125420 learned_addr learned my addr 172.21.15.87:0/1720125420 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc006a840 0x7ffbc0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbffff640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc019cf50 0x7ffbc019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 -- 172.21.15.87:0/1720125420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc019cf50 msgr2=0x7ffbc019d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc019cf50 0x7ffbc019d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 -- 172.21.15.87:0/1720125420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc006a840 msgr2=0x7ffbc0198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc006a840 0x7ffbc0198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 -- 172.21.15.87:0/1720125420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc019da40 con 0x7ffbc0107270 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400c970 tx=0x7ffbb400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.387+0000 7ffbbcff9640 1 -- 172.21.15.87:0/1720125420 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4014070 con 0x7ffbc0107270 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.388+0000 7ffbbcff9640 1 -- 172.21.15.87:0/1720125420 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb4004440 con 0x7ffbc0107270 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.388+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc019dd30 con 0x7ffbc0107270 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.388+0000 7ffbbcff9640 1 -- 172.21.15.87:0/1720125420 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4003e50 con 0x7ffbc0107270 2024-04-22T13:38:57.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.388+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc01a5520 con 0x7ffbc0107270 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbbcff9640 1 -- 172.21.15.87:0/1720125420 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffbb4017070 con 0x7ffbc0107270 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 Processor -- start 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 -- start start 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbbcff9640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 0x7ffba80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf44611cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbbcff9640 1 -- 172.21.15.87:0/1720125420 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ffbb40d0350 con 0x7ffbc0107270 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf4461d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bf44575f00 con 0x55bf44612290 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bf44513590 con 0x55bf446118e0 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bf44571ec0 con 0x55bf4461ae10 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.390+0000 7ffbbffff640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 0x7ffba80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf44611cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49864/0 (socket says 172.21.15.87:49864) 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 -- 172.21.15.87:0/2168052570 learned_addr learned my addr 172.21.15.87:0/2168052570 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 -- 172.21.15.87:0/2168052570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 msgr2=0x55bf4461d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf4461d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 -- 172.21.15.87:0/2168052570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 msgr2=0x55bf44611cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf44611cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 -- 172.21.15.87:0/2168052570 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf443507e0 con 0x55bf44612290 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbffff640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 0x7ffba80a6380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80092f0 tx=0x7ffbb8010740 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4098d00 tx=0x7ffbb409ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0f53d63e56658f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffba5ffb640 1 -- 172.21.15.87:0/2168052570 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40b5070 con 0x55bf44612290 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffba5ffb640 1 -- 172.21.15.87:0/2168052570 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb40a4fa0 con 0x55bf44612290 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.391+0000 7ffba5ffb640 1 -- 172.21.15.87:0/2168052570 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40a0170 con 0x55bf44612290 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2168052570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 msgr2=0x55bf4461a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4098d00 tx=0x7ffbb409ca30 comp rx=0 tx=0).stop 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2168052570 shutdown_connections 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf4461d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf44612290 0x55bf4461a840 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2168052570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf44611cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2168052570 >> 172.21.15.87:0/2168052570 conn(0x55bf4460bfb0 msgr2=0x55bf4460c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2168052570 shutdown_connections 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2168052570 wait complete. 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 -- start start 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.392+0000 7ffbc8c38740 1 Processor -- start 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf44612290 0x55bf446183c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf44618900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bf4461dd80 con 0x55bf446118e0 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bf4452f580 con 0x55bf44612290 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bf4461df00 con 0x55bf4461ae10 2024-04-22T13:38:57.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 10 client.0 _ll_register_callbacks cb 0x55bf44535650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49866/0 (socket says 172.21.15.87:49866) 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 -- 172.21.15.87:0/385278305 learned_addr learned my addr 172.21.15.87:0/385278305 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf44612290 0x55bf446183c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbffff640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf44618900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 -- 172.21.15.87:0/385278305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 msgr2=0x55bf44618900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf44618900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 -- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf44612290 msgr2=0x55bf446183c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf44612290 0x55bf446183c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 -- 172.21.15.87:0/385278305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf443b0f70 con 0x55bf446118e0 2024-04-22T13:38:57.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009900 tx=0x7ffbb002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319ba7ca7ddbd638 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb002cc40 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ffbb0005250 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.393+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb40a4fa0 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffbb0005450 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 10 client.17592 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bf44248170 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb40a4fa0 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ffbb0003820 con 0x55bf446118e0 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 10 client.17592 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 10 client.17592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 20 client.17592 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf4462b340 0x55bf4462d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 4 client.17592 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf44583920 con 0x55bf4462b340 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.394+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf4462b340 0x55bf4462d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.395+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf4462b340 0x55bf4462d7c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf1cecfd0bca3a1 server_cookie=4229e1ca0ea6ecdb in_seq=0 out_seq=0 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.395+0000 7ffb9effd640 10 client.17592 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffb9effd640 1 -- 172.21.15.87:0/385278305 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf44583920 con 0x55bf4462b340 2024-04-22T13:38:57.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffb9effd640 10 client.17592 handle_command_reply: tid=0 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 client.17592 shutdown 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 msgr2=0x55bf44617e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009900 tx=0x7ffbb002ca10 comp rx=0 tx=0).stop 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 shutdown_connections 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf4462b340 0x55bf4462d7c0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf4461ae10 0x55bf44618900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf44612290 0x55bf446183c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffb9effd640 0 client.17592 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/385278305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf446118e0 0x55bf44617e80 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 >> 172.21.15.87:0/385278305 conn(0x55bf4460bfb0 msgr2=0x55bf4460dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 shutdown_connections 2024-04-22T13:38:57.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 1 -- 172.21.15.87:0/385278305 wait complete. 2024-04-22T13:38:57.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.396+0000 7ffbc8c38740 20 client.17592 trim_cache size 0 max 16384 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 Processor -- start 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 -- start start 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf4461a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44626c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01b0350 con 0x55bf4461ae10 2024-04-22T13:38:57.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01adc20 con 0x55bf446118e0 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.413+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01aba90 con 0x55bf44612290 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49882/0 (socket says 172.21.15.87:49882) 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 -- 172.21.15.87:0/441966688 learned_addr learned my addr 172.21.15.87:0/441966688 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf4461a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44626c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 -- 172.21.15.87:0/441966688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 msgr2=0x55bf44626c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44626c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 -- 172.21.15.87:0/441966688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 msgr2=0x55bf4461a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf4461a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 -- 172.21.15.87:0/441966688 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf44606480 con 0x55bf4461ae10 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44626c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffbbffff640 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8013760 tx=0x7ffbb8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17436713b73b614c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffba57fa640 1 -- 172.21.15.87:0/441966688 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8011820 con 0x55bf4461ae10 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffba57fa640 1 -- 172.21.15.87:0/441966688 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb8029410 con 0x55bf4461ae10 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.414+0000 7ffba57fa640 1 -- 172.21.15.87:0/441966688 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb801f670 con 0x55bf4461ae10 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- 172.21.15.87:0/441966688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 msgr2=0x55bf44616de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8013760 tx=0x7ffbb8015000 comp rx=0 tx=0).stop 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- 172.21.15.87:0/441966688 shutdown_connections 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44616de0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44626c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/441966688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf4461a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- 172.21.15.87:0/441966688 >> 172.21.15.87:0/441966688 conn(0x55bf4460bfb0 msgr2=0x55bf4460dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- 172.21.15.87:0/441966688 shutdown_connections 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- 172.21.15.87:0/441966688 wait complete. 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 -- start start 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.415+0000 7ffbc8c38740 1 Processor -- start 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf446235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44620120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01ac0e0 con 0x55bf4461ae10 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01ad550 con 0x55bf446118e0 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc01ac7b0 con 0x55bf44612290 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 10 client.0 _ll_register_callbacks cb 0x7ffbb002c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49890/0 (socket says 172.21.15.87:49890) 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 -- 172.21.15.87:0/2377773823 learned_addr learned my addr 172.21.15.87:0/2377773823 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbeffd640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44620120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf446235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 -- 172.21.15.87:0/2377773823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 msgr2=0x55bf44620120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44620120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 -- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 msgr2=0x55bf446235b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf446235b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 -- 172.21.15.87:0/2377773823 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf44620eb0 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbbffff640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80297f0 tx=0x7ffbb8029820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba1687f1521d15a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8013260 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7ffbb8004a40 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.416+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb8029410 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffbb8013400 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 10 client.17598 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bf44692950 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb8029410 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ffbb800f040 con 0x55bf4461ae10 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 10 client.17598 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 20 client.17598 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf44692ff0 0x55bf44695410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 4 client.17598 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bf4457d790 con 0x55bf44692ff0 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf44692ff0 0x55bf44695410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffb9effd640 10 client.17598 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.417+0000 7ffbbf7fe640 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf44692ff0 0x55bf44695410 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c5f23aea2c39ed0 server_cookie=5407d07bc9f314a3 in_seq=0 out_seq=0 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffb9effd640 1 -- 172.21.15.87:0/2377773823 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bf4457d790 con 0x55bf44692ff0 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffb9effd640 10 client.17598 handle_command_reply: tid=0 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 client.17598 shutdown 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 msgr2=0x55bf44620680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80297f0 tx=0x7ffbb8029820 comp rx=0 tx=0).stop 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 shutdown_connections 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bf44692ff0 0x55bf44695410 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bf4461ae10 0x55bf44620680 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf44612290 0x55bf44620120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffb9effd640 0 client.17598 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 --2- 172.21.15.87:0/2377773823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf446118e0 0x55bf446235b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 >> 172.21.15.87:0/2377773823 conn(0x55bf4460bfb0 msgr2=0x55bf44617e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 shutdown_connections 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 1 -- 172.21.15.87:0/2377773823 wait complete. 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.419+0000 7ffbc8c38740 20 client.17598 trim_cache size 0 max 16384 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "2af2e673-3d3b-4b20-9be4-d32c3112a183", 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:38:57.421 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 msgr2=0x7ffba80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 0x7ffba80a6380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80092f0 tx=0x7ffbb8010740 comp rx=0 tx=0).stop 2024-04-22T13:38:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 msgr2=0x7ffbc0198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400c970 tx=0x7ffbb400ce30 comp rx=0 tx=0).stop 2024-04-22T13:38:57.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 shutdown_connections 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ffba80a3ec0 0x7ffba80a6380 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc019cf50 0x7ffbc019d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc0107270 0x7ffbc0198bd0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 --2- 172.21.15.87:0/1720125420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbc006a840 0x7ffbc0198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 >> 172.21.15.87:0/1720125420 conn(0x7ffbc0077eb0 msgr2=0x7ffbc0100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 shutdown_connections 2024-04-22T13:38:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.420+0000 7ffbc6043640 1 -- 172.21.15.87:0/1720125420 wait complete. 2024-04-22T13:38:57.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:38:57.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.532+0000 7fc68ac63640 1 -- 172.21.15.87:0/3367755603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 msgr2=0x7fc68410dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.532+0000 7fc68ac63640 1 --2- 172.21.15.87:0/3367755603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc68410dd20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009ca0 tx=0x7fc67c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- 172.21.15.87:0/3367755603 shutdown_connections 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- 172.21.15.87:0/3367755603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc68410dd20 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- 172.21.15.87:0/3367755603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc684101a30 0x7fc684105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- 172.21.15.87:0/3367755603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- 172.21.15.87:0/3367755603 >> 172.21.15.87:0/3367755603 conn(0x7fc684078000 msgr2=0x7fc684078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- 172.21.15.87:0/3367755603 shutdown_connections 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- 172.21.15.87:0/3367755603 wait complete. 2024-04-22T13:38:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 Processor -- start 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- start start 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 0x7fc6841a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841110c0 con 0x7fc684106150 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc684110f40 con 0x7fc68419a000 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.533+0000 7fc68ac63640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc684111240 con 0x7fc684101080 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49906/0 (socket says 172.21.15.87:49906) 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 -- 172.21.15.87:0/311092163 learned_addr learned my addr 172.21.15.87:0/311092163 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc6889ff640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc689200640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 0x7fc6841a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 -- 172.21.15.87:0/311092163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 msgr2=0x7fc684199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 -- 172.21.15.87:0/311092163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 msgr2=0x7fc6841a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 0x7fc6841a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 -- 172.21.15.87:0/311092163 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6841a4da0 con 0x7fc684106150 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc6889ff640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc689200640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 0x7fc6841a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc683fff640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc67000d770 tx=0x7fc67000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc681ffb640 1 -- 172.21.15.87:0/311092163 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670017070 con 0x7fc684106150 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc681ffb640 1 -- 172.21.15.87:0/311092163 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc670009c30 con 0x7fc684106150 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc681ffb640 1 -- 172.21.15.87:0/311092163 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67000f620 con 0x7fc684106150 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6841a5090 con 0x7fc684106150 2024-04-22T13:38:57.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.534+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6841a55e0 con 0x7fc684106150 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.536+0000 7fc681ffb640 1 -- 172.21.15.87:0/311092163 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc67001a070 con 0x7fc684106150 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 Processor -- start 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 -- start start 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc53b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc544540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc546e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556dcc4a01e0 con 0x556dcc53b580 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556dcc43d390 con 0x556dcc53bea0 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc68d858740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556dcc49c1a0 con 0x556dcc544a80 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc681ffb640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 0x7fc6740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc689200640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc53b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc689200640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc53b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49916/0 (socket says 172.21.15.87:49916) 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc689200640 1 -- 172.21.15.87:0/3772448699 learned_addr learned my addr 172.21.15.87:0/3772448699 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc544540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc689200640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 0x7fc6740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc683fff640 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc546e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc681ffb640 1 -- 172.21.15.87:0/311092163 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fc6700d02f0 con 0x7fc684106150 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc6889ff640 1 -- 172.21.15.87:0/3772448699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 msgr2=0x556dcc546e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc546e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc6889ff640 1 -- 172.21.15.87:0/3772448699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 msgr2=0x556dcc53b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.537+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc53b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc6889ff640 1 -- 172.21.15.87:0/3772448699 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dcc27c7d0 con 0x556dcc53bea0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc689200640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 0x7fc6740a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc67c032460 tx=0x7fc67c005f10 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc544540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc67800cbd0 tx=0x7fc67800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f13f2bcb0fc49c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc66effd640 1 -- 172.21.15.87:0/3772448699 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678005220 con 0x556dcc53bea0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc66effd640 1 -- 172.21.15.87:0/3772448699 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc678004c50 con 0x556dcc53bea0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.538+0000 7fc66effd640 1 -- 172.21.15.87:0/3772448699 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678010700 con 0x556dcc53bea0 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- 172.21.15.87:0/3772448699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 msgr2=0x556dcc544540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc544540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc67800cbd0 tx=0x7fc67800e720 comp rx=0 tx=0).stop 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- 172.21.15.87:0/3772448699 shutdown_connections 2024-04-22T13:38:57.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc546e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc544540 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- 172.21.15.87:0/3772448699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc53b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- 172.21.15.87:0/3772448699 >> 172.21.15.87:0/3772448699 conn(0x556dcc535cb0 msgr2=0x556dcc536090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- 172.21.15.87:0/3772448699 shutdown_connections 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- 172.21.15.87:0/3772448699 wait complete. 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 -- start start 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 Processor -- start 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.539+0000 7fc68d858740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc542520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556dcc547870 con 0x556dcc53b580 2024-04-22T13:38:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556dcc4592f0 con 0x556dcc53bea0 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556dcc5479f0 con 0x556dcc544a80 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 10 client.0 _ll_register_callbacks cb 0x556dcc466580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49922/0 (socket says 172.21.15.87:49922) 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 -- 172.21.15.87:0/3446421466 learned_addr learned my addr 172.21.15.87:0/3446421466 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc689200640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc542520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc683fff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 -- 172.21.15.87:0/3446421466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 msgr2=0x556dcc542520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc542520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 -- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 msgr2=0x556dcc541fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 -- 172.21.15.87:0/3446421466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dcc1b23f0 con 0x556dcc53b580 2024-04-22T13:38:57.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc689200640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc542520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc683fff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc6889ff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc67800c940 tx=0x7fc678004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed97bfbc54fea7d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc663fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67802a060 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fc678005150 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc678004c50 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc678005350 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 10 client.17610 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.540+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556dcc1720f0 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc678004c50 con 0x556dcc53b580 2024-04-22T13:38:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc67801a620 con 0x556dcc53b580 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 10 client.17610 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 10 client.17610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 20 client.17610 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5549f0 0x556dcc556de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 4 client.17610 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dcc4adc00 con 0x556dcc5549f0 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc683fff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5549f0 0x556dcc556de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc663fff640 10 client.17610 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.541+0000 7fc683fff640 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5549f0 0x556dcc556de0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86307b159938cab server_cookie=4444d3cf7449b51b in_seq=0 out_seq=0 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc663fff640 1 -- 172.21.15.87:0/3446421466 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dcc4adc00 con 0x556dcc5549f0 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc663fff640 10 client.17610 handle_command_reply: tid=0 2024-04-22T13:38:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 client.17610 shutdown 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 msgr2=0x556dcc541aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc67800c940 tx=0x7fc678004b60 comp rx=0 tx=0).stop 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 shutdown_connections 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5549f0 0x556dcc556de0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc542520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc663fff640 0 client.17610 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 --2- 172.21.15.87:0/3446421466 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc541aa0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 >> 172.21.15.87:0/3446421466 conn(0x556dcc535cb0 msgr2=0x556dcc537a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 shutdown_connections 2024-04-22T13:38:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 1 -- 172.21.15.87:0/3446421466 wait complete. 2024-04-22T13:38:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.542+0000 7fc68d858740 20 client.17610 trim_cache size 0 max 16384 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.559+0000 7fc68d858740 1 Processor -- start 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.559+0000 7fc68d858740 1 -- start start 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc537740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841b0350 con 0x556dcc53b580 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841adc20 con 0x556dcc53bea0 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc68d858740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841aba90 con 0x556dcc544a80 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49938/0 (socket says 172.21.15.87:49938) 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 -- 172.21.15.87:0/4230527296 learned_addr learned my addr 172.21.15.87:0/4230527296 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc683fff640 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 -- 172.21.15.87:0/4230527296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 msgr2=0x556dcc537740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc537740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 -- 172.21.15.87:0/4230527296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 msgr2=0x556dcc541380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 -- 172.21.15.87:0/4230527296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dcc53c850 con 0x556dcc53b580 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc6889ff640 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc6780058c0 tx=0x7fc67800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826aeeb56b11f0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc66e7fc640 1 -- 172.21.15.87:0/4230527296 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67802a070 con 0x556dcc53b580 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc66e7fc640 1 -- 172.21.15.87:0/4230527296 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc678005150 con 0x556dcc53b580 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.560+0000 7fc66e7fc640 1 -- 172.21.15.87:0/4230527296 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67801ad20 con 0x556dcc53b580 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- 172.21.15.87:0/4230527296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 msgr2=0x556dcc550730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc6780058c0 tx=0x7fc67800ef50 comp rx=0 tx=0).stop 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- 172.21.15.87:0/4230527296 shutdown_connections 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc544a80 0x556dcc537740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc53bea0 0x556dcc541380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- 172.21.15.87:0/4230527296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc550730 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- 172.21.15.87:0/4230527296 >> 172.21.15.87:0/4230527296 conn(0x556dcc535cb0 msgr2=0x556dcc537030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- 172.21.15.87:0/4230527296 shutdown_connections 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- 172.21.15.87:0/4230527296 wait complete. 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- start start 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 Processor -- start 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc53bea0 0x556dcc548740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 0x556dcc54c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841ac0e0 con 0x556dcc53b580 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841ad550 con 0x556dcc544a80 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841ac7b0 con 0x556dcc53bea0 2024-04-22T13:38:57.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.561+0000 7fc68d858740 10 client.0 _ll_register_callbacks cb 0x7fc678014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49950/0 (socket says 172.21.15.87:49950) 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 -- 172.21.15.87:0/605905506 learned_addr learned my addr 172.21.15.87:0/605905506 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc689200640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 0x556dcc54c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc683fff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc53bea0 0x556dcc548740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 -- 172.21.15.87:0/605905506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc53bea0 msgr2=0x556dcc548740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc53bea0 0x556dcc548740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 -- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 msgr2=0x556dcc54c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 0x556dcc54c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 -- 172.21.15.87:0/605905506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dcc54cc60 con 0x556dcc53b580 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc689200640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 0x556dcc54c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc6889ff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc67801ab90 tx=0x7fc678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b4fd44e3edea0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc663fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:38:57.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67802a030 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7fc678004020 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc678005150 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc678004220 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc68d858740 10 client.17616 fetch_fsmap learned FSMap version 23 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556dcc5ba760 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.562+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc678005150 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc678038480 con 0x556dcc53b580 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 10 client.17616 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 10 client.17616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 20 client.17616 populate_metadata read hostname 'smithi087' 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5badf0 0x556dcc5bd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 4 client.17616 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556dcc4a7a70 con 0x556dcc5badf0 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc683fff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5badf0 0x556dcc5bd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:38:57.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc683fff640 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5badf0 0x556dcc5bd210 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82d64d9d7be33a9 server_cookie=4b006407614e1433 in_seq=0 out_seq=0 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc663fff640 10 client.17616 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc663fff640 1 -- 172.21.15.87:0/605905506 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556dcc4a7a70 con 0x556dcc5badf0 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.563+0000 7fc663fff640 10 client.17616 handle_command_reply: tid=0 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 client.17616 shutdown 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 msgr2=0x556dcc548200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc67801ab90 tx=0x7fc678004a40 comp rx=0 tx=0).stop 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 shutdown_connections 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556dcc5badf0 0x556dcc5bd210 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556dcc544a80 0x556dcc54c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc663fff640 0 client.17616 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556dcc53bea0 0x556dcc548740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 --2- 172.21.15.87:0/605905506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556dcc53b580 0x556dcc548200 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 >> 172.21.15.87:0/605905506 conn(0x556dcc535cb0 msgr2=0x556dcc5378c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 shutdown_connections 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 1 -- 172.21.15.87:0/605905506 wait complete. 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68d858740 20 client.17616 trim_cache size 0 max 16384 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "2af2e673-3d3b-4b20-9be4-d32c3112a183": { 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "2af2e673-3d3b-4b20-9be4-d32c3112a183", 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:38:57.567 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 msgr2=0x7fc6740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 0x7fc6740a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc67c032460 tx=0x7fc67c005f10 comp rx=0 tx=0).stop 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 msgr2=0x7fc684199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc67000d770 tx=0x7fc67000dc30 comp rx=0 tx=0).stop 2024-04-22T13:38:57.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 shutdown_connections 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6740a3d50 0x7fc6740a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc68419a000 0x7fc6841a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc684106150 0x7fc684199ac0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 --2- 172.21.15.87:0/311092163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc684101080 0x7fc684199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 >> 172.21.15.87:0/311092163 conn(0x7fc684078000 msgr2=0x7fc6840fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.564+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 shutdown_connections 2024-04-22T13:38:57.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:38:57.565+0000 7fc68ac63640 1 -- 172.21.15.87:0/311092163 wait complete. 2024-04-22T13:38:57.571 INFO:tasks.cephfs.filesystem:scrub status for tag:2af2e673-3d3b-4b20-9be4-d32c3112a183 - {'path': '/', 'tag': '2af2e673-3d3b-4b20-9be4-d32c3112a183', 'options': 'recursive,force'} 2024-04-22T13:38:58.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:58 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:38:58.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:58 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:58.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:58 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:38:58.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:58 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:58.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:58 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:38:58.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:58 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:58.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:58 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:38:58.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:58 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:58.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:58 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:38:58.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:58 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:58.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:58 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:38:58.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:58 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:38:59.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:38:59 smithi120 ceph-mon[31101]: pgmap v219: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 422 op/s 2024-04-22T13:38:59.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:38:59 smithi132 ceph-mon[31531]: pgmap v219: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 422 op/s 2024-04-22T13:38:59.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:38:59 smithi087 ceph-mon[27999]: pgmap v219: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 422 op/s 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 -- 172.21.15.87:0/2935975455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f4102b80 msgr2=0x7f16f4106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 --2- 172.21.15.87:0/2935975455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f4102b80 0x7f16f4106bc0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009900 tx=0x7f16e402bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 -- 172.21.15.87:0/2935975455 shutdown_connections 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 --2- 172.21.15.87:0/2935975455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f41072a0 0x7f16f410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 --2- 172.21.15.87:0/2935975455 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f4102b80 0x7f16f4106bc0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.164+0000 7f16fbf50640 1 --2- 172.21.15.87:0/2935975455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41021d0 0x7f16f41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- 172.21.15.87:0/2935975455 >> 172.21.15.87:0/2935975455 conn(0x7f16f40793e0 msgr2=0x7f16f40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- 172.21.15.87:0/2935975455 shutdown_connections 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- 172.21.15.87:0/2935975455 wait complete. 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 Processor -- start 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- start start 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41072a0 0x7f16f4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f419cf60 0x7f16f419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f16f4112240 con 0x7f16f41021d0 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41120c0 con 0x7f16f419cf60 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.165+0000 7f16fbf50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41123c0 con 0x7f16f41072a0 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37366/0 (socket says 172.21.15.87:37366) 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 -- 172.21.15.87:0/3360211507 learned_addr learned my addr 172.21.15.87:0/3360211507 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f94eb640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41072a0 0x7f16f4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f419cf60 0x7f16f419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 -- 172.21.15.87:0/3360211507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41072a0 msgr2=0x7f16f4198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41072a0 0x7f16f4198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 -- 172.21.15.87:0/3360211507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f419cf60 msgr2=0x7f16f419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f419cf60 0x7f16f419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 -- 172.21.15.87:0/3360211507 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f419da50 con 0x7f16f41021d0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16f9cec640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f16f000b4a0 tx=0x7f16f000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16eaffd640 1 -- 172.21.15.87:0/3360211507 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f0014070 con 0x7f16f41021d0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16eaffd640 1 -- 172.21.15.87:0/3360211507 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16f0010ac0 con 0x7f16f41021d0 2024-04-22T13:39:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f419dd40 con 0x7f16f41021d0 2024-04-22T13:39:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16eaffd640 1 -- 172.21.15.87:0/3360211507 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f0017c30 con 0x7f16f41021d0 2024-04-22T13:39:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.166+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f41a54b0 con 0x7f16f41021d0 2024-04-22T13:39:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.168+0000 7f16eaffd640 1 -- 172.21.15.87:0/3360211507 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f16f0017070 con 0x7f16f41021d0 2024-04-22T13:39:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 Processor -- start 2024-04-22T13:39:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 -- start start 2024-04-22T13:39:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb54fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16eaffd640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 0x7f16e00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb604d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bd6bab98d0 con 0x55bd6bb558d0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd6ba56e80 con 0x55bd6bb54b60 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fec1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bd6bab5890 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16f9cec640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 0x7f16e00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37374/0 (socket says 172.21.15.87:37374) 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.169+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2135560022 learned_addr learned my addr 172.21.15.87:0/2135560022 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16f94eb640 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb54fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16eaffd640 1 -- 172.21.15.87:0/3360211507 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f16f00cfa60 con 0x7f16f41021d0 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2135560022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb5e0e0 msgr2=0x55bd6bb604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2135560022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 msgr2=0x55bd6bb54fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb54fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2135560022 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6ba78320 con 0x55bd6bb558d0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16f9cec640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 0x7f16e00a6280 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f16f0098520 tx=0x7f16f006d590 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f16ec014790 tx=0x7f16ec0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40fe510aed374c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16dffff640 1 -- 172.21.15.87:0/2135560022 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec01a070 con 0x55bd6bb558d0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16dffff640 1 -- 172.21.15.87:0/2135560022 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ec012910 con 0x55bd6bb558d0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.170+0000 7f16dffff640 1 -- 172.21.15.87:0/2135560022 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec005420 con 0x55bd6bb558d0 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- 172.21.15.87:0/2135560022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 msgr2=0x55bd6bb54620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f16ec014790 tx=0x7f16ec0098e0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- 172.21.15.87:0/2135560022 shutdown_connections 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb54fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2135560022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb54620 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- 172.21.15.87:0/2135560022 >> 172.21.15.87:0/2135560022 conn(0x55bd6bb1bc90 msgr2=0x55bd6bb4f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- 172.21.15.87:0/2135560022 shutdown_connections 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- 172.21.15.87:0/2135560022 wait complete. 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 -- start start 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.171+0000 7f16fec1a740 1 Processor -- start 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb5da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bd6bb60e10 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd6ba41eb0 con 0x55bd6bb558d0 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bd6bb60f90 con 0x55bd6bb54b60 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fec1a740 10 client.0 _ll_register_callbacks cb 0x55bd6ba6eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37382/0 (socket says 172.21.15.87:37382) 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2054090271 learned_addr learned my addr 172.21.15.87:0/2054090271 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16f94eb640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb5da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2054090271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 msgr2=0x55bd6bb5da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb5da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb558d0 msgr2=0x55bd6bb5ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 -- 172.21.15.87:0/2054090271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd6b7cc210 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009c10 tx=0x7f16ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31744132c8f8a7c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.172+0000 7f16dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec022070 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f16ec004c50 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ec012910 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f16ec034050 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 10 client.17628 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bd6b898620 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ec012910 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f16ec00f040 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 10 client.17628 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 10 client.17628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 20 client.17628 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb6dfb0 0x55bd6bb703a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 4 client.17628 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.173+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd6bacd880 con 0x55bd6bb6dfb0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.174+0000 7f16f94eb640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb6dfb0 0x55bd6bb703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.174+0000 7f16f94eb640 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb6dfb0 0x55bd6bb703a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eb83e5f093192f server_cookie=717c6e21fa54f87b in_seq=0 out_seq=0 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.174+0000 7f16dd7fa640 10 client.17628 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:00.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16dd7fa640 1 -- 172.21.15.87:0/2054090271 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd6bacd880 con 0x55bd6bb6dfb0 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16dd7fa640 10 client.17628 handle_command_reply: tid=0 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 client.17628 shutdown 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 msgr2=0x55bd6bb5b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009c10 tx=0x7f16ec009b20 comp rx=0 tx=0).stop 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 shutdown_connections 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb6dfb0 0x55bd6bb703a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb5e0e0 0x55bd6bb5b2d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16dd7fa640 0 client.17628 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 --2- 172.21.15.87:0/2054090271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb5da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 >> 172.21.15.87:0/2054090271 conn(0x55bd6bb1bc90 msgr2=0x55bd6bb515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 shutdown_connections 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 1 -- 172.21.15.87:0/2054090271 wait complete. 2024-04-22T13:39:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.175+0000 7f16fec1a740 20 client.17628 trim_cache size 0 max 16384 2024-04-22T13:39:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 Processor -- start 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 -- start start 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb5a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb51340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41b0340 con 0x55bd6bb558d0 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41adc10 con 0x55bd6bb1bc90 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fec1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41aba80 con 0x55bd6bb54b60 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37396/0 (socket says 172.21.15.87:37396) 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 -- 172.21.15.87:0/1325303840 learned_addr learned my addr 172.21.15.87:0/1325303840 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16f94eb640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb51340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb5a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 -- 172.21.15.87:0/1325303840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 msgr2=0x55bd6bb51340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb51340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 -- 172.21.15.87:0/1325303840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb1bc90 msgr2=0x55bd6bb5a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb5a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16fa4ed640 1 -- 172.21.15.87:0/1325303840 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6bb53e30 con 0x55bd6bb558d0 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.190+0000 7f16f94eb640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb51340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f16ec014790 tx=0x7f16ec016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4654cd8db9db5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16df7fe640 1 -- 172.21.15.87:0/1325303840 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec022030 con 0x55bd6bb558d0 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16df7fe640 1 -- 172.21.15.87:0/1325303840 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ec014dd0 con 0x55bd6bb558d0 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16df7fe640 1 -- 172.21.15.87:0/1325303840 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec01a040 con 0x55bd6bb558d0 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16fec1a740 1 -- 172.21.15.87:0/1325303840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 msgr2=0x55bd6bb5d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.191+0000 7f16fec1a740 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f16ec014790 tx=0x7f16ec016040 comp rx=0 tx=0).stop 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- 172.21.15.87:0/1325303840 shutdown_connections 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb5d4c0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb54b60 0x55bd6bb51340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- 172.21.15.87:0/1325303840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb5a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- 172.21.15.87:0/1325303840 >> 172.21.15.87:0/1325303840 conn(0x55bd6bb5b430 msgr2=0x55bd6bb50c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- 172.21.15.87:0/1325303840 shutdown_connections 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- 172.21.15.87:0/1325303840 wait complete. 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- start start 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 Processor -- start 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb66570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41ac0d0 con 0x55bd6bb558d0 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41ad540 con 0x55bd6bb54b60 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f16f41ac7a0 con 0x55bd6bb1bc90 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fec1a740 10 client.0 _ll_register_callbacks cb 0x7f16e4005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.192+0000 7f16fa4ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37412/0 (socket says 172.21.15.87:37412) 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 -- 172.21.15.87:0/3737217508 learned_addr learned my addr 172.21.15.87:0/3737217508 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb66570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16f94eb640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 -- 172.21.15.87:0/3737217508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb1bc90 msgr2=0x55bd6bb66570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb66570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 -- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 msgr2=0x55bd6bb618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 -- 172.21.15.87:0/3737217508 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd6bb62620 con 0x55bd6bb558d0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16f94eb640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fa4ed640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009700 tx=0x7f16ec01eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7997dd6ddc6d2ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec01a070 con 0x55bd6bb558d0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f16ec014230 con 0x55bd6bb558d0 2024-04-22T13:39:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd6ba593d0 con 0x55bd6bb558d0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f16ec04e050 con 0x55bd6bb558d0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fec1a740 10 client.17634 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bd6bbd42d0 con 0x55bd6bb558d0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.193+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bd6ba593d0 con 0x55bd6bb558d0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f16ec045070 con 0x55bd6bb558d0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 10 client.17634 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 10 client.17634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 20 client.17634 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb5e0e0 0x55bd6bbd68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 4 client.17634 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd6bac72f0 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16f94eb640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb5e0e0 0x55bd6bbd68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16f94eb640 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb5e0e0 0x55bd6bbd68c0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5bedea3951dc38f server_cookie=ce518eaadf98228d in_seq=0 out_seq=0 2024-04-22T13:39:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.194+0000 7f16dd7fa640 10 client.17634 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16dd7fa640 1 -- 172.21.15.87:0/3737217508 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd6bac72f0 con 0x55bd6bb5e0e0 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16dd7fa640 10 client.17634 handle_command_reply: tid=0 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 client.17634 shutdown 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 msgr2=0x55bd6bb61e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009700 tx=0x7f16ec01eee0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 shutdown_connections 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bd6bb5e0e0 0x55bd6bbd68c0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bd6bb558d0 0x55bd6bb61e20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16dd7fa640 0 client.17634 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd6bb54b60 0x55bd6bb618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 --2- 172.21.15.87:0/3737217508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bd6bb1bc90 0x55bd6bb66570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 >> 172.21.15.87:0/3737217508 conn(0x55bd6bb5b430 msgr2=0x55bd6bb51450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.203+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 shutdown_connections 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fec1a740 1 -- 172.21.15.87:0/3737217508 wait complete. 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fec1a740 20 client.17634 trim_cache size 0 max 16384 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:39:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 msgr2=0x7f16e00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 0x7f16e00a6280 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f16f0098520 tx=0x7f16f006d590 comp rx=0 tx=0).stop 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 msgr2=0x7f16f41986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f16f000b4a0 tx=0x7f16f000b960 comp rx=0 tx=0).stop 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 shutdown_connections 2024-04-22T13:39:00.202 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f16e00a3dc0 0x7f16e00a6280 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f419cf60 0x7f16f419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f41072a0 0x7f16f4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 --2- 172.21.15.87:0/3360211507 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16f41021d0 0x7f16f41986a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 >> 172.21.15.87:0/3360211507 conn(0x7f16f40793e0 msgr2=0x7f16f40ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 shutdown_connections 2024-04-22T13:39:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:00.204+0000 7f16fbf50640 1 -- 172.21.15.87:0/3360211507 wait complete. 2024-04-22T13:39:00.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:00 smithi120 ceph-mon[31101]: pgmap v220: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 15 MiB/s wr, 172 op/s 2024-04-22T13:39:00.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:00 smithi132 ceph-mon[31531]: pgmap v220: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 15 MiB/s wr, 172 op/s 2024-04-22T13:39:00.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:00 smithi087 ceph-mon[27999]: pgmap v220: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 15 MiB/s wr, 172 op/s 2024-04-22T13:39:01.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:01 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:39:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:01 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:39:01.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:01 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:39:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:02 smithi120 ceph-mon[31101]: pgmap v221: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.5 MiB/s wr, 119 op/s 2024-04-22T13:39:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:02 smithi132 ceph-mon[31531]: pgmap v221: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.5 MiB/s wr, 119 op/s 2024-04-22T13:39:02.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:02 smithi087 ceph-mon[27999]: pgmap v221: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.5 MiB/s wr, 119 op/s 2024-04-22T13:39:04.269 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.2 2024-04-22T13:39:04.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 -- 172.21.15.87:0/2503604236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec06b040 msgr2=0x7f38ec10edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 --2- 172.21.15.87:0/2503604236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec06b040 0x7f38ec10edc0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009ca0 tx=0x7f38e402c180 comp rx=0 tx=0).stop 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 -- 172.21.15.87:0/2503604236 shutdown_connections 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 --2- 172.21.15.87:0/2503604236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec06b040 0x7f38ec10edc0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 --2- 172.21.15.87:0/2503604236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 0x7f38ec06ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 --2- 172.21.15.87:0/2503604236 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec108020 0x7f38ec108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 -- 172.21.15.87:0/2503604236 >> 172.21.15.87:0/2503604236 conn(0x7f38ec077e30 msgr2=0x7f38ec078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 -- 172.21.15.87:0/2503604236 shutdown_connections 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.385+0000 7f38f4759640 1 -- 172.21.15.87:0/2503604236 wait complete. 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 Processor -- start 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 -- start start 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 0x7f38ec1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 0x7f38ec198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec112150 con 0x7f38ec108020 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec111fd0 con 0x7f38ec06a680 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f4759640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1122d0 con 0x7f38ec06b040 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37436/0 (socket says 172.21.15.87:37436) 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 -- 172.21.15.87:0/696570982 learned_addr learned my addr 172.21.15.87:0/696570982 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f24f5640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 0x7f38ec1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f1cf4640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 0x7f38ec198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 -- 172.21.15.87:0/696570982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 msgr2=0x7f38ec198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 0x7f38ec198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 -- 172.21.15.87:0/696570982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 msgr2=0x7f38ec1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 0x7f38ec1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f2cf6640 1 -- 172.21.15.87:0/696570982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ec19d5e0 con 0x7f38ec108020 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.386+0000 7f38f1cf4640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 0x7f38ec198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38f2cf6640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f38e402c790 tx=0x7f38e4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e4039040 con 0x7f38ec108020 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e4044070 con 0x7f38ec108020 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ec19d870 con 0x7f38ec108020 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e40327c0 con 0x7f38ec108020 2024-04-22T13:39:04.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.387+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ec1a5160 con 0x7f38ec108020 2024-04-22T13:39:04.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.388+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ec069230 con 0x7f38ec108020 2024-04-22T13:39:04.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.389+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f38e4004240 con 0x7f38ec108020 2024-04-22T13:39:04.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.389+0000 7f38e37fe640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 0x7f38d80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.390+0000 7f38f24f5640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 0x7f38d80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.390+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f38e40f2b50 con 0x7f38ec108020 2024-04-22T13:39:04.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.390+0000 7f38f24f5640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 0x7f38d80a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38e800ab60 tx=0x7f38e800a2d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.394+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38e40bf340 con 0x7f38ec108020 2024-04-22T13:39:04.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.611+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38ec10a470 con 0x7f38ec108020 2024-04-22T13:39:04.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.612+0000 7f38e37fe640 1 -- 172.21.15.87:0/696570982 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f38e40bec40 con 0x7f38ec108020 2024-04-22T13:39:04.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:39:04.607 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:39:04.608 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 msgr2=0x7f38d80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 0x7f38d80a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38e800ab60 tx=0x7f38e800a2d0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 msgr2=0x7f38ec19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f38e402c790 tx=0x7f38e4003c20 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 shutdown_connections 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38d80a3da0 0x7f38d80a6260 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f38ec108020 0x7f38ec19ceb0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38ec06b040 0x7f38ec198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 --2- 172.21.15.87:0/696570982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38ec06a680 0x7f38ec1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 >> 172.21.15.87:0/696570982 conn(0x7f38ec077e30 msgr2=0x7f38ec0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 shutdown_connections 2024-04-22T13:39:04.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.614+0000 7f38f4759640 1 -- 172.21.15.87:0/696570982 wait complete. 2024-04-22T13:39:04.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:04 smithi087 ceph-mon[27999]: pgmap v222: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.5 MiB/s wr, 73 op/s 2024-04-22T13:39:04.620 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 74.2 --await' 2024-04-22T13:39:04.621 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 74.2 --await 2024-04-22T13:39:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:04 smithi120 ceph-mon[31101]: pgmap v222: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.5 MiB/s wr, 73 op/s 2024-04-22T13:39:04.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:04 smithi132 ceph-mon[31531]: pgmap v222: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.5 MiB/s wr, 73 op/s 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- 172.21.15.87:0/1810832898 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 msgr2=0x7f068c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 --2- 172.21.15.87:0/1810832898 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c1014f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0688009900 tx=0x7f068802bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- 172.21.15.87:0/1810832898 shutdown_connections 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 --2- 172.21.15.87:0/1810832898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 0x7f068c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 --2- 172.21.15.87:0/1810832898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c101ac0 0x7f068c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 --2- 172.21.15.87:0/1810832898 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c1014f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- 172.21.15.87:0/1810832898 >> 172.21.15.87:0/1810832898 conn(0x7f068c077fb0 msgr2=0x7f068c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- 172.21.15.87:0/1810832898 shutdown_connections 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- 172.21.15.87:0/1810832898 wait complete. 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 Processor -- start 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.732+0000 7f06935a4640 1 -- start start 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 0x7f068c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c198590 0x7f068c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f068c112210 con 0x7f068c101110 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f068c112090 con 0x7f068c198590 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f068c112390 con 0x7f068c106190 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37456/0 (socket says 172.21.15.87:37456) 2024-04-22T13:39:04.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 -- 172.21.15.87:0/3390984957 learned_addr learned my addr 172.21.15.87:0/3390984957 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691b41640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c198590 0x7f068c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0690b3f640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 0x7f068c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 -- 172.21.15.87:0/3390984957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 msgr2=0x7f068c19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 0x7f068c19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 -- 172.21.15.87:0/3390984957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c198590 msgr2=0x7f068c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c198590 0x7f068c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 -- 172.21.15.87:0/3390984957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1990b0 con 0x7f068c101110 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f0691340640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0688005940 tx=0x7f068802c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06827fc640 1 -- 172.21.15.87:0/3390984957 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688043020 con 0x7f068c101110 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06827fc640 1 -- 172.21.15.87:0/3390984957 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f068802ca70 con 0x7f068c101110 2024-04-22T13:39:04.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c199340 con 0x7f068c101110 2024-04-22T13:39:04.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06827fc640 1 -- 172.21.15.87:0/3390984957 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880039f0 con 0x7f068c101110 2024-04-22T13:39:04.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.733+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1a5590 con 0x7f068c101110 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.736+0000 7f06827fc640 1 -- 172.21.15.87:0/3390984957 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0688046020 con 0x7f068c101110 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.736+0000 7f0696196740 1 Processor -- start 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.736+0000 7f0696196740 1 -- start start 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.736+0000 7f0696196740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.736+0000 7f0696196740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b839d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0696196740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b83c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0696196740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56067b78d0b0 con 0x56067b82c6f0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0696196740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56067b73e680 con 0x56067b82d010 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0696196740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56067b74ac60 con 0x56067b83a2d0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37470/0 (socket says 172.21.15.87:37470) 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 -- 172.21.15.87:0/1609424248 learned_addr learned my addr 172.21.15.87:0/1609424248 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f06827fc640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 0x7f06780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b83c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691340640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b839d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 -- 172.21.15.87:0/1609424248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 msgr2=0x56067b83c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b83c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 -- 172.21.15.87:0/1609424248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 msgr2=0x56067b839d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b839d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 -- 172.21.15.87:0/1609424248 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56067b56fc90 con 0x56067b82c6f0 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b83c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:04.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f06827fc640 1 -- 172.21.15.87:0/3390984957 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f06880f2870 con 0x7f068c101110 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 0x7f06780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0684011430 tx=0x7f06840118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0286d599ba830d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f06777fe640 1 -- 172.21.15.87:0/1609424248 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068400f6e0 con 0x56067b82c6f0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f06777fe640 1 -- 172.21.15.87:0/1609424248 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f068400fd30 con 0x56067b82c6f0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f06777fe640 1 -- 172.21.15.87:0/1609424248 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684012c50 con 0x56067b82c6f0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.737+0000 7f0691b41640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 0x7f06780a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0684022000 tx=0x7f068401f000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- 172.21.15.87:0/1609424248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 msgr2=0x56067b82cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0684011430 tx=0x7f06840118f0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- 172.21.15.87:0/1609424248 shutdown_connections 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b83c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b839d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 --2- 172.21.15.87:0/1609424248 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b82cad0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- 172.21.15.87:0/1609424248 >> 172.21.15.87:0/1609424248 conn(0x56067b8270f0 msgr2=0x56067b8274d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- 172.21.15.87:0/1609424248 shutdown_connections 2024-04-22T13:39:04.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- 172.21.15.87:0/1609424248 wait complete. 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.738+0000 7f0696196740 1 -- start start 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 Processor -- start 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b837880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b837dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56067b790400 con 0x56067b82c6f0 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56067b74c840 con 0x56067b82d010 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56067b795720 con 0x56067b83a2d0 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0696196740 10 client.0 _ll_register_callbacks cb 0x56067b79dea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37484/0 (socket says 172.21.15.87:37484) 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 -- 172.21.15.87:0/1938073596 learned_addr learned my addr 172.21.15.87:0/1938073596 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b837880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691b41640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b837dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 -- 172.21.15.87:0/1938073596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 msgr2=0x56067b837dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b837dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 -- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 msgr2=0x56067b837880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b837880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 -- 172.21.15.87:0/1938073596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56067b5d0420 con 0x56067b82c6f0 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691b41640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b837dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0691340640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f06880f0170 tx=0x7f06880be560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0daf67bcb75ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.739+0000 7f0674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880be960 con 0x56067b82c6f0 2024-04-22T13:39:04.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f06880bafd0 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f068400fd30 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f06880c2d10 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 10 client.17652 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56067b463580 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f068400fd30 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f06880d50d0 con 0x56067b82c6f0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 10 client.17652 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 10 client.17652 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 20 client.17652 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b84a010 0x56067b84c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 4 client.17652 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.740+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56067b79eb10 con 0x56067b84a010 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.741+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b84a010 0x56067b84c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.741+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b84a010 0x56067b84c4b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa0cb104c6475f4 server_cookie=4ab56d67ef6f2b4b in_seq=0 out_seq=0 2024-04-22T13:39:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.741+0000 7f0674ff9640 10 client.17652 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0674ff9640 1 -- 172.21.15.87:0/1938073596 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56067b79eb10 con 0x56067b84a010 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0674ff9640 10 client.17652 handle_command_reply: tid=0 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 client.17652 shutdown 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 msgr2=0x56067b837340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f06880f0170 tx=0x7f06880be560 comp rx=0 tx=0).stop 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 shutdown_connections 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b84a010 0x56067b84c4b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b837dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0674ff9640 0 client.17652 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b837880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 --2- 172.21.15.87:0/1938073596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82c6f0 0x56067b837340 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 >> 172.21.15.87:0/1938073596 conn(0x56067b8270f0 msgr2=0x56067b828e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 shutdown_connections 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 1 -- 172.21.15.87:0/1938073596 wait complete. 2024-04-22T13:39:04.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.742+0000 7f0696196740 20 client.17652 trim_cache size 0 max 16384 2024-04-22T13:39:04.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 Processor -- start 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 -- start start 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b82c6f0 0x56067b839a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b836280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1ab9c0 con 0x56067b83a2d0 2024-04-22T13:39:04.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1a9830 con 0x56067b82d010 2024-04-22T13:39:04.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0696196740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1ab2f0 con 0x56067b82c6f0 2024-04-22T13:39:04.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0691b41640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0691b41640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37488/0 (socket says 172.21.15.87:37488) 2024-04-22T13:39:04.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0691b41640 1 -- 172.21.15.87:0/1397754104 learned_addr learned my addr 172.21.15.87:0/1397754104 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0691340640 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b82c6f0 0x56067b839a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.759+0000 7f0691b41640 1 -- 172.21.15.87:0/1397754104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b82c6f0 msgr2=0x56067b839a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0691b41640 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b82c6f0 0x56067b839a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0691b41640 1 -- 172.21.15.87:0/1397754104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 msgr2=0x56067b836280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0691b41640 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b836280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0691b41640 1 -- 172.21.15.87:0/1397754104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56067b82d740 con 0x56067b83a2d0 2024-04-22T13:39:04.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0691b41640 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0684011400 tx=0x7f0684004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b833b31b339ba296 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0676ffd640 1 -- 172.21.15.87:0/1397754104 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684009bf0 con 0x56067b83a2d0 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0676ffd640 1 -- 172.21.15.87:0/1397754104 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06840099c0 con 0x56067b83a2d0 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0676ffd640 1 -- 172.21.15.87:0/1397754104 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068400f4e0 con 0x56067b83a2d0 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 -- 172.21.15.87:0/1397754104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 msgr2=0x56067b845c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0684011400 tx=0x7f0684004a00 comp rx=0 tx=0).stop 2024-04-22T13:39:04.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 -- 172.21.15.87:0/1397754104 shutdown_connections 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b83a2d0 0x56067b845c90 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82d010 0x56067b836280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 --2- 172.21.15.87:0/1397754104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b82c6f0 0x56067b839a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.760+0000 7f0696196740 1 -- 172.21.15.87:0/1397754104 >> 172.21.15.87:0/1397754104 conn(0x56067b8270f0 msgr2=0x56067b8261d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- 172.21.15.87:0/1397754104 shutdown_connections 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- 172.21.15.87:0/1397754104 wait complete. 2024-04-22T13:39:04.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- start start 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 Processor -- start 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 0x56067b83e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b8af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1aa550 con 0x56067b82d010 2024-04-22T13:39:04.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1a9e80 con 0x56067b82c6f0 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1aac20 con 0x56067b83a2d0 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0696196740 10 client.0 _ll_register_callbacks cb 0x7f06880df0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0690b3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0690b3f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37490/0 (socket says 172.21.15.87:37490) 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0690b3f640 1 -- 172.21.15.87:0/1794295637 learned_addr learned my addr 172.21.15.87:0/1794295637 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:04.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.761+0000 7f0691340640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 0x56067b83e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0691b41640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b8af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 -- 172.21.15.87:0/1794295637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 msgr2=0x56067b8af5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b8af5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 -- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 msgr2=0x56067b83e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 0x56067b83e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 -- 172.21.15.87:0/1794295637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56067b8afb30 con 0x56067b82d010 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0691340640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 0x56067b83e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0691b41640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b8af5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0690b3f640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f067c009cf0 tx=0x7f067c014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8308db6d68da8675 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c004250 con 0x56067b82d010 2024-04-22T13:39:04.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8455+0+0 (secure 0 0 0) 0x7f067c0043f0 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06840099c0 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f067c02c050 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 10 client.17658 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56067b8b0d40 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06840099c0 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f067c0039e0 con 0x56067b82d010 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 10 client.17658 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:04.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 10 client.17658 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 20 client.17658 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b8b1560 0x56067b8b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 4 client.17658 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.2, "await": true}] 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.762+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.2, "await": true}) -- 0x56067b798980 con 0x56067b8b1560 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.763+0000 7f0691340640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b8b1560 0x56067b8b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.763+0000 7f0674ff9640 10 client.17658 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:04.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.763+0000 7f0691340640 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b8b1560 0x56067b8b3980 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e70e02f4f59df94 server_cookie=68863d2b681ca489 in_seq=0 out_seq=0 2024-04-22T13:39:04.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0674ff9640 1 -- 172.21.15.87:0/1794295637 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56067b798980 con 0x56067b8b1560 2024-04-22T13:39:04.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0674ff9640 10 client.17658 handle_command_reply: tid=0 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 client.17658 shutdown 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 msgr2=0x56067b83e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f067c009cf0 tx=0x7f067c014840 comp rx=0 tx=0).stop 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 shutdown_connections 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56067b8b1560 0x56067b8b3980 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56067b83a2d0 0x56067b8af5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56067b82d010 0x56067b83e540 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0674ff9640 0 client.17658 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 --2- 172.21.15.87:0/1794295637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56067b82c6f0 0x56067b83e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 >> 172.21.15.87:0/1794295637 conn(0x56067b8270f0 msgr2=0x56067b837300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 shutdown_connections 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.778+0000 7f0696196740 1 -- 172.21.15.87:0/1794295637 wait complete. 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f0696196740 20 client.17658 trim_cache size 0 max 16384 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 26, 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "12b0a2f9": { 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 26, 2024-04-22T13:39:04.778 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 74.2, 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:04.779 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:04.780 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 msgr2=0x7f06780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 0x7f06780a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0684022000 tx=0x7f068401f000 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 msgr2=0x7f068c19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0688005940 tx=0x7f068802c5f0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 shutdown_connections 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f06780a3d20 0x7f06780a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f068c198590 0x7f068c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f068c106190 0x7f068c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 --2- 172.21.15.87:0/3390984957 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f068c101110 0x7f068c19e3e0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 >> 172.21.15.87:0/3390984957 conn(0x7f068c077fb0 msgr2=0x7f068c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 shutdown_connections 2024-04-22T13:39:04.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:04.779+0000 7f06935a4640 1 -- 172.21.15.87:0/3390984957 wait complete. 2024-04-22T13:39:04.785 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 12b0a2f9, quiesce took 0.5 seconds. Will release after: 14.2 2024-04-22T13:39:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/696570982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:05.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/696570982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:05.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/696570982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:06.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:06 smithi120 ceph-mon[31101]: pgmap v223: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 MiB/s wr, 59 op/s 2024-04-22T13:39:06.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:06 smithi132 ceph-mon[31531]: pgmap v223: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 MiB/s wr, 59 op/s 2024-04-22T13:39:06.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:06 smithi087 ceph-mon[27999]: pgmap v223: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 MiB/s wr, 59 op/s 2024-04-22T13:39:08.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:08 smithi120 ceph-mon[31101]: pgmap v224: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-22T13:39:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:08 smithi132 ceph-mon[31531]: pgmap v224: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-22T13:39:08.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:08 smithi087 ceph-mon[27999]: pgmap v224: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-04-22T13:39:09.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:09 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:39:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:09 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:39:09.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:09 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:39:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:39:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:39:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:39:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:10 smithi120 ceph-mon[31101]: pgmap v225: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 330 KiB/s wr, 30 op/s 2024-04-22T13:39:10.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:39:10.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:39:10.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:39:10.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:10 smithi132 ceph-mon[31531]: pgmap v225: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 330 KiB/s wr, 30 op/s 2024-04-22T13:39:10.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:39:10.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:39:10.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:39:10.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:10 smithi087 ceph-mon[27999]: pgmap v225: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 330 KiB/s wr, 30 op/s 2024-04-22T13:39:12.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:12 smithi087 ceph-mon[27999]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T13:39:12.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:12 smithi087 ceph-mon[27999]: pgmap v227: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-04-22T13:39:12.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:12 smithi120 ceph-mon[31101]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T13:39:12.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:12 smithi120 ceph-mon[31101]: pgmap v227: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-04-22T13:39:12.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:12 smithi132 ceph-mon[31531]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T13:39:12.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:12 smithi132 ceph-mon[31531]: pgmap v227: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-04-22T13:39:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 -- 172.21.15.87:0/3365890103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 msgr2=0x7fe99c06add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/3365890103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c06add0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 -- 172.21.15.87:0/3365890103 shutdown_connections 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/3365890103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 0x7fe99c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/3365890103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c06b3a0 0x7fe99c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/3365890103 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c06add0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 -- 172.21.15.87:0/3365890103 >> 172.21.15.87:0/3365890103 conn(0x7fe99c076c50 msgr2=0x7fe99c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 -- 172.21.15.87:0/3365890103 shutdown_connections 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.302+0000 7fe9a29f4640 1 -- 172.21.15.87:0/3365890103 wait complete. 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 Processor -- start 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 -- start start 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 0x7fe99c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c101cf0 0x7fe99c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1121f0 con 0x7fe99c06a9f0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c112070 con 0x7fe99c101cf0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe9a29f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c112370 con 0x7fe99c107260 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55578/0 (socket says 172.21.15.87:55578) 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.303+0000 7fe99bfff640 1 -- 172.21.15.87:0/2564815647 learned_addr learned my addr 172.21.15.87:0/2564815647 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c101cf0 0x7fe99c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 0x7fe99c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 -- 172.21.15.87:0/2564815647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 msgr2=0x7fe99c1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 0x7fe99c1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 -- 172.21.15.87:0/2564815647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c101cf0 msgr2=0x7fe99c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c101cf0 0x7fe99c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 -- 172.21.15.87:0/2564815647 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99c1a6f80 con 0x7fe99c06a9f0 2024-04-22T13:39:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe98c005940 tx=0x7fe98c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9997fa640 1 -- 172.21.15.87:0/2564815647 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c043020 con 0x7fe99c06a9f0 2024-04-22T13:39:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9997fa640 1 -- 172.21.15.87:0/2564815647 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98c02cbe0 con 0x7fe99c06a9f0 2024-04-22T13:39:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99c1a7210 con 0x7fe99c06a9f0 2024-04-22T13:39:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9997fa640 1 -- 172.21.15.87:0/2564815647 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c003bf0 con 0x7fe99c06a9f0 2024-04-22T13:39:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.304+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe99c1a7760 con 0x7fe99c06a9f0 2024-04-22T13:39:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.306+0000 7fe9997fa640 1 -- 172.21.15.87:0/2564815647 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe98c046020 con 0x7fe99c06a9f0 2024-04-22T13:39:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 Processor -- start 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 -- start start 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9997fa640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 0x7fe9840a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ec8d0 0x557c7f4eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557c7f4508d0 con 0x557c7f4ebb60 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c7f3ede80 con 0x557c7f4f50e0 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557c7f44c890 con 0x557c7f4ec8d0 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9997fa640 1 -- 172.21.15.87:0/2564815647 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe98c0f2930 con 0x7fe99c06a9f0 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 0x7fe9840a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe99b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe99b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55582/0 (socket says 172.21.15.87:55582) 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe99b7fe640 1 -- 172.21.15.87:0/492620929 learned_addr learned my addr 172.21.15.87:0/492620929 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe99bfff640 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ec8d0 0x557c7f4eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.307+0000 7fe99b7fe640 1 -- 172.21.15.87:0/492620929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ec8d0 msgr2=0x557c7f4eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ec8d0 0x557c7f4eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe99b7fe640 1 -- 172.21.15.87:0/492620929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 msgr2=0x557c7f4f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe99b7fe640 1 -- 172.21.15.87:0/492620929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c7f40f320 con 0x557c7f4ebb60 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 0x7fe9840a62b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe994005fd0 tx=0x7fe994012ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe99000bc40 tx=0x7fe99000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15eed74a0f35718a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe9827fc640 1 -- 172.21.15.87:0/492620929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990015070 con 0x557c7f4ebb60 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe9827fc640 1 -- 172.21.15.87:0/492620929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9900118f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.308+0000 7fe9827fc640 1 -- 172.21.15.87:0/492620929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9900107f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- 172.21.15.87:0/492620929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 msgr2=0x557c7f4ebfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe99000bc40 tx=0x7fe99000f730 comp rx=0 tx=0).stop 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- 172.21.15.87:0/492620929 shutdown_connections 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4ebfe0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/492620929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ec8d0 0x557c7f4eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- 172.21.15.87:0/492620929 >> 172.21.15.87:0/492620929 conn(0x557c7f4b2c90 msgr2=0x557c7f4e6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- 172.21.15.87:0/492620929 shutdown_connections 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- 172.21.15.87:0/492620929 wait complete. 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 -- start start 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 Processor -- start 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.309+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557c7f4f7e10 con 0x557c7f4ec8d0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c7f3d8eb0 con 0x557c7f4f50e0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557c7f4f7f90 con 0x557c7f4ebb60 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 10 client.0 _ll_register_callbacks cb 0x557c7f405b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55598/0 (socket says 172.21.15.87:55598) 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 -- 172.21.15.87:0/2670244855 learned_addr learned my addr 172.21.15.87:0/2670244855 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 -- 172.21.15.87:0/2670244855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 msgr2=0x557c7f4f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 -- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 msgr2=0x557c7f4f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 -- 172.21.15.87:0/2670244855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c7f163210 con 0x557c7f4ec8d0 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe990006010 tx=0x7fe990011670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7a0ef3f0f303a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe97b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990028060 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe990011cb0 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9900118f0 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe990015050 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 10 client.17670 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.310+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557c7f22f620 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9900118f0 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe990004a90 con 0x557c7f4ec8d0 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 10 client.17670 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 10 client.17670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 20 client.17670 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f505030 0x557c7f507420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 4 client.17670 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c7f464880 con 0x557c7f505030 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f505030 0x557c7f507420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe97b7fe640 10 client.17670 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.311+0000 7fe99bfff640 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f505030 0x557c7f507420 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df00561ca22d309e server_cookie=95c16eed45affe4f in_seq=0 out_seq=0 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe97b7fe640 1 -- 172.21.15.87:0/2670244855 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c7f464880 con 0x557c7f505030 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe97b7fe640 10 client.17670 handle_command_reply: tid=0 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 client.17670 shutdown 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 msgr2=0x557c7f4f1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe990006010 tx=0x7fe990011670 comp rx=0 tx=0).stop 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 shutdown_connections 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f505030 0x557c7f507420 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4f50e0 0x557c7f4f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f1e40 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe97b7fe640 0 client.17670 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/2670244855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 >> 172.21.15.87:0/2670244855 conn(0x557c7f4b2c90 msgr2=0x557c7f4e85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 shutdown_connections 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.312+0000 7fe9a55ee740 1 -- 172.21.15.87:0/2670244855 wait complete. 2024-04-22T13:39:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.313+0000 7fe9a55ee740 20 client.17670 trim_cache size 0 max 16384 2024-04-22T13:39:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 Processor -- start 2024-04-22T13:39:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 -- start start 2024-04-22T13:39:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4ecec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1b25b0 con 0x557c7f4ebb60 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1afe80 con 0x557c7f4ec8d0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1adcf0 con 0x557c7f4f50e0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55610/0 (socket says 172.21.15.87:55610) 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.327+0000 7fe99bfff640 1 -- 172.21.15.87:0/257512202 learned_addr learned my addr 172.21.15.87:0/257512202 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 -- 172.21.15.87:0/257512202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 msgr2=0x557c7f4ecec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4ecec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4ecec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 -- 172.21.15.87:0/257512202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 msgr2=0x557c7f4f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 -- 172.21.15.87:0/257512202 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c7f4f2a50 con 0x557c7f4ebb60 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe99bfff640 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0bae50 tx=0x7fe98c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9529e99a10a9b910 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe981ffb640 1 -- 172.21.15.87:0/257512202 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0bf0f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe981ffb640 1 -- 172.21.15.87:0/257512202 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98c0be9f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.328+0000 7fe981ffb640 1 -- 172.21.15.87:0/257512202 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0054d0 con 0x557c7f4ebb60 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- 172.21.15.87:0/257512202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 msgr2=0x557c7f4f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0bae50 tx=0x7fe98c0036f0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- 172.21.15.87:0/257512202 shutdown_connections 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4ecec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/257512202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f4f4570 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- 172.21.15.87:0/257512202 >> 172.21.15.87:0/257512202 conn(0x557c7f4b2c90 msgr2=0x557c7f4f0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- 172.21.15.87:0/257512202 shutdown_connections 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- 172.21.15.87:0/257512202 wait complete. 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 -- start start 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.329+0000 7fe9a55ee740 1 Processor -- start 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1ae340 con 0x557c7f4ebb60 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1af7b0 con 0x557c7f4ec8d0 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c1aea10 con 0x557c7f4f50e0 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 10 client.0 _ll_register_callbacks cb 0x7fe99000ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55616/0 (socket says 172.21.15.87:55616) 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 -- 172.21.15.87:0/3828722770 learned_addr learned my addr 172.21.15.87:0/3828722770 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a0f91640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 -- 172.21.15.87:0/3828722770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 msgr2=0x557c7f4eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 -- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 msgr2=0x557c7f4ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 -- 172.21.15.87:0/3828722770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c7f4eae30 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe99bfff640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0f0250 tx=0x7fe98c0453f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e08ca0f8df1ad9f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe97b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0d9060 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe98c0054d0 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe98c0be9f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe98c0bf0f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 10 client.17676 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557c7f56ad50 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.330+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98c0be9f0 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe98c0454a0 con 0x557c7f4ebb60 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 10 client.17676 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 10 client.17676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 20 client.17676 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f56b360 0x557c7f56d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 4 client.17676 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:39:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557c7f3f03d0 con 0x557c7f56b360 2024-04-22T13:39:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f56b360 0x557c7f56d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe97b7fe640 10 client.17676 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.331+0000 7fe99b7fe640 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f56b360 0x557c7f56d770 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b3d6271a920e3ad server_cookie=c6efee3ec848361d in_seq=0 out_seq=0 2024-04-22T13:39:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe97b7fe640 1 -- 172.21.15.87:0/3828722770 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557c7f3f03d0 con 0x557c7f56b360 2024-04-22T13:39:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe97b7fe640 10 client.17676 handle_command_reply: tid=0 2024-04-22T13:39:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 client.17676 shutdown 2024-04-22T13:39:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 msgr2=0x557c7f50a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0f0250 tx=0x7fe98c0453f0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 shutdown_connections 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557c7f56b360 0x557c7f56d770 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557c7f4f50e0 0x557c7f4eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe97b7fe640 0 client.17676 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c7f4ec8d0 0x557c7f4ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 --2- 172.21.15.87:0/3828722770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557c7f4ebb60 0x557c7f50a640 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 >> 172.21.15.87:0/3828722770 conn(0x557c7f4b2c90 msgr2=0x557c7f4ecec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 shutdown_connections 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 1 -- 172.21.15.87:0/3828722770 wait complete. 2024-04-22T13:39:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.339+0000 7fe9a55ee740 20 client.17676 trim_cache size 0 max 16384 2024-04-22T13:39:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:39:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:39:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:39:13.337 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 msgr2=0x7fe9840a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 0x7fe9840a62b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe994005fd0 tx=0x7fe994012ab0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 msgr2=0x7fe99c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe98c005940 tx=0x7fe98c02c7e0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 shutdown_connections 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9840a3df0 0x7fe9840a62b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c101cf0 0x7fe99c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe99c107260 0x7fe99c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 --2- 172.21.15.87:0/2564815647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe99c06a9f0 0x7fe99c101270 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 >> 172.21.15.87:0/2564815647 conn(0x7fe99c076c50 msgr2=0x7fe99c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 shutdown_connections 2024-04-22T13:39:13.338 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:13.340+0000 7fe9a29f4640 1 -- 172.21.15.87:0/2564815647 wait complete. 2024-04-22T13:39:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:14 smithi120 ceph-mon[31101]: pgmap v228: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T13:39:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:14 smithi132 ceph-mon[31531]: pgmap v228: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T13:39:14.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:14 smithi087 ceph-mon[27999]: pgmap v228: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T13:39:16.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:16 smithi120 ceph-mon[31101]: pgmap v229: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-22T13:39:16.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:16 smithi132 ceph-mon[31531]: pgmap v229: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-22T13:39:16.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:16 smithi087 ceph-mon[27999]: pgmap v229: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-22T13:39:18.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:18 smithi120 ceph-mon[31101]: pgmap v230: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:18.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:18 smithi132 ceph-mon[31531]: pgmap v230: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:18.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:18 smithi087 ceph-mon[27999]: pgmap v230: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:18.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:39:19.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 -- 172.21.15.87:0/4054423278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f80102b30 msgr2=0x7f2f80106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/4054423278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f80102b30 0x7f2f80106b70 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009900 tx=0x7f2f7c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 -- 172.21.15.87:0/4054423278 shutdown_connections 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/4054423278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 0x7f2f8010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/4054423278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f80102b30 0x7f2f80106b70 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/4054423278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 0x7f2f80102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 -- 172.21.15.87:0/4054423278 >> 172.21.15.87:0/4054423278 conn(0x7f2f80079350 msgr2=0x7f2f80079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 -- 172.21.15.87:0/4054423278 shutdown_connections 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.102+0000 7f2f87a4a640 1 -- 172.21.15.87:0/4054423278 wait complete. 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 Processor -- start 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 -- start start 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 0x7f2f801986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 0x7f2f80198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80112170 con 0x7f2f8019cf90 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80111ff0 con 0x7f2f80102180 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f87a4a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2f801122f0 con 0x7f2f80107250 2024-04-22T13:39:19.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55626/0 (socket says 172.21.15.87:55626) 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 -- 172.21.15.87:0/570825827 learned_addr learned my addr 172.21.15.87:0/570825827 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f857e6640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 0x7f2f801986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f84fe5640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 0x7f2f80198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 -- 172.21.15.87:0/570825827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 msgr2=0x7f2f80198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 0x7f2f80198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 -- 172.21.15.87:0/570825827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 msgr2=0x7f2f801986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 0x7f2f801986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.103+0000 7f2f85fe7640 1 -- 172.21.15.87:0/570825827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8019da80 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f85fe7640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800cce0 tx=0x7f2f7800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7800f450 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f7800faa0 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8019dd70 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f78018b70 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.104+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f801a5510 con 0x7f2f8019cf90 2024-04-22T13:39:19.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.105+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f80068de0 con 0x7f2f8019cf90 2024-04-22T13:39:19.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.106+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2f78018d10 con 0x7f2f8019cf90 2024-04-22T13:39:19.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.107+0000 7f2f767fc640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 0x7f2f6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.107+0000 7f2f857e6640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 0x7f2f6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.107+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7f2f780d0be0 con 0x7f2f8019cf90 2024-04-22T13:39:19.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.107+0000 7f2f857e6640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 0x7f2f6c0a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000bbe0 tx=0x7f2f7000b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.111+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f78014030 con 0x7f2f8019cf90 2024-04-22T13:39:19.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.327+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f8010a4e0 con 0x7f2f8019cf90 2024-04-22T13:39:19.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.328+0000 7f2f767fc640 1 -- 172.21.15.87:0/570825827 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f2f7809d290 con 0x7f2f8019cf90 2024-04-22T13:39:19.324 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:39:19.324 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:39:19.324 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 msgr2=0x7f2f6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 0x7f2f6c0a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000bbe0 tx=0x7f2f7000b390 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 msgr2=0x7f2f8019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800cce0 tx=0x7f2f7800a620 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 shutdown_connections 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2f6c0a3d20 0x7f2f6c0a61e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2f8019cf90 0x7f2f8019d440 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f80107250 0x7f2f80198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 --2- 172.21.15.87:0/570825827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f80102180 0x7f2f801986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 >> 172.21.15.87:0/570825827 conn(0x7f2f80079350 msgr2=0x7f2f80100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 shutdown_connections 2024-04-22T13:39:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.330+0000 7f2f87a4a640 1 -- 172.21.15.87:0/570825827 wait complete. 2024-04-22T13:39:19.337 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 12b0a2f9 --release --await' 2024-04-22T13:39:19.337 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 12b0a2f9 --release --await 2024-04-22T13:39:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 -- 172.21.15.87:0/2190842140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103230 msgr2=0x7fe6d4103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 --2- 172.21.15.87:0/2190842140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103230 0x7fe6d4103610 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009900 tx=0x7fe6d002bcd0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 -- 172.21.15.87:0/2190842140 shutdown_connections 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 --2- 172.21.15.87:0/2190842140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d410c600 0x7fe6d410ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 --2- 172.21.15.87:0/2190842140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4103b50 0x7fe6d410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 --2- 172.21.15.87:0/2190842140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103230 0x7fe6d4103610 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 -- 172.21.15.87:0/2190842140 >> 172.21.15.87:0/2190842140 conn(0x7fe6d4077f80 msgr2=0x7fe6d4078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 -- 172.21.15.87:0/2190842140 shutdown_connections 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.447+0000 7fe6db443640 1 -- 172.21.15.87:0/2190842140 wait complete. 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 Processor -- start 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 -- start start 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4103230 0x7fe6d419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d410c600 0x7fe6d41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4112230 con 0x7fe6d4103b50 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41120b0 con 0x7fe6d4103230 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6db443640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41123b0 con 0x7fe6d410c600 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55634/0 (socket says 172.21.15.87:55634) 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6d89de640 1 -- 172.21.15.87:0/3879085884 learned_addr learned my addr 172.21.15.87:0/3879085884 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.448+0000 7fe6d91df640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4103230 0x7fe6d419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d410c600 0x7fe6d41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 -- 172.21.15.87:0/3879085884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d410c600 msgr2=0x7fe6d41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d410c600 0x7fe6d41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 -- 172.21.15.87:0/3879085884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4103230 msgr2=0x7fe6d419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4103230 0x7fe6d419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 -- 172.21.15.87:0/3879085884 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d4198d50 con 0x7fe6d4103b50 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400d950 tx=0x7fe6c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6ca7fc640 1 -- 172.21.15.87:0/3879085884 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4017030 con 0x7fe6d4103b50 2024-04-22T13:39:19.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6ca7fc640 1 -- 172.21.15.87:0/3879085884 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c4004020 con 0x7fe6d4103b50 2024-04-22T13:39:19.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d4199040 con 0x7fe6d4103b50 2024-04-22T13:39:19.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6ca7fc640 1 -- 172.21.15.87:0/3879085884 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40047a0 con 0x7fe6d4103b50 2024-04-22T13:39:19.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.449+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d41a52e0 con 0x7fe6d4103b50 2024-04-22T13:39:19.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.451+0000 7fe6ca7fc640 1 -- 172.21.15.87:0/3879085884 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe6c4009a30 con 0x7fe6d4103b50 2024-04-22T13:39:19.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 Processor -- start 2024-04-22T13:39:19.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 -- start start 2024-04-22T13:39:19.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01310 0x555968b016f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555968a66500 con 0x555968b0a810 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555968a0c6a0 con 0x555968b01c30 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6de038740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555968a624c0 con 0x555968b01310 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6ca7fc640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55640/0 (socket says 172.21.15.87:55640) 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d89de640 1 -- 172.21.15.87:0/3893310776 learned_addr learned my addr 172.21.15.87:0/3893310776 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d91df640 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01310 0x555968b016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6ca7fc640 1 -- 172.21.15.87:0/3879085884 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe6c40d0250 con 0x7fe6d4103b50 2024-04-22T13:39:19.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d89de640 1 -- 172.21.15.87:0/3893310776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01310 msgr2=0x555968b016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.452+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01310 0x555968b016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6d89de640 1 -- 172.21.15.87:0/3893310776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 msgr2=0x555968b0a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6d89de640 1 -- 172.21.15.87:0/3893310776 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555968a2b5e0 con 0x555968b0a810 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009370 tx=0x7fe6cc009f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005fd0 tx=0x7fe6c4004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a92605935c8221 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3893310776 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c409c900 con 0x555968b0a810 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3893310776 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c409c200 con 0x555968b0a810 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.453+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3893310776 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40cdc30 con 0x555968b0a810 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- 172.21.15.87:0/3893310776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 msgr2=0x555968b0cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4005fd0 tx=0x7fe6c4004550 comp rx=0 tx=0).stop 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- 172.21.15.87:0/3893310776 shutdown_connections 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b0cc00 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- 172.21.15.87:0/3893310776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01310 0x555968b016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- 172.21.15.87:0/3893310776 >> 172.21.15.87:0/3893310776 conn(0x555968afba40 msgr2=0x555968afbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- 172.21.15.87:0/3893310776 shutdown_connections 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- 172.21.15.87:0/3893310776 wait complete. 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- start start 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 Processor -- start 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b0a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b07d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x555968b0d5a0 con 0x555968b01c30 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555968a1fa60 con 0x555968b01310 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555968b0d720 con 0x555968b0a810 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6de038740 10 client.0 _ll_register_callbacks cb 0x555968778110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:19.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6d89de640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55656/0 (socket says 172.21.15.87:55656) 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.454+0000 7fe6d89de640 1 -- 172.21.15.87:0/4280399406 learned_addr learned my addr 172.21.15.87:0/4280399406 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d91df640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b0a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 -- 172.21.15.87:0/4280399406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 msgr2=0x555968b07d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b07d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 -- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 msgr2=0x555968b0a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b0a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 -- 172.21.15.87:0/4280399406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555968778920 con 0x555968b01c30 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6d89de640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40cda60 tx=0x7fe6c4098510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06084aadfcda322 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4009e20 con 0x555968b01c30 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe6c40cdc30 con 0x555968b01c30 2024-04-22T13:39:19.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c409c200 con 0x555968b01c30 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe6c40a05b0 con 0x555968b01c30 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 10 client.17694 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555968844d30 con 0x555968b01c30 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c409c200 con 0x555968b01c30 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6c40ac5f0 con 0x555968b01c30 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 10 client.17694 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 10 client.17694 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 20 client.17694 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.455+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b1a740 0x555968b1cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.456+0000 7fe6de038740 4 client.17694 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.456+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555968a39a10 con 0x555968b1a740 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.456+0000 7fe6d91df640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b1a740 0x555968b1cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.456+0000 7fe6d91df640 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b1a740 0x555968b1cb30 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c66e277bba0ef83a server_cookie=8e083b9144dec825 in_seq=0 out_seq=0 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.456+0000 7fe6bcff9640 10 client.17694 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:19.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6bcff9640 1 -- 172.21.15.87:0/4280399406 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555968a39a10 con 0x555968b1a740 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6bcff9640 10 client.17694 handle_command_reply: tid=0 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 client.17694 shutdown 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 msgr2=0x555968b075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40cda60 tx=0x7fe6c4098510 comp rx=0 tx=0).stop 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 shutdown_connections 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b1a740 0x555968b1cb30 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b07d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01c30 0x555968b075d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6bcff9640 0 client.17694 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 --2- 172.21.15.87:0/4280399406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b0a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 >> 172.21.15.87:0/4280399406 conn(0x555968afba40 msgr2=0x555968afd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 shutdown_connections 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 1 -- 172.21.15.87:0/4280399406 wait complete. 2024-04-22T13:39:19.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.457+0000 7fe6de038740 20 client.17694 trim_cache size 0 max 16384 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 Processor -- start 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 -- start start 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b12d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01c30 0x555968b220d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41b0400 con 0x555968b0a810 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41adcd0 con 0x555968b01310 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.474+0000 7fe6de038740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41abb40 con 0x555968b01c30 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55662/0 (socket says 172.21.15.87:55662) 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 -- 172.21.15.87:0/487038506 learned_addr learned my addr 172.21.15.87:0/487038506 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d91df640 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b12d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d89de640 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01c30 0x555968b220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 -- 172.21.15.87:0/487038506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01c30 msgr2=0x555968b220d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01c30 0x555968b220d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 -- 172.21.15.87:0/487038506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 msgr2=0x555968b12d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b12d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 -- 172.21.15.87:0/487038506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555968b08080 con 0x555968b0a810 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6d99e0640 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc010620 tx=0x7fe6cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc8cd72112e73dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6beffd640 1 -- 172.21.15.87:0/487038506 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc01d700 con 0x555968b0a810 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6beffd640 1 -- 172.21.15.87:0/487038506 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc01dd50 con 0x555968b0a810 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.475+0000 7fe6beffd640 1 -- 172.21.15.87:0/487038506 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc026390 con 0x555968b0a810 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- 172.21.15.87:0/487038506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 msgr2=0x555968b22610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc010620 tx=0x7fe6cc018000 comp rx=0 tx=0).stop 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- 172.21.15.87:0/487038506 shutdown_connections 2024-04-22T13:39:19.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b0a810 0x555968b22610 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b01c30 0x555968b220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- 172.21.15.87:0/487038506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01310 0x555968b12d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- 172.21.15.87:0/487038506 >> 172.21.15.87:0/487038506 conn(0x555968afba40 msgr2=0x555968afab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- 172.21.15.87:0/487038506 shutdown_connections 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- 172.21.15.87:0/487038506 wait complete. 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- start start 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 Processor -- start 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b0b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ac190 con 0x555968b01310 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ad600 con 0x555968b01c30 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.476+0000 7fe6de038740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ac860 con 0x555968b0a810 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6de038740 10 client.0 _ll_register_callbacks cb 0x7fe6d0005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55676/0 (socket says 172.21.15.87:55676) 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 -- 172.21.15.87:0/3155707930 learned_addr learned my addr 172.21.15.87:0/3155707930 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 -- 172.21.15.87:0/3155707930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 msgr2=0x555968b0b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b0b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 -- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 msgr2=0x555968b0b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 -- 172.21.15.87:0/3155707930 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555968b84b20 con 0x555968b01310 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6d91df640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fe6d002c210 tx=0x7fe6d0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba28025648c5bca5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d004f060 con 0x555968b01310 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8499+0+0 (secure 0 0 0) 0x7fe6d003a040 con 0x555968b01310 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6cc01dd50 con 0x555968b01310 2024-04-22T13:39:19.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe6d002ca90 con 0x555968b01310 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6de038740 10 client.17700 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555968b85e40 con 0x555968b01310 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.477+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc01dd50 con 0x555968b01310 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6d0004850 con 0x555968b01310 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 10 client.17700 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 10 client.17700 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 20 client.17700 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b865f0 0x555968b88a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 4 client.17700 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "12b0a2f9", "release": true, "await": true}] 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "12b0a2f9", "release": true, "await": true}) -- 0x555968a73f20 con 0x555968b865f0 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b865f0 0x555968b88a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6bcff9640 10 client.17700 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.478+0000 7fe6d89de640 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b865f0 0x555968b88a50 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b6b129d77b21e6 server_cookie=88bb781f74dae3 in_seq=0 out_seq=0 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6bcff9640 1 -- 172.21.15.87:0/3155707930 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555968a73f20 con 0x555968b865f0 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6bcff9640 10 client.17700 handle_command_reply: tid=0 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 client.17700 shutdown 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 msgr2=0x555968b0ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fe6d002c210 tx=0x7fe6d0004430 comp rx=0 tx=0).stop 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 shutdown_connections 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x555968b865f0 0x555968b88a50 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555968b0a810 0x555968b0b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555968b01c30 0x555968b0b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6bcff9640 0 client.17700 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 --2- 172.21.15.87:0/3155707930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x555968b01310 0x555968b0ad10 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 >> 172.21.15.87:0/3155707930 conn(0x555968afba40 msgr2=0x555968afcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 shutdown_connections 2024-04-22T13:39:19.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.481+0000 7fe6de038740 1 -- 172.21.15.87:0/3155707930 wait complete. 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6de038740 20 client.17700 trim_cache size 0 max 16384 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 28, 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "12b0a2f9": { 2024-04-22T13:39:19.482 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 28, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 74.2, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:19.483 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:19.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:19.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:19.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:19.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:19.484 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 msgr2=0x7fe6c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009370 tx=0x7fe6cc009f90 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 msgr2=0x7fe6d419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400d950 tx=0x7fe6c400de10 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 shutdown_connections 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d410c600 0x7fe6d41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4103b50 0x7fe6d419e960 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 --2- 172.21.15.87:0/3879085884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4103230 0x7fe6d419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 >> 172.21.15.87:0/3879085884 conn(0x7fe6d4077f80 msgr2=0x7fe6d40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 shutdown_connections 2024-04-22T13:39:19.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:19.482+0000 7fe6db443640 1 -- 172.21.15.87:0/3879085884 wait complete. 2024-04-22T13:39:19.489 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 12b0a2f9, seconds elapsed: 15.2 2024-04-22T13:39:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:19 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/570825827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:19 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/570825827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:19.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:19 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/570825827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:20.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:20 smithi120 ceph-mon[31101]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T13:39:20.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:20 smithi120 ceph-mon[31101]: pgmap v232: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-22T13:39:20.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:20 smithi132 ceph-mon[31531]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T13:39:20.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:20 smithi132 ceph-mon[31531]: pgmap v232: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-22T13:39:20.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:20 smithi087 ceph-mon[27999]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T13:39:20.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:20 smithi087 ceph-mon[27999]: pgmap v232: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-22T13:39:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:22 smithi120 ceph-mon[31101]: pgmap v233: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:22.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:22 smithi132 ceph-mon[31531]: pgmap v233: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:22.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:22 smithi087 ceph-mon[27999]: pgmap v233: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T13:39:24.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:24 smithi120 ceph-mon[31101]: pgmap v234: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:24.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:24 smithi132 ceph-mon[31531]: pgmap v234: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:24.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:24 smithi087 ceph-mon[27999]: pgmap v234: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 -- 172.21.15.87:0/325594617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 msgr2=0x7ff81c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 --2- 172.21.15.87:0/325594617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c1014d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009ca0 tx=0x7ff80c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 -- 172.21.15.87:0/325594617 shutdown_connections 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 --2- 172.21.15.87:0/325594617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 0x7ff81c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 --2- 172.21.15.87:0/325594617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c101aa0 0x7ff81c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 --2- 172.21.15.87:0/325594617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c1014d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 -- 172.21.15.87:0/325594617 >> 172.21.15.87:0/325594617 conn(0x7ff81c077fd0 msgr2=0x7ff81c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 -- 172.21.15.87:0/325594617 shutdown_connections 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 -- 172.21.15.87:0/325594617 wait complete. 2024-04-22T13:39:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.436+0000 7ff8218b3640 1 Processor -- start 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 -- start start 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 0x7ff81c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c19cfd0 0x7ff81c19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1121f0 con 0x7ff81c1010f0 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c112070 con 0x7ff81c19cfd0 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8218b3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c112370 con 0x7ff81c106170 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47318/0 (socket says 172.21.15.87:47318) 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 -- 172.21.15.87:0/1991147322 learned_addr learned my addr 172.21.15.87:0/1991147322 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 0x7ff81c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c19cfd0 0x7ff81c19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 -- 172.21.15.87:0/1991147322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 msgr2=0x7ff81c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 0x7ff81c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 -- 172.21.15.87:0/1991147322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c19cfd0 msgr2=0x7ff81c19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c19cfd0 0x7ff81c19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 -- 172.21.15.87:0/1991147322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c19dac0 con 0x7ff81c1010f0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff81affd640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005e30 tx=0x7ff80c034780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.437+0000 7ff8208b1640 1 -- 172.21.15.87:0/1991147322 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c039040 con 0x7ff81c1010f0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.438+0000 7ff8208b1640 1 -- 172.21.15.87:0/1991147322 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff80c044070 con 0x7ff81c1010f0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.438+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c19dd50 con 0x7ff81c1010f0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.438+0000 7ff8208b1640 1 -- 172.21.15.87:0/1991147322 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c003e60 con 0x7ff81c1010f0 2024-04-22T13:39:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.438+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c1a54b0 con 0x7ff81c1010f0 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff8208b1640 1 -- 172.21.15.87:0/1991147322 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff80c0324c0 con 0x7ff81c1010f0 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 Processor -- start 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 -- start start 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2005620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe2005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557fe1f6a8d0 con 0x557fe20134f0 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557fe1f07e80 con 0x557fe20068d0 2024-04-22T13:39:26.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff82457d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557fe1f66890 con 0x557fe2005b60 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.440+0000 7ff8208b1640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 0x7ff8080a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47334/0 (socket says 172.21.15.87:47334) 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 -- 172.21.15.87:0/845306959 learned_addr learned my addr 172.21.15.87:0/845306959 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 0x7ff8080a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff8208b1640 1 -- 172.21.15.87:0/1991147322 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7ff80c0f2a80 con 0x7ff81c1010f0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81affd640 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe2005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2005620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 -- 172.21.15.87:0/845306959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 msgr2=0x557fe2005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe2005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 -- 172.21.15.87:0/845306959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 msgr2=0x557fe2005620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2005620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 -- 172.21.15.87:0/845306959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe1f29320 con 0x557fe20134f0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff80400dc50 tx=0x7ff80400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd277ac2616af51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.444 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 0x7ff8080a6280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff814005d50 tx=0x7ff814005dd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff811ffb640 1 -- 172.21.15.87:0/845306959 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804015070 con 0x557fe20134f0 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff811ffb640 1 -- 172.21.15.87:0/845306959 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff80400de80 con 0x557fe20134f0 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.441+0000 7ff811ffb640 1 -- 172.21.15.87:0/845306959 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804011440 con 0x557fe20134f0 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- 172.21.15.87:0/845306959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 msgr2=0x557fe20158e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff80400dc50 tx=0x7ff80400f730 comp rx=0 tx=0).stop 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- 172.21.15.87:0/845306959 shutdown_connections 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe20158e0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe2005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 --2- 172.21.15.87:0/845306959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2005620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- 172.21.15.87:0/845306959 >> 172.21.15.87:0/845306959 conn(0x557fe1fccc90 msgr2=0x557fe2000850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- 172.21.15.87:0/845306959 shutdown_connections 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- 172.21.15.87:0/845306959 wait complete. 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.442+0000 7ff82457d740 1 -- start start 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 Processor -- start 2024-04-22T13:39:26.445 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2010220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 0x557fe20107c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557fe2016220 con 0x557fe2005b60 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557fe1ef2eb0 con 0x557fe20068d0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557fe20163a0 con 0x557fe20134f0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff82457d740 10 client.0 _ll_register_callbacks cb 0x557fe1f1fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47344/0 (socket says 172.21.15.87:47344) 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 -- 172.21.15.87:0/4260692220 learned_addr learned my addr 172.21.15.87:0/4260692220 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2010220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 0x557fe20107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 -- 172.21.15.87:0/4260692220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 msgr2=0x557fe20107c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 0x557fe20107c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.446 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 -- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 msgr2=0x557fe2010220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2010220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81affd640 1 -- 172.21.15.87:0/4260692220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe1c7d210 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.443+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 0x557fe20107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff81affd640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0bafd0 tx=0x7ff80c0be770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f42c49713e87724 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0d8060 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7ff80c0bb1e0 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557fe1f7e880 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff80c035df0 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 10 client.17712 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557fe1d49620 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557fe1f7e880 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff80c0d9020 con 0x557fe2005b60 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 10 client.17712 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 10 client.17712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:26.447 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 20 client.17712 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2023590 0x557fe20259e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 4 client.17712 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.444+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fe1f0a3d0 con 0x557fe2023590 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.445+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2023590 0x557fe20259e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.445+0000 7ff7faffd640 10 client.17712 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.445+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2023590 0x557fe20259e0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7d1ecbc2eeb070 server_cookie=67cbaf61bba14965 in_seq=0 out_seq=0 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff7faffd640 1 -- 172.21.15.87:0/4260692220 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fe1f0a3d0 con 0x557fe2023590 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff7faffd640 10 client.17712 handle_command_reply: tid=0 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 client.17712 shutdown 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 msgr2=0x557fe2012e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0bafd0 tx=0x7ff80c0be770 comp rx=0 tx=0).stop 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 shutdown_connections 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2023590 0x557fe20259e0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20134f0 0x557fe20107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe2010220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.448 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 --2- 172.21.15.87:0/4260692220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe2005b60 0x557fe2012e90 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff7faffd640 0 client.17712 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:26.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 >> 172.21.15.87:0/4260692220 conn(0x557fe1fccc90 msgr2=0x557fe20025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 shutdown_connections 2024-04-22T13:39:26.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 1 -- 172.21.15.87:0/4260692220 wait complete. 2024-04-22T13:39:26.449 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.446+0000 7ff82457d740 20 client.17712 trim_cache size 0 max 16384 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 Processor -- start 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 -- start start 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 0x557fe201f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20068d0 0x557fe2012950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1adfa0 con 0x557fe20134f0 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1ab870 con 0x557fe2005b60 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff82457d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1a96e0 con 0x557fe20068d0 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47354/0 (socket says 172.21.15.87:47354) 2024-04-22T13:39:26.462 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 -- 172.21.15.87:0/3435345765 learned_addr learned my addr 172.21.15.87:0/3435345765 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81affd640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 0x557fe201f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20068d0 0x557fe2012950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 -- 172.21.15.87:0/3435345765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20068d0 msgr2=0x557fe2012950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20068d0 0x557fe2012950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 -- 172.21.15.87:0/3435345765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 msgr2=0x557fe201f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 0x557fe201f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81b7fe640 1 -- 172.21.15.87:0/3435345765 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe2016100 con 0x557fe20134f0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.461+0000 7ff81affd640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 0x557fe201f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff81400bdb0 tx=0x7ff814005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad06e2ba0a057670 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff8117fa640 1 -- 172.21.15.87:0/3435345765 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814024070 con 0x557fe20134f0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff8117fa640 1 -- 172.21.15.87:0/3435345765 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8140054d0 con 0x557fe20134f0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff8117fa640 1 -- 172.21.15.87:0/3435345765 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140206c0 con 0x557fe20134f0 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 -- 172.21.15.87:0/3435345765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 msgr2=0x557fe2012e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff81400bdb0 tx=0x7ff814005ce0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 -- 172.21.15.87:0/3435345765 shutdown_connections 2024-04-22T13:39:26.463 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe2012e90 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe20068d0 0x557fe2012950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 --2- 172.21.15.87:0/3435345765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe2005b60 0x557fe201f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 -- 172.21.15.87:0/3435345765 >> 172.21.15.87:0/3435345765 conn(0x557fe1fccc90 msgr2=0x557fe20156b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 -- 172.21.15.87:0/3435345765 shutdown_connections 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.462+0000 7ff82457d740 1 -- 172.21.15.87:0/3435345765 wait complete. 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 -- start start 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 Processor -- start 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe200c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe200cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1a9d30 con 0x557fe20134f0 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1ab1a0 con 0x557fe20068d0 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c1aa400 con 0x557fe2005b60 2024-04-22T13:39:26.464 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff82457d740 10 client.0 _ll_register_callbacks cb 0x7ff80c0baee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff81b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff81b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47358/0 (socket says 172.21.15.87:47358) 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff81b7fe640 1 -- 172.21.15.87:0/2272283980 learned_addr learned my addr 172.21.15.87:0/2272283980 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.463+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe200cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81affd640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe200c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 -- 172.21.15.87:0/2272283980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 msgr2=0x557fe200c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe200c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 -- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 msgr2=0x557fe200cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe200cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 -- 172.21.15.87:0/2272283980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe2004e30 con 0x557fe20134f0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81a7fc640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe200cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff81b7fe640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff81400c010 tx=0x7ff8140048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0f4893ee901bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814035060 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8140054d0 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7ff814020460 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff814024040 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 10 client.17718 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557fe201b840 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8140054d0 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff81403a610 con 0x557fe20134f0 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 10 client.17718 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 10 client.17718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 20 client.17718 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2089b30 0x557fe208bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 4 client.17718 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:39:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.464+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557fe1f782f0 con 0x557fe2089b30 2024-04-22T13:39:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.465+0000 7ff81affd640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2089b30 0x557fe208bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.465+0000 7ff7faffd640 10 client.17718 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.465+0000 7ff81affd640 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2089b30 0x557fe208bf20 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72619b5182b357a1 server_cookie=46890adc842e0d3 in_seq=0 out_seq=0 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff7faffd640 1 -- 172.21.15.87:0/2272283980 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557fe1f782f0 con 0x557fe2089b30 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff7faffd640 10 client.17718 handle_command_reply: tid=0 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 client.17718 shutdown 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 msgr2=0x557fe200d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff81400c010 tx=0x7ff8140048f0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 shutdown_connections 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557fe2089b30 0x557fe208bf20 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fe20134f0 0x557fe200d010 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fe20068d0 0x557fe200cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 --2- 172.21.15.87:0/2272283980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fe2005b60 0x557fe200c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff7faffd640 0 client.17718 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 >> 172.21.15.87:0/2272283980 conn(0x557fe1fccc90 msgr2=0x557fe200ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 shutdown_connections 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 1 -- 172.21.15.87:0/2272283980 wait complete. 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.471+0000 7ff82457d740 20 client.17718 trim_cache size 0 max 16384 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:39:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 msgr2=0x7ff8080a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 0x7ff8080a6280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff814005d50 tx=0x7ff814005dd0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 msgr2=0x7ff81c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005e30 tx=0x7ff80c034780 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 shutdown_connections 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8080a3dc0 0x7ff8080a6280 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff81c19cfd0 0x7ff81c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff81c106170 0x7ff81c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 --2- 172.21.15.87:0/1991147322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff81c1010f0 0x7ff81c198710 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 >> 172.21.15.87:0/1991147322 conn(0x7ff81c077fd0 msgr2=0x7ff81c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 shutdown_connections 2024-04-22T13:39:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:26.472+0000 7ff8218b3640 1 -- 172.21.15.87:0/1991147322 wait complete. 2024-04-22T13:39:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:26 smithi120 ceph-mon[31101]: pgmap v235: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:26 smithi132 ceph-mon[31531]: pgmap v235: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:26.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:26 smithi087 ceph-mon[27999]: pgmap v235: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:39:27.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:39:27.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/161464894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b28102b50 msgr2=0x7f4b28106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/161464894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b28102b50 0x7f4b28106b90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009900 tx=0x7f4b1802bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/161464894 shutdown_connections 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/161464894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 0x7f4b2810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/161464894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b28102b50 0x7f4b28106b90 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/161464894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b281021a0 0x7f4b28102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/161464894 >> 172.21.15.87:0/161464894 conn(0x7f4b280793e0 msgr2=0x7f4b280797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/161464894 shutdown_connections 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.689+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/161464894 wait complete. 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 Processor -- start 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 -- start start 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 0x7f4b2819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b28198590 0x7f4b28198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28112190 con 0x7f4b281021a0 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28112010 con 0x7f4b28198590 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b2d8a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28112310 con 0x7f4b28107270 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b26ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b26ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47370/0 (socket says 172.21.15.87:47370) 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b26ffd640 1 -- 172.21.15.87:0/925968534 learned_addr learned my addr 172.21.15.87:0/925968534 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b267fc640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 0x7f4b2819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.690+0000 7f4b277fe640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b28198590 0x7f4b28198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 -- 172.21.15.87:0/925968534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 msgr2=0x7f4b2819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 0x7f4b2819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 -- 172.21.15.87:0/925968534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b28198590 msgr2=0x7f4b28198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b28198590 0x7f4b28198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 -- 172.21.15.87:0/925968534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b281990b0 con 0x7f4b281021a0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000e9f0 tx=0x7f4b1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b2c8a3640 1 -- 172.21.15.87:0/925968534 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000cd80 con 0x7f4b281021a0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b2c8a3640 1 -- 172.21.15.87:0/925968534 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b100108e0 con 0x7f4b281021a0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b281993a0 con 0x7f4b281021a0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b2c8a3640 1 -- 172.21.15.87:0/925968534 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10018980 con 0x7f4b281021a0 2024-04-22T13:39:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.691+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b281a55f0 con 0x7f4b281021a0 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.693+0000 7f4b2c8a3640 1 -- 172.21.15.87:0/925968534 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4b10010430 con 0x7f4b281021a0 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 Processor -- start 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 -- start start 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb05960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb0e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5578aba6a1e0 con 0x5578abb0ea80 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5578aba07390 con 0x5578abb05580 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b30572740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5578aba661a0 con 0x5578abb05ea0 2024-04-22T13:39:27.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47386/0 (socket says 172.21.15.87:47386) 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 -- 172.21.15.87:0/1018047349 learned_addr learned my addr 172.21.15.87:0/1018047349 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b2c8a3640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 0x7f4b140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb0e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b277fe640 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb05960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b277fe640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 0x7f4b140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 -- 172.21.15.87:0/1018047349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 msgr2=0x5578abb0e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb0e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 -- 172.21.15.87:0/1018047349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 msgr2=0x5578abb05960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb05960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b2c8a3640 1 -- 172.21.15.87:0/925968534 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4b100d0c60 con 0x7f4b281021a0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.694+0000 7f4b267fc640 1 -- 172.21.15.87:0/1018047349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ab8467d0 con 0x5578abb0ea80 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.695+0000 7f4b267fc640 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f4b2806b670 tx=0x7f4b18005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4616ddcff61edc9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.695+0000 7f4b277fe640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 0x7f4b140a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009370 tx=0x7f4b20009240 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.695+0000 7f4b1dffb640 1 -- 172.21.15.87:0/1018047349 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18044020 con 0x5578abb0ea80 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.695+0000 7f4b1dffb640 1 -- 172.21.15.87:0/1018047349 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b1802cc90 con 0x5578abb0ea80 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.695+0000 7f4b1dffb640 1 -- 172.21.15.87:0/1018047349 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b180046c0 con 0x5578abb0ea80 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- 172.21.15.87:0/1018047349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 msgr2=0x5578abb10e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f4b2806b670 tx=0x7f4b18005dc0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- 172.21.15.87:0/1018047349 shutdown_connections 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb10e70 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb0e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- 172.21.15.87:0/1018047349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb05960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- 172.21.15.87:0/1018047349 >> 172.21.15.87:0/1018047349 conn(0x5578abaffcb0 msgr2=0x5578abb00090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- 172.21.15.87:0/1018047349 shutdown_connections 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- 172.21.15.87:0/1018047349 wait complete. 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- start start 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 Processor -- start 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05580 0x5578abb0ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb0bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5578abb11810 con 0x5578abb0ea80 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5578aba232f0 con 0x5578abb05ea0 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.696+0000 7f4b30572740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5578abb11990 con 0x5578abb05580 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b30572740 10 client.0 _ll_register_callbacks cb 0x5578aba30580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47392/0 (socket says 172.21.15.87:47392) 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 -- 172.21.15.87:0/3425002820 learned_addr learned my addr 172.21.15.87:0/3425002820 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b267fc640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb0bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05580 0x5578abb0ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 -- 172.21.15.87:0/3425002820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05580 msgr2=0x5578abb0ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05580 0x5578abb0ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 -- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 msgr2=0x5578abb0bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb0bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 -- 172.21.15.87:0/3425002820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ab77c3f0 con 0x5578abb0ea80 2024-04-22T13:39:27.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b267fc640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb0bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b277fe640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f4b200092b0 tx=0x7f4b20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f960e83c2cdde8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20026020 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4b20022870 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b1802cc90 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4b20031050 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b30572740 10 client.17730 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.697+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5578ab73c0f0 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b1802cc90 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4b2000f830 con 0x5578abb0ea80 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 10 client.17730 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 10 client.17730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 20 client.17730 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb1eac0 0x5578abb20f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 4 client.17730 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:27.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578aba77c00 con 0x5578abb1eac0 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb1eac0 0x5578abb20f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b06ffd640 10 client.17730 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.698+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb1eac0 0x5578abb20f20 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=448151cafa3dd28c server_cookie=ed8e4b6bfbbdd707 in_seq=0 out_seq=0 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b06ffd640 1 -- 172.21.15.87:0/3425002820 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578aba77c00 con 0x5578abb1eac0 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b06ffd640 10 client.17730 handle_command_reply: tid=0 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 client.17730 shutdown 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 msgr2=0x5578abb0c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f4b200092b0 tx=0x7f4b20004a40 comp rx=0 tx=0).stop 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 shutdown_connections 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb1eac0 0x5578abb20f20 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abb0c4f0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb0bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 --2- 172.21.15.87:0/3425002820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05580 0x5578abb0ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b06ffd640 0 client.17730 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:27.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.699+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 >> 172.21.15.87:0/3425002820 conn(0x5578abaffcb0 msgr2=0x5578abb01a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.700+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 shutdown_connections 2024-04-22T13:39:27.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.700+0000 7f4b30572740 1 -- 172.21.15.87:0/3425002820 wait complete. 2024-04-22T13:39:27.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.700+0000 7f4b30572740 20 client.17730 trim_cache size 0 max 16384 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 Processor -- start 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 -- start start 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb0e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb1a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281b0490 con 0x5578abb0ea80 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281add60 con 0x5578abb05580 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b30572740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281abbd0 con 0x5578abb05ea0 2024-04-22T13:39:27.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47404/0 (socket says 172.21.15.87:47404) 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 -- 172.21.15.87:0/4275462911 learned_addr learned my addr 172.21.15.87:0/4275462911 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb0e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b267fc640 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb1a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 -- 172.21.15.87:0/4275462911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 msgr2=0x5578abb1a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb1a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 -- 172.21.15.87:0/4275462911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 msgr2=0x5578abb0e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb0e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.717+0000 7f4b277fe640 1 -- 172.21.15.87:0/4275462911 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578abb00630 con 0x5578abb0ea80 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b277fe640 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009820 tx=0x7f4b200098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a43520b0ab10bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b1d7fa640 1 -- 172.21.15.87:0/4275462911 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20026040 con 0x5578abb0ea80 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b1d7fa640 1 -- 172.21.15.87:0/4275462911 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b2003a070 con 0x5578abb0ea80 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b1d7fa640 1 -- 172.21.15.87:0/4275462911 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20022670 con 0x5578abb0ea80 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 -- 172.21.15.87:0/4275462911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 msgr2=0x5578abafebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009820 tx=0x7f4b200098f0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 -- 172.21.15.87:0/4275462911 shutdown_connections 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb0ea80 0x5578abafebf0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb05ea0 0x5578abb1a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 --2- 172.21.15.87:0/4275462911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05580 0x5578abb0e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 -- 172.21.15.87:0/4275462911 >> 172.21.15.87:0/4275462911 conn(0x5578abaffcb0 msgr2=0x5578abafe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.718+0000 7f4b30572740 1 -- 172.21.15.87:0/4275462911 shutdown_connections 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 -- 172.21.15.87:0/4275462911 wait complete. 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 -- start start 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 Processor -- start 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb08570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb0ea80 0x5578abb08c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281ac220 con 0x5578abb05580 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281ad690 con 0x5578abb05ea0 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b281ac8f0 con 0x5578abb0ea80 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b30572740 10 client.0 _ll_register_callbacks cb 0x7f4b20014790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b26ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b26ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47418/0 (socket says 172.21.15.87:47418) 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b26ffd640 1 -- 172.21.15.87:0/760696906 learned_addr learned my addr 172.21.15.87:0/760696906 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.719+0000 7f4b267fc640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb08570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 -- 172.21.15.87:0/760696906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb0ea80 msgr2=0x5578abb08c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb0ea80 0x5578abb08c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 -- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 msgr2=0x5578abb08570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb08570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 -- 172.21.15.87:0/760696906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578abb84060 con 0x5578abb05580 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b267fc640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb08570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b26ffd640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4b10099040 tx=0x7f4b1001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd11d609c7b2d29d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b100a5850 con 0x5578abb05580 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4b10099250 con 0x5578abb05580 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b2003a070 con 0x5578abb05580 2024-04-22T13:39:27.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4b100bd0b0 con 0x5578abb05580 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 10 client.17736 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5578abb851e0 con 0x5578abb05580 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b2003a070 con 0x5578abb05580 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4b10018b20 con 0x5578abb05580 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 10 client.17736 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 20 client.17736 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb858c0 0x5578abb87ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 4 client.17736 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.720+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5578aba71a70 con 0x5578abb858c0 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b267fc640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb858c0 0x5578abb87ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b06ffd640 10 client.17736 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b267fc640 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb858c0 0x5578abb87ce0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ed8b605901514 server_cookie=e9f04ffe986a1d2a in_seq=0 out_seq=0 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b06ffd640 1 -- 172.21.15.87:0/760696906 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5578aba71a70 con 0x5578abb858c0 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b06ffd640 10 client.17736 handle_command_reply: tid=0 2024-04-22T13:39:27.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b30572740 1 client.17736 shutdown 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 msgr2=0x5578abb170e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.721+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f4b10099040 tx=0x7f4b1001f000 comp rx=0 tx=0).stop 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 shutdown_connections 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5578abb858c0 0x5578abb87ce0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578abb0ea80 0x5578abb08c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578abb05ea0 0x5578abb08570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b06ffd640 0 client.17736 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 --2- 172.21.15.87:0/760696906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5578abb05580 0x5578abb170e0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 >> 172.21.15.87:0/760696906 conn(0x5578abaffcb0 msgr2=0x5578abafed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 shutdown_connections 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 1 -- 172.21.15.87:0/760696906 wait complete. 2024-04-22T13:39:27.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b30572740 20 client.17736 trim_cache size 0 max 16384 2024-04-22T13:39:27.725 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:39:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:39:27.725 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:27.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 msgr2=0x7f4b140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 0x7f4b140a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009370 tx=0x7f4b20009240 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 msgr2=0x7f4b2819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000e9f0 tx=0x7f4b1000eeb0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 shutdown_connections 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4b140a3d20 0x7f4b140a61e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b28198590 0x7f4b28198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b28107270 0x7f4b2819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 --2- 172.21.15.87:0/925968534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b281021a0 0x7f4b2819e3e0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 >> 172.21.15.87:0/925968534 conn(0x7f4b280793e0 msgr2=0x7f4b280fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 shutdown_connections 2024-04-22T13:39:27.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.722+0000 7f4b2d8a5640 1 -- 172.21.15.87:0/925968534 wait complete. 2024-04-22T13:39:27.730 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:39:27.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 -- 172.21.15.87:0/4082174475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 msgr2=0x7fd83010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 --2- 172.21.15.87:0/4082174475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd83010ee40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fd828009ca0 tx=0x7fd82802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 -- 172.21.15.87:0/4082174475 shutdown_connections 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 --2- 172.21.15.87:0/4082174475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd83010ee40 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 --2- 172.21.15.87:0/4082174475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd830102b50 0x7fd830106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 --2- 172.21.15.87:0/4082174475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 0x7fd830102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 -- 172.21.15.87:0/4082174475 >> 172.21.15.87:0/4082174475 conn(0x7fd8300793a0 msgr2=0x7fd8300797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 -- 172.21.15.87:0/4082174475 shutdown_connections 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.832+0000 7fd8369b2640 1 -- 172.21.15.87:0/4082174475 wait complete. 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 Processor -- start 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 -- start start 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 0x7fd8301a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd83019a680 0x7fd83019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd8301121e0 con 0x7fd830107270 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112060 con 0x7fd8301021a0 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd8369b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112360 con 0x7fd83019a680 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47446/0 (socket says 172.21.15.87:47446) 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 -- 172.21.15.87:0/272852978 learned_addr learned my addr 172.21.15.87:0/272852978 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82ffff640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 0x7fd8301a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 -- 172.21.15.87:0/272852978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd83019a680 msgr2=0x7fd83019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd83019a680 0x7fd83019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 -- 172.21.15.87:0/272852978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 msgr2=0x7fd8301a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 0x7fd8301a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 -- 172.21.15.87:0/272852978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83019b280 con 0x7fd830107270 2024-04-22T13:39:27.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.833+0000 7fd82f7fe640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fd82400d740 tx=0x7fd82400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.834+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824017070 con 0x7fd830107270 2024-04-22T13:39:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.834+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd82400de50 con 0x7fd830107270 2024-04-22T13:39:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.834+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824004ce0 con 0x7fd830107270 2024-04-22T13:39:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.834+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301a7360 con 0x7fd830107270 2024-04-22T13:39:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.834+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8301a7710 con 0x7fd830107270 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.835+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd830103670 con 0x7fd830107270 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.836+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd824004e80 con 0x7fd830107270 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.836+0000 7fd82d7fa640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 0x7fd8180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.836+0000 7fd82ffff640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 0x7fd8180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.836+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fd8240d05c0 con 0x7fd830107270 2024-04-22T13:39:27.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.837+0000 7fd82ffff640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 0x7fd8180a61e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd820005e10 tx=0x7fd820005da0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:27.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:27.840+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd82409cca0 con 0x7fd830107270 2024-04-22T13:39:28.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.055+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd830102bf0 con 0x7fd830107270 2024-04-22T13:39:28.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.056+0000 7fd82d7fa640 1 -- 172.21.15.87:0/272852978 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fd82409c5a0 con 0x7fd830107270 2024-04-22T13:39:28.053 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:39:28.053 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:39:28.053 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 msgr2=0x7fd8180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 0x7fd8180a61e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd820005e10 tx=0x7fd820005da0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 msgr2=0x7fd8301a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fd82400d740 tx=0x7fd82400dc00 comp rx=0 tx=0).stop 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 shutdown_connections 2024-04-22T13:39:28.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd8180a3d20 0x7fd8180a61e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd83019a680 0x7fd83019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd830107270 0x7fd8301a0b00 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 --2- 172.21.15.87:0/272852978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8301021a0 0x7fd8301a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 >> 172.21.15.87:0/272852978 conn(0x7fd8300793a0 msgr2=0x7fd83010d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 shutdown_connections 2024-04-22T13:39:28.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.058+0000 7fd8369b2640 1 -- 172.21.15.87:0/272852978 wait complete. 2024-04-22T13:39:28.067 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:39:28.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 -- 172.21.15.87:0/677794865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 msgr2=0x7f04b4101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/677794865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b4101600 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009930 tx=0x7f04a402bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:28.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 -- 172.21.15.87:0/677794865 shutdown_connections 2024-04-22T13:39:28.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/677794865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 0x7f04b410dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/677794865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b4101600 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/677794865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 0x7f04b4107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 -- 172.21.15.87:0/677794865 >> 172.21.15.87:0/677794865 conn(0x7f04b4076c50 msgr2=0x7f04b4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 -- 172.21.15.87:0/677794865 shutdown_connections 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.172+0000 7f04bc3f4640 1 -- 172.21.15.87:0/677794865 wait complete. 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 Processor -- start 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 -- start start 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 0x7f04b419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 0x7f04b4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41110e0 con 0x7f04b41011a0 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4110f60 con 0x7f04b4101b40 2024-04-22T13:39:28.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04bc3f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111260 con 0x7f04b4107030 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47466/0 (socket says 172.21.15.87:47466) 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 -- 172.21.15.87:0/1754133620 learned_addr learned my addr 172.21.15.87:0/1754133620 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba991640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 0x7f04b4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04b998f640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 0x7f04b419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 -- 172.21.15.87:0/1754133620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 msgr2=0x7f04b4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 0x7f04b4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 -- 172.21.15.87:0/1754133620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 msgr2=0x7f04b419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 0x7f04b419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.173+0000 7f04ba190640 1 -- 172.21.15.87:0/1754133620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4198d90 con 0x7f04b41011a0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04ba190640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f04b000d770 tx=0x7f04b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04ab7fe640 1 -- 172.21.15.87:0/1754133620 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0017070 con 0x7f04b41011a0 2024-04-22T13:39:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04ab7fe640 1 -- 172.21.15.87:0/1754133620 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b00054d0 con 0x7f04b41011a0 2024-04-22T13:39:28.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b4199080 con 0x7f04b41011a0 2024-04-22T13:39:28.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04ab7fe640 1 -- 172.21.15.87:0/1754133620 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0004c30 con 0x7f04b41011a0 2024-04-22T13:39:28.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.174+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41a52d0 con 0x7f04b41011a0 2024-04-22T13:39:28.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.176+0000 7f04ab7fe640 1 -- 172.21.15.87:0/1754133620 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f04b0004e10 con 0x7f04b41011a0 2024-04-22T13:39:28.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.176+0000 7f04befe4740 1 Processor -- start 2024-04-22T13:39:28.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 -- start start 2024-04-22T13:39:28.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ab7fe640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 0x7f04a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c216be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5598c20c51f0 con 0x5598c2160e80 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5598c20624b0 con 0x5598c2169a00 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04befe4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5598c20c11b0 con 0x5598c21604d0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ba190640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 0x7f04a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47480/0 (socket says 172.21.15.87:47480) 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 -- 172.21.15.87:0/1481346584 learned_addr learned my addr 172.21.15.87:0/1481346584 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ba991640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c216be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ab7fe640 1 -- 172.21.15.87:0/1754133620 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f04b00d0290 con 0x7f04b41011a0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ba991640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 -- 172.21.15.87:0/1481346584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 msgr2=0x5598c21608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 -- 172.21.15.87:0/1481346584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 msgr2=0x5598c216be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c216be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04b998f640 1 -- 172.21.15.87:0/1481346584 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c20541b0 con 0x5598c2160e80 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.177+0000 7f04ba991640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c216be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04ba190640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 0x7f04a00a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f04b00cdad0 tx=0x7f04b0091700 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04b998f640 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f087cbf404a29964 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04a8bfe640 1 -- 172.21.15.87:0/1481346584 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4044020 con 0x5598c2160e80 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04a8bfe640 1 -- 172.21.15.87:0/1481346584 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a402cc90 con 0x5598c2160e80 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04a8bfe640 1 -- 172.21.15.87:0/1481346584 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40046c0 con 0x5598c2160e80 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04befe4740 1 -- 172.21.15.87:0/1481346584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 msgr2=0x5598c2169430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.178+0000 7f04befe4740 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a4005e80 comp rx=0 tx=0).stop 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- 172.21.15.87:0/1481346584 shutdown_connections 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c216be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c2169430 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- 172.21.15.87:0/1481346584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- 172.21.15.87:0/1481346584 >> 172.21.15.87:0/1481346584 conn(0x5598c215aba0 msgr2=0x5598c215af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- 172.21.15.87:0/1481346584 shutdown_connections 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- 172.21.15.87:0/1481346584 wait complete. 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- start start 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 Processor -- start 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 0x5598c2166670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5598c216c910 con 0x5598c2169a00 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5598c207e500 con 0x5598c2160e80 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5598c216ca90 con 0x5598c21604d0 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04befe4740 10 client.0 _ll_register_callbacks cb 0x5598c204cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04ba991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04ba991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47484/0 (socket says 172.21.15.87:47484) 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04ba991640 1 -- 172.21.15.87:0/3476810155 learned_addr learned my addr 172.21.15.87:0/3476810155 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.179+0000 7f04b998f640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 0x5598c2166670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 -- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 msgr2=0x5598c21692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 -- 172.21.15.87:0/3476810155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 msgr2=0x5598c2166670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 0x5598c2166670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 -- 172.21.15.87:0/3476810155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c1dd73f0 con 0x5598c2169a00 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04b998f640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 0x5598c2166670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04ba991640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f04ac012bd0 tx=0x7f04ac014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf423dc56f009de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac010ea0 con 0x5598c2169a00 2024-04-22T13:39:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f04ac004d20 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04a402cc90 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f04ac003970 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 10 client.17754 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5598c1ea17d0 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a402cc90 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f04ac01e030 con 0x5598c2169a00 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 10 client.17754 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 10 client.17754 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.180+0000 7f04befe4740 20 client.17754 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c2179b20 0x5598c217bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f04befe4740 4 client.17754 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598c2064930 con 0x5598c2179b20 2024-04-22T13:39:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f04b998f640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c2179b20 0x5598c217bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f04b998f640 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c2179b20 0x5598c217bf10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f54108662c1caed6 server_cookie=a0500ebb4eb0a4af in_seq=0 out_seq=0 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.181+0000 7f049dffb640 10 client.17754 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f049dffb640 1 -- 172.21.15.87:0/3476810155 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598c2064930 con 0x5598c2179b20 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f049dffb640 10 client.17754 handle_command_reply: tid=0 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 client.17754 shutdown 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 msgr2=0x5598c2166bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f04ac012bd0 tx=0x7f04ac014040 comp rx=0 tx=0).stop 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 shutdown_connections 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c2179b20 0x5598c217bf10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2169a00 0x5598c2166bd0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2160e80 0x5598c2166670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f049dffb640 0 client.17754 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 --2- 172.21.15.87:0/3476810155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c21604d0 0x5598c21692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 >> 172.21.15.87:0/3476810155 conn(0x5598c215aba0 msgr2=0x5598c215c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 shutdown_connections 2024-04-22T13:39:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 1 -- 172.21.15.87:0/3476810155 wait complete. 2024-04-22T13:39:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.182+0000 7f04befe4740 20 client.17754 trim_cache size 0 max 16384 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 Processor -- start 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 -- start start 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2160e80 0x5598c2159a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c2168da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41b0380 con 0x5598c21604d0 2024-04-22T13:39:28.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41adc50 con 0x5598c2169a00 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04befe4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41abac0 con 0x5598c2160e80 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04ba991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.195+0000 7f04ba991640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47494/0 (socket says 172.21.15.87:47494) 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 -- 172.21.15.87:0/2779740003 learned_addr learned my addr 172.21.15.87:0/2779740003 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 -- 172.21.15.87:0/2779740003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2160e80 msgr2=0x5598c2159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2160e80 0x5598c2159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 -- 172.21.15.87:0/2779740003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 msgr2=0x5598c2168da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c2168da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 -- 172.21.15.87:0/2779740003 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c2166f40 con 0x5598c21604d0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f04ba991640 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000f80 tx=0x7f04ac009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4317de5a9be3efc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f049ffff640 1 -- 172.21.15.87:0/2779740003 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0286b0 con 0x5598c21604d0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f049ffff640 1 -- 172.21.15.87:0/2779740003 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac012070 con 0x5598c21604d0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.196+0000 7f049ffff640 1 -- 172.21.15.87:0/2779740003 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac005040 con 0x5598c21604d0 2024-04-22T13:39:28.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- 172.21.15.87:0/2779740003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 msgr2=0x5598c21757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000f80 tx=0x7f04ac009cc0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- 172.21.15.87:0/2779740003 shutdown_connections 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c2169a00 0x5598c2168da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2160e80 0x5598c2159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- 172.21.15.87:0/2779740003 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c21604d0 0x5598c21757d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- 172.21.15.87:0/2779740003 >> 172.21.15.87:0/2779740003 conn(0x5598c215aba0 msgr2=0x5598c2159370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- 172.21.15.87:0/2779740003 shutdown_connections 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- 172.21.15.87:0/2779740003 wait complete. 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- start start 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 Processor -- start 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 0x5598c2171f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2169a00 0x5598c216d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41ac110 con 0x5598c2160e80 2024-04-22T13:39:28.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41ad580 con 0x5598c21604d0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41ac7e0 con 0x5598c2169a00 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04befe4740 10 client.0 _ll_register_callbacks cb 0x7f04ac0128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04b998f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04b998f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47496/0 (socket says 172.21.15.87:47496) 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04b998f640 1 -- 172.21.15.87:0/2783139614 learned_addr learned my addr 172.21.15.87:0/2783139614 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04ba991640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 0x5598c2171f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04b998f640 1 -- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2169a00 msgr2=0x5598c216d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.197+0000 7f04b998f640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2169a00 0x5598c216d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04b998f640 1 -- 172.21.15.87:0/2783139614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 msgr2=0x5598c2171f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04b998f640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 0x5598c2171f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04b998f640 1 -- 172.21.15.87:0/2783139614 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c2160c20 con 0x5598c2160e80 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04ba991640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 0x5598c2171f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04b998f640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009930 tx=0x7f04a4005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7435e8732d89c9b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4044040 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f04a4053bd0 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04ac012070 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f04a4053dd0 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 10 client.17760 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5598c21dfd40 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac012070 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f04a40035f0 con 0x5598c2160e80 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 10 client.17760 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 10 client.17760 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 20 client.17760 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c21e00d0 0x5598c21e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 4 client.17760 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.198+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5598c20d2c10 con 0x5598c21e00d0 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04ba991640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c21e00d0 0x5598c21e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04ba991640 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c21e00d0 0x5598c21e24f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f713c89b8b635f06 server_cookie=d16fe9e0cf0b255c in_seq=0 out_seq=0 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f049dffb640 10 client.17760 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f049dffb640 1 -- 172.21.15.87:0/2783139614 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5598c20d2c10 con 0x5598c21e00d0 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f049dffb640 10 client.17760 handle_command_reply: tid=0 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 client.17760 shutdown 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 msgr2=0x5598c216d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009930 tx=0x7f04a4005be0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 shutdown_connections 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5598c21e00d0 0x5598c21e24f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5598c2169a00 0x5598c216d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5598c2160e80 0x5598c216d380 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f049dffb640 0 client.17760 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 --2- 172.21.15.87:0/2783139614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598c21604d0 0x5598c2171f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 >> 172.21.15.87:0/2783139614 conn(0x5598c215aba0 msgr2=0x5598c215c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 shutdown_connections 2024-04-22T13:39:28.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 1 -- 172.21.15.87:0/2783139614 wait complete. 2024-04-22T13:39:28.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.199+0000 7f04befe4740 20 client.17760 trim_cache size 0 max 16384 2024-04-22T13:39:28.203 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 msgr2=0x7f04a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 0x7f04a00a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f04b00cdad0 tx=0x7f04b0091700 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 msgr2=0x7f04b419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f04b000d770 tx=0x7f04b000dc30 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 shutdown_connections 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f04a00a3d20 0x7f04a00a61e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04b4107030 0x7f04b4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b4101b40 0x7f04b419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 --2- 172.21.15.87:0/1754133620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04b41011a0 0x7f04b419e410 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 >> 172.21.15.87:0/1754133620 conn(0x7f04b4076c50 msgr2=0x7f04b406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 shutdown_connections 2024-04-22T13:39:28.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.200+0000 7f04bc3f4640 1 -- 172.21.15.87:0/1754133620 wait complete. 2024-04-22T13:39:28.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- 172.21.15.87:0/4247705261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac107260 msgr2=0x7f15ac10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/4247705261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac107260 0x7f15ac10ee30 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f1594009900 tx=0x7f159402bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- 172.21.15.87:0/4247705261 shutdown_connections 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/4247705261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac107260 0x7f15ac10ee30 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/4247705261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac102b40 0x7f15ac106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/4247705261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15ac102190 0x7f15ac102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- 172.21.15.87:0/4247705261 >> 172.21.15.87:0/4247705261 conn(0x7f15ac0793a0 msgr2=0x7f15ac0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- 172.21.15.87:0/4247705261 shutdown_connections 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- 172.21.15.87:0/4247705261 wait complete. 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 Processor -- start 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.312+0000 7f15b2e4a640 1 -- start start 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 0x7f15ac198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15ac19cf40 0x7f15ac19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac112220 con 0x7f15ac102190 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1120a0 con 0x7f15ac19cf40 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1123a0 con 0x7f15ac107260 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47514/0 (socket says 172.21.15.87:47514) 2024-04-22T13:39:28.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 -- 172.21.15.87:0/130194940 learned_addr learned my addr 172.21.15.87:0/130194940 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15abfff640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 0x7f15ac198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 -- 172.21.15.87:0/130194940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 msgr2=0x7f15ac198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 0x7f15ac198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 -- 172.21.15.87:0/130194940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15ac19cf40 msgr2=0x7f15ac19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15ac19cf40 0x7f15ac19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 -- 172.21.15.87:0/130194940 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac19da30 con 0x7f15ac102190 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15abfff640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 0x7f15ac198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b0be6640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c920 tx=0x7f15a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15a9ffb640 1 -- 172.21.15.87:0/130194940 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4007c70 con 0x7f15ac102190 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15a9ffb640 1 -- 172.21.15.87:0/130194940 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a4007e10 con 0x7f15ac102190 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac19dd20 con 0x7f15ac102190 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.313+0000 7f15a9ffb640 1 -- 172.21.15.87:0/130194940 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4018820 con 0x7f15ac102190 2024-04-22T13:39:28.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.314+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac1a5510 con 0x7f15ac102190 2024-04-22T13:39:28.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.316+0000 7f15a9ffb640 1 -- 172.21.15.87:0/130194940 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f15a400f450 con 0x7f15ac102190 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.316+0000 7f15b5a3e740 1 Processor -- start 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.316+0000 7f15b5a3e740 1 -- start start 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5605bc7191f0 con 0x5605bc7b4e80 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605bc6b64b0 con 0x5605bc7b44d0 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5605bc7151b0 con 0x5605bc7bda00 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15a9ffb640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 0x7f15980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47516/0 (socket says 172.21.15.87:47516) 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 -- 172.21.15.87:0/1261749588 learned_addr learned my addr 172.21.15.87:0/1261749588 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b13e7640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 0x7f15980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15abfff640 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b13e7640 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15a9ffb640 1 -- 172.21.15.87:0/130194940 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f15a40d09f0 con 0x7f15ac102190 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 -- 172.21.15.87:0/1261749588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 msgr2=0x5605bc7bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 -- 172.21.15.87:0/1261749588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 msgr2=0x5605bc7b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 -- 172.21.15.87:0/1261749588 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bc6a81b0 con 0x5605bc7b4e80 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b13e7640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 0x7f15980a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f159402c170 tx=0x7f1594005e80 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ce090 tx=0x7f15a40c7a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87b3ee8e4d434c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15a2ffd640 1 -- 172.21.15.87:0/1261749588 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4067480 con 0x5605bc7b4e80 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15a2ffd640 1 -- 172.21.15.87:0/1261749588 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a409c970 con 0x5605bc7b4e80 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.317+0000 7f15a2ffd640 1 -- 172.21.15.87:0/1261749588 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a409ac70 con 0x5605bc7b4e80 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1261749588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 msgr2=0x5605bc7bd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ce090 tx=0x7f15a40c7a80 comp rx=0 tx=0).stop 2024-04-22T13:39:28.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1261749588 shutdown_connections 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7bd430 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1261749588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1261749588 >> 172.21.15.87:0/1261749588 conn(0x5605bc7aeba0 msgr2=0x5605bc7aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1261749588 shutdown_connections 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1261749588 wait complete. 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.318+0000 7f15b5a3e740 1 -- start start 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 Processor -- start 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7bd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5605bc7c0970 con 0x5605bc7b4e80 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605bc6d2500 con 0x5605bc7b44d0 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5605bc7c0af0 con 0x5605bc7bda00 2024-04-22T13:39:28.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b5a3e740 10 client.0 _ll_register_callbacks cb 0x5605bc6a0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47520/0 (socket says 172.21.15.87:47520) 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 -- 172.21.15.87:0/841203537 learned_addr learned my addr 172.21.15.87:0/841203537 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b0be6640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15b13e7640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 -- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 msgr2=0x5605bc7bad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 -- 172.21.15.87:0/841203537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 msgr2=0x5605bc7bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7bd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 -- 172.21.15.87:0/841203537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bc42b3f0 con 0x5605bc7b4e80 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f15abfff640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f159c013a20 tx=0x7f159c013ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8594a9252f892842 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.319+0000 7f158bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c01b3f0 con 0x5605bc7b4e80 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f159c01b590 con 0x5605bc7b4e80 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a409c970 con 0x5605bc7b4e80 2024-04-22T13:39:28.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f159c01b7d0 con 0x5605bc7b4e80 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 10 client.17772 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5605bc4f57d0 con 0x5605bc7b4e80 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a409c970 con 0x5605bc7b4e80 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f159c01a030 con 0x5605bc7b4e80 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 10 client.17772 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 10 client.17772 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 20 client.17772 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc7cdcf0 0x5605bc7d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 4 client.17772 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.320+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605bc6b8930 con 0x5605bc7cdcf0 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.321+0000 7f15b0be6640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc7cdcf0 0x5605bc7d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.321+0000 7f158bfff640 10 client.17772 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.321+0000 7f15b0be6640 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc7cdcf0 0x5605bc7d0140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1d1d1fe251f6731 server_cookie=cab781aa80691b36 in_seq=0 out_seq=0 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f158bfff640 1 -- 172.21.15.87:0/841203537 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605bc6b8930 con 0x5605bc7cdcf0 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f158bfff640 10 client.17772 handle_command_reply: tid=0 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 client.17772 shutdown 2024-04-22T13:39:28.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 msgr2=0x5605bc7ba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f159c013a20 tx=0x7f159c013ee0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 shutdown_connections 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc7cdcf0 0x5605bc7d0140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7bda00 0x5605bc7bad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b4e80 0x5605bc7ba710 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/841203537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7b44d0 0x5605bc7bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f158bfff640 0 client.17772 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 >> 172.21.15.87:0/841203537 conn(0x5605bc7aeba0 msgr2=0x5605bc7b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 shutdown_connections 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 1 -- 172.21.15.87:0/841203537 wait complete. 2024-04-22T13:39:28.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.322+0000 7f15b5a3e740 20 client.17772 trim_cache size 0 max 16384 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 Processor -- start 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 -- start start 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7bce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7bd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b0340 con 0x5605bc7b44d0 2024-04-22T13:39:28.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1adc10 con 0x5605bc7bda00 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1aba80 con 0x5605bc7b4e80 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47526/0 (socket says 172.21.15.87:47526) 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.335+0000 7f15b0be6640 1 -- 172.21.15.87:0/2303937769 learned_addr learned my addr 172.21.15.87:0/2303937769 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15abfff640 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7bce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 -- 172.21.15.87:0/2303937769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 msgr2=0x5605bc7bce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7bce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 -- 172.21.15.87:0/2303937769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 msgr2=0x5605bc7bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 -- 172.21.15.87:0/2303937769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bc7c0020 con 0x5605bc7b44d0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b0be6640 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f15a409c8b0 tx=0x7f15a40bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184a0b7c6948620e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15a27fc640 1 -- 172.21.15.87:0/2303937769 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a40a55e0 con 0x5605bc7b44d0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15a27fc640 1 -- 172.21.15.87:0/2303937769 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a40a2140 con 0x5605bc7b44d0 2024-04-22T13:39:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15a27fc640 1 -- 172.21.15.87:0/2303937769 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4098fe0 con 0x5605bc7b44d0 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 -- 172.21.15.87:0/2303937769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 msgr2=0x5605bc7c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f15a409c8b0 tx=0x7f15a40bd040 comp rx=0 tx=0).stop 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 -- 172.21.15.87:0/2303937769 shutdown_connections 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7bce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.336+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/2303937769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7c9830 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- 172.21.15.87:0/2303937769 >> 172.21.15.87:0/2303937769 conn(0x5605bc7aeba0 msgr2=0x5605bc7adc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- 172.21.15.87:0/2303937769 shutdown_connections 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- 172.21.15.87:0/2303937769 wait complete. 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- start start 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 Processor -- start 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1ac0d0 con 0x5605bc7b44d0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1ad540 con 0x5605bc7bda00 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1ac7a0 con 0x5605bc7b4e80 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b5a3e740 10 client.0 _ll_register_callbacks cb 0x7f159c013630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b0be6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47532/0 (socket says 172.21.15.87:47532) 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.337+0000 7f15b0be6640 1 -- 172.21.15.87:0/1934765020 learned_addr learned my addr 172.21.15.87:0/1934765020 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15abfff640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b13e7640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 -- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 msgr2=0x5605bc7b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 -- 172.21.15.87:0/1934765020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 msgr2=0x5605bc7b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7b7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 -- 172.21.15.87:0/1934765020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bc833420 con 0x5605bc7b44d0 2024-04-22T13:39:28.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15abfff640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b0be6640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ce180 tx=0x7f15a400fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43b5ef09ac1dccb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a40a55e0 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f15a400f7b0 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a40a2140 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f15a4098fe0 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 10 client.17778 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5605bc8346e0 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a40a2140 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f15a409cbd0 con 0x5605bc7b44d0 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 10 client.17778 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 10 client.17778 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:39:28.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 20 client.17778 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc8348b0 0x5605bc836ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 4 client.17778 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.338+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605bc726c10 con 0x5605bc8348b0 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15abfff640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc8348b0 0x5605bc836ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f158bfff640 10 client.17778 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15abfff640 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc8348b0 0x5605bc836ca0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd8d4718d63bcbf9 server_cookie=be4fdd1c613e5024 in_seq=0 out_seq=0 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f158bfff640 1 -- 172.21.15.87:0/1934765020 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605bc726c10 con 0x5605bc8348b0 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f158bfff640 10 client.17778 handle_command_reply: tid=0 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 client.17778 shutdown 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 msgr2=0x5605bc7b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f15a40ce180 tx=0x7f15a400fe60 comp rx=0 tx=0).stop 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 shutdown_connections 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5605bc8348b0 0x5605bc836ca0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605bc7bda00 0x5605bc7b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605bc7b4e80 0x5605bc7b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f158bfff640 0 client.17778 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 --2- 172.21.15.87:0/1934765020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5605bc7b44d0 0x5605bc7b7530 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 >> 172.21.15.87:0/1934765020 conn(0x5605bc7aeba0 msgr2=0x5605bc7b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 shutdown_connections 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.339+0000 7f15b5a3e740 1 -- 172.21.15.87:0/1934765020 wait complete. 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b5a3e740 20 client.17778 trim_cache size 0 max 16384 2024-04-22T13:39:28.343 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 msgr2=0x7f15980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 0x7f15980a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f159402c170 tx=0x7f1594005e80 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 msgr2=0x7f15ac198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c920 tx=0x7f15a400cde0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 shutdown_connections 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f15980a3d20 0x7f15980a61e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15ac19cf40 0x7f15ac19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac107260 0x7f15ac198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 --2- 172.21.15.87:0/130194940 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15ac102190 0x7f15ac198680 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 >> 172.21.15.87:0/130194940 conn(0x7f15ac0793a0 msgr2=0x7f15ac0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 shutdown_connections 2024-04-22T13:39:28.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.340+0000 7f15b2e4a640 1 -- 172.21.15.87:0/130194940 wait complete. 2024-04-22T13:39:28.347 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:28 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/272852978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:28.348 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:39:28.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:28 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/272852978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- 172.21.15.87:0/2874532196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec106160 msgr2=0x7f7eec10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- 172.21.15.87:0/2874532196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec106160 0x7f7eec10dd30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009ca0 tx=0x7f7ee402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- 172.21.15.87:0/2874532196 shutdown_connections 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- 172.21.15.87:0/2874532196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec106160 0x7f7eec10dd30 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- 172.21.15.87:0/2874532196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec101a40 0x7f7eec105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- 172.21.15.87:0/2874532196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec101090 0x7f7eec101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- 172.21.15.87:0/2874532196 >> 172.21.15.87:0/2874532196 conn(0x7f7eec077f50 msgr2=0x7f7eec078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- 172.21.15.87:0/2874532196 shutdown_connections 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- 172.21.15.87:0/2874532196 wait complete. 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 Processor -- start 2024-04-22T13:39:28.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- start start 2024-04-22T13:39:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 0x7f7eec198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec19cf00 0x7f7eec19d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1121e0 con 0x7f7eec101090 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec112060 con 0x7f7eec106160 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.452+0000 7f7ef3160640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec112360 con 0x7f7eec19cf00 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47556/0 (socket says 172.21.15.87:47556) 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3690778627 learned_addr learned my addr 172.21.15.87:0/3690778627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7eebfff640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 0x7f7eec198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec19cf00 0x7f7eec19d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3690778627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec19cf00 msgr2=0x7f7eec19d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec19cf00 0x7f7eec19d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3690778627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 msgr2=0x7f7eec198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 0x7f7eec198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3690778627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec19d9f0 con 0x7f7eec101090 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7eebfff640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 0x7f7eec198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e9f0 tx=0x7f7ee000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ee9ffb640 1 -- 172.21.15.87:0/3690778627 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000cd80 con 0x7f7eec101090 2024-04-22T13:39:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ee9ffb640 1 -- 172.21.15.87:0/3690778627 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee00108e0 con 0x7f7eec101090 2024-04-22T13:39:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ee9ffb640 1 -- 172.21.15.87:0/3690778627 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0018980 con 0x7f7eec101090 2024-04-22T13:39:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eec19dce0 con 0x7f7eec101090 2024-04-22T13:39:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.453+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eec1a5520 con 0x7f7eec101090 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ee9ffb640 1 -- 172.21.15.87:0/3690778627 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7ee0010430 con 0x7f7eec101090 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 Processor -- start 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 -- start start 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe97f0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560fe97551f0 con 0x560fe97f0d60 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560fe96f24b0 con 0x560fe97f9e60 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.456+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560fe97511b0 con 0x560fe97f03b0 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47568/0 (socket says 172.21.15.87:47568) 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3889428495 learned_addr learned my addr 172.21.15.87:0/3889428495 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ee9ffb640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 0x7f7edc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe97f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7eebfff640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3889428495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 msgr2=0x560fe97f0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe97f0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3889428495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 msgr2=0x560fe97fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 -- 172.21.15.87:0/3889428495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe96e41b0 con 0x560fe97f0d60 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe97f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7eebfff640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 0x7f7edc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ee9ffb640 1 -- 172.21.15.87:0/3690778627 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f7ee00d0b40 con 0x7f7eec101090 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00994f0 tx=0x7f7ee0098ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de23ef979a017e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ed6ffd640 1 -- 172.21.15.87:0/3889428495 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0025170 con 0x560fe97f0d60 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ed6ffd640 1 -- 172.21.15.87:0/3889428495 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee0010720 con 0x560fe97f0d60 2024-04-22T13:39:28.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.457+0000 7f7ed6ffd640 1 -- 172.21.15.87:0/3889428495 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0025330 con 0x560fe97f0d60 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3889428495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 msgr2=0x560fe97f9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 0x7f7edc0a6260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4004af0 tx=0x7f7ee402e040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00994f0 tx=0x7f7ee0098ef0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3889428495 shutdown_connections 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe97f9890 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3889428495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe97f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3889428495 >> 172.21.15.87:0/3889428495 conn(0x560fe97eaba0 msgr2=0x560fe97eaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3889428495 shutdown_connections 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3889428495 wait complete. 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 -- start start 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.458+0000 7f7ef5e32740 1 Processor -- start 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f03b0 0x560fe9800250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f0d60 0x560fe97fd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560fe9800ce0 con 0x560fe97f9e60 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560fe970e500 con 0x560fe97f03b0 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560fe9800e60 con 0x560fe97f0d60 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef5e32740 10 client.0 _ll_register_callbacks cb 0x560fe96dcb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47576/0 (socket says 172.21.15.87:47576) 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 -- 172.21.15.87:0/3881275280 learned_addr learned my addr 172.21.15.87:0/3881275280 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7eebfff640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f0d60 0x560fe97fd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f03b0 0x560fe9800250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 -- 172.21.15.87:0/3881275280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f0d60 msgr2=0x560fe97fd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f0d60 0x560fe97fd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 -- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f03b0 msgr2=0x560fe9800250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f03b0 0x560fe9800250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 -- 172.21.15.87:0/3881275280 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe94673f0 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009ca0 tx=0x7f7ee4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96974f52f557920 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.459+0000 7f7ecbfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4051060 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f7ee4003970 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560fe96f4930 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7ee4039040 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 10 client.17790 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560fe95317d0 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560fe96f4930 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7ee4056470 con 0x560fe97f9e60 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 10 client.17790 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 10 client.17790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 20 client.17790 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe980df80 0x560fe9810400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 4 client.17790 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fe9762c10 con 0x560fe980df80 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.460+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe980df80 0x560fe9810400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.461+0000 7f7ecbfff640 10 client.17790 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.461+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe980df80 0x560fe9810400 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1752e565a959db10 server_cookie=e61670dafe406e74 in_seq=0 out_seq=0 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.461+0000 7f7ecbfff640 1 -- 172.21.15.87:0/3881275280 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fe9762c10 con 0x560fe980df80 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.461+0000 7f7ecbfff640 10 client.17790 handle_command_reply: tid=0 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.461+0000 7f7ef5e32740 1 client.17790 shutdown 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 msgr2=0x560fe97fdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009ca0 tx=0x7f7ee4005c50 comp rx=0 tx=0).stop 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 shutdown_connections 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe980df80 0x560fe9810400 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fdb20 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f0d60 0x560fe97fd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/3881275280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f03b0 0x560fe9800250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ecbfff640 0 client.17790 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:28.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 >> 172.21.15.87:0/3881275280 conn(0x560fe97eaba0 msgr2=0x560fe97ec8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 shutdown_connections 2024-04-22T13:39:28.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 1 -- 172.21.15.87:0/3881275280 wait complete. 2024-04-22T13:39:28.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.462+0000 7f7ef5e32740 20 client.17790 trim_cache size 0 max 16384 2024-04-22T13:39:28.465 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:28 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/272852978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 Processor -- start 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 -- start start 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe9809be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f0d60 0x560fe97f9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1ae060 con 0x560fe97f9e60 2024-04-22T13:39:28.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1ab930 con 0x560fe97f0d60 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1a97a0 con 0x560fe97f03b0 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47590/0 (socket says 172.21.15.87:47590) 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 -- 172.21.15.87:0/2443703194 learned_addr learned my addr 172.21.15.87:0/2443703194 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7eebfff640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f0d60 0x560fe97f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe9809be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 -- 172.21.15.87:0/2443703194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 msgr2=0x560fe9809be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe9809be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 -- 172.21.15.87:0/2443703194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f0d60 msgr2=0x560fe97f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f0d60 0x560fe97f9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef16fd640 1 -- 172.21.15.87:0/2443703194 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe97effe0 con 0x560fe97f9e60 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.475+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe9809be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4056840 tx=0x7f7ee4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd01696cfb66d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ed67fc640 1 -- 172.21.15.87:0/2443703194 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4051070 con 0x560fe97f9e60 2024-04-22T13:39:28.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ed67fc640 1 -- 172.21.15.87:0/2443703194 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee4045030 con 0x560fe97f9e60 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ed67fc640 1 -- 172.21.15.87:0/2443703194 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4040910 con 0x560fe97f9e60 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 -- 172.21.15.87:0/2443703194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 msgr2=0x560fe97fc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4056840 tx=0x7f7ee4005ef0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 -- 172.21.15.87:0/2443703194 shutdown_connections 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f9e60 0x560fe97fc600 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f0d60 0x560fe97f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/2443703194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe9809be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 -- 172.21.15.87:0/2443703194 >> 172.21.15.87:0/2443703194 conn(0x560fe97eaba0 msgr2=0x560fe97fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 -- 172.21.15.87:0/2443703194 shutdown_connections 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.476+0000 7f7ef5e32740 1 -- 172.21.15.87:0/2443703194 wait complete. 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 -- start start 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 Processor -- start 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe98060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1a9df0 con 0x560fe97f0d60 2024-04-22T13:39:28.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1ab260 con 0x560fe97f9e60 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec1aa4c0 con 0x560fe97f03b0 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef5e32740 10 client.0 _ll_register_callbacks cb 0x7f7ee4032340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47596/0 (socket says 172.21.15.87:47596) 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 -- 172.21.15.87:0/4165566903 learned_addr learned my addr 172.21.15.87:0/4165566903 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe98060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 -- 172.21.15.87:0/4165566903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 msgr2=0x560fe98060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe98060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 -- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 msgr2=0x560fe97fa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.477+0000 7f7eebfff640 1 -- 172.21.15.87:0/4165566903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe97faa70 con 0x560fe97f0d60 2024-04-22T13:39:28.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef16fd640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7eebfff640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7eec06b6e0 tx=0x7f7ed8011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f44990f8443b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8020060 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f7ed8004030 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee4045030 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7ed8010ea0 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 10 client.17796 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560fe9874030 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee4045030 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7ed801a070 con 0x560fe97f0d60 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 10 client.17796 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 10 client.17796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 20 client.17796 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe9874810 0x560fe9876c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 4 client.17796 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:39:28.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.478+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fe97fdeb0 con 0x560fe9874810 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe9874810 0x560fe9876c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ecbfff640 10 client.17796 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef0efc640 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe9874810 0x560fe9876c30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54a75eb01e124ac server_cookie=eb3303b897aa4039 in_seq=0 out_seq=0 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ecbfff640 1 -- 172.21.15.87:0/4165566903 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7ee009de50 con 0x560fe9874810 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ecbfff640 10 client.17796 handle_command_reply: tid=0 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 client.17796 shutdown 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 msgr2=0x560fe9806630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7eec06b6e0 tx=0x7f7ed8011e00 comp rx=0 tx=0).stop 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 shutdown_connections 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560fe9874810 0x560fe9876c30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe97f9e60 0x560fe97fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ecbfff640 0 client.17796 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560fe97f0d60 0x560fe9806630 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 --2- 172.21.15.87:0/4165566903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fe97f03b0 0x560fe98060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 >> 172.21.15.87:0/4165566903 conn(0x560fe97eaba0 msgr2=0x560fe97f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 shutdown_connections 2024-04-22T13:39:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 1 -- 172.21.15.87:0/4165566903 wait complete. 2024-04-22T13:39:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.479+0000 7f7ef5e32740 20 client.17796 trim_cache size 0 max 16384 2024-04-22T13:39:28.485 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 msgr2=0x7f7edc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 0x7f7edc0a6260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4004af0 tx=0x7f7ee402e040 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 msgr2=0x7f7eec198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e9f0 tx=0x7f7ee000eeb0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 shutdown_connections 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f7edc0a3da0 0x7f7edc0a6260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7eec19cf00 0x7f7eec19d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eec106160 0x7f7eec198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 --2- 172.21.15.87:0/3690778627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7eec101090 0x7f7eec198640 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 >> 172.21.15.87:0/3690778627 conn(0x7f7eec077f50 msgr2=0x7f7eec0fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 shutdown_connections 2024-04-22T13:39:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:28.480+0000 7f7ef3160640 1 -- 172.21.15.87:0/3690778627 wait complete. 2024-04-22T13:39:29.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:29 smithi120 ceph-mon[31101]: pgmap v236: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T13:39:29.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:29 smithi132 ceph-mon[31531]: pgmap v236: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T13:39:29.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:29 smithi087 ceph-mon[27999]: pgmap v236: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T13:39:29.488 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:39:29.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:39:29.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:39:29.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 -- 172.21.15.87:0/3248227254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314107260 msgr2=0x7fc31410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 --2- 172.21.15.87:0/3248227254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314107260 0x7fc31410ee30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009ca0 tx=0x7fc30c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 -- 172.21.15.87:0/3248227254 shutdown_connections 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 --2- 172.21.15.87:0/3248227254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314107260 0x7fc31410ee30 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 --2- 172.21.15.87:0/3248227254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102b40 0x7fc314106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 --2- 172.21.15.87:0/3248227254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314102190 0x7fc314102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 -- 172.21.15.87:0/3248227254 >> 172.21.15.87:0/3248227254 conn(0x7fc3140fd990 msgr2=0x7fc3140ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 -- 172.21.15.87:0/3248227254 shutdown_connections 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.607+0000 7fc31a175640 1 -- 172.21.15.87:0/3248227254 wait complete. 2024-04-22T13:39:29.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 Processor -- start 2024-04-22T13:39:29.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 -- start start 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102190 0x7fc314074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 0x7fc314074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141121d0 con 0x7fc314074cd0 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc314112050 con 0x7fc314107260 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc31a175640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc314112350 con 0x7fc314102190 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc313fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc313fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41158/0 (socket says 172.21.15.87:41158) 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.608+0000 7fc313fff640 1 -- 172.21.15.87:0/1870728481 learned_addr learned my addr 172.21.15.87:0/1870728481 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 0x7fc314074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102190 0x7fc314074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 -- 172.21.15.87:0/1870728481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102190 msgr2=0x7fc314074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102190 0x7fc314074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 -- 172.21.15.87:0/1870728481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 msgr2=0x7fc314074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 0x7fc314074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 -- 172.21.15.87:0/1870728481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3141a6fe0 con 0x7fc314074cd0 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 0x7fc314074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc313fff640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc30c005e30 tx=0x7fc30c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc310ff9640 1 -- 172.21.15.87:0/1870728481 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c041020 con 0x7fc314074cd0 2024-04-22T13:39:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc310ff9640 1 -- 172.21.15.87:0/1870728481 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c032970 con 0x7fc314074cd0 2024-04-22T13:39:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3141a7270 con 0x7fc314074cd0 2024-04-22T13:39:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc310ff9640 1 -- 172.21.15.87:0/1870728481 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c039040 con 0x7fc314074cd0 2024-04-22T13:39:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.609+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3141a7760 con 0x7fc314074cd0 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.611+0000 7fc310ff9640 1 -- 172.21.15.87:0/1870728481 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc30c044020 con 0x7fc314074cd0 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 Processor -- start 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 -- start start 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078aae10 0x5650078ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x565007805f00 con 0x5650078a18e0 2024-04-22T13:39:29.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650077a3590 con 0x5650078aae10 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc31cd67740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565007801ec0 con 0x5650078a2290 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41166/0 (socket says 172.21.15.87:41166) 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 -- 172.21.15.87:0/4120128733 learned_addr learned my addr 172.21.15.87:0/4120128733 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc3137fe640 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc310ff9640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 0x7fc2fc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078aae10 0x5650078ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 -- 172.21.15.87:0/4120128733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 msgr2=0x5650078aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078aa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc313fff640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 0x7fc2fc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 -- 172.21.15.87:0/4120128733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078aae10 msgr2=0x5650078ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078aae10 0x5650078ad260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc312ffd640 1 -- 172.21.15.87:0/4120128733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650075e07e0 con 0x5650078a18e0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.612+0000 7fc310ff9640 1 -- 172.21.15.87:0/1870728481 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc30c0f2500 con 0x7fc314074cd0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc312ffd640 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc30800bf00 tx=0x7fc30800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e33cc187c6b5247 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc2f9ffb640 1 -- 172.21.15.87:0/4120128733 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308005020 con 0x5650078a18e0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc313fff640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 0x7fc2fc0a6210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc30c034fd0 tx=0x7fc30c090030 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc2f9ffb640 1 -- 172.21.15.87:0/4120128733 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3080047a0 con 0x5650078a18e0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc2f9ffb640 1 -- 172.21.15.87:0/4120128733 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30800f630 con 0x5650078a18e0 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 -- 172.21.15.87:0/4120128733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 msgr2=0x5650078a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc30800bf00 tx=0x7fc30800b690 comp rx=0 tx=0).stop 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 -- 172.21.15.87:0/4120128733 shutdown_connections 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078aae10 0x5650078ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 --2- 172.21.15.87:0/4120128733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a18e0 0x5650078a1cc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 -- 172.21.15.87:0/4120128733 >> 172.21.15.87:0/4120128733 conn(0x56500789bfb0 msgr2=0x56500789c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 -- 172.21.15.87:0/4120128733 shutdown_connections 2024-04-22T13:39:29.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.613+0000 7fc31cd67740 1 -- 172.21.15.87:0/4120128733 wait complete. 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 -- start start 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 Processor -- start 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078a83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5650078add80 con 0x5650078aae10 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650077bf580 con 0x5650078a18e0 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5650078adf00 con 0x5650078a2290 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc31cd67740 10 client.0 _ll_register_callbacks cb 0x5650077c5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.614+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41180/0 (socket says 172.21.15.87:41180) 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 -- 172.21.15.87:0/1013226787 learned_addr learned my addr 172.21.15.87:0/1013226787 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078a83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 -- 172.21.15.87:0/1013226787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 msgr2=0x5650078a83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078a83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 -- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 msgr2=0x5650078a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 -- 172.21.15.87:0/1013226787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565007640f70 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc308004770 tx=0x7fc308004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f62f220d4819472 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc2f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308004b20 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc308004fa0 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3080047a0 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc308004cc0 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc31cd67740 10 client.17808 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5650074d8170 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.615+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3080047a0 con 0x5650078aae10 2024-04-22T13:39:29.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc30802d9d0 con 0x5650078aae10 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 10 client.17808 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 10 client.17808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 20 client.17808 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5650078bb350 0x5650078bd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 4 client.17808 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565007813920 con 0x5650078bb350 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5650078bb350 0x5650078bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc2f2ffd640 10 client.17808 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.616+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5650078bb350 0x5650078bd740 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2102243f232b5c5e server_cookie=834b976422544120 in_seq=0 out_seq=0 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/1013226787 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565007813920 con 0x5650078bb350 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc2f2ffd640 10 client.17808 handle_command_reply: tid=0 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 client.17808 shutdown 2024-04-22T13:39:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 msgr2=0x5650078a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc308004770 tx=0x7fc308004700 comp rx=0 tx=0).stop 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 shutdown_connections 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5650078bb350 0x5650078bd740 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x5650078a88e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a2290 0x5650078a83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1013226787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc2f2ffd640 0 client.17808 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 >> 172.21.15.87:0/1013226787 conn(0x56500789bfb0 msgr2=0x56500789dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 shutdown_connections 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 1 -- 172.21.15.87:0/1013226787 wait complete. 2024-04-22T13:39:29.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.617+0000 7fc31cd67740 20 client.17808 trim_cache size 0 max 16384 2024-04-22T13:39:29.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.634+0000 7fc31cd67740 1 Processor -- start 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.634+0000 7fc31cd67740 1 -- start start 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a18e0 0x5650078aa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 0x5650078b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141b2550 con 0x5650078aae10 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141afe20 con 0x5650078a2290 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc31cd67740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141adc90 con 0x5650078a18e0 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41190/0 (socket says 172.21.15.87:41190) 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 -- 172.21.15.87:0/1151297694 learned_addr learned my addr 172.21.15.87:0/1151297694 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a18e0 0x5650078aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 0x5650078b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 -- 172.21.15.87:0/1151297694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a18e0 msgr2=0x5650078aa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a18e0 0x5650078aa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 -- 172.21.15.87:0/1151297694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 msgr2=0x5650078b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 0x5650078b6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 -- 172.21.15.87:0/1151297694 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650078adc60 con 0x5650078aae10 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc3137fe640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 0x5650078b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc312ffd640 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc3080059f0 tx=0x7fc30800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36aa4d27f2b680ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc2f97fa640 1 -- 172.21.15.87:0/1151297694 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308011be0 con 0x5650078aae10 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc2f97fa640 1 -- 172.21.15.87:0/1151297694 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30800b5f0 con 0x5650078aae10 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.635+0000 7fc2f97fa640 1 -- 172.21.15.87:0/1151297694 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30802d430 con 0x5650078aae10 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- 172.21.15.87:0/1151297694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 msgr2=0x56500789ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc3080059f0 tx=0x7fc30800be10 comp rx=0 tx=0).stop 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- 172.21.15.87:0/1151297694 shutdown_connections 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078aae10 0x56500789ae90 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a2290 0x5650078b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 --2- 172.21.15.87:0/1151297694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078a18e0 0x5650078aa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- 172.21.15.87:0/1151297694 >> 172.21.15.87:0/1151297694 conn(0x56500789bfb0 msgr2=0x56500789a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- 172.21.15.87:0/1151297694 shutdown_connections 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- 172.21.15.87:0/1151297694 wait complete. 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.636+0000 7fc31cd67740 1 -- start start 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 Processor -- start 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 0x5650078aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141ae2e0 con 0x5650078a2290 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141af750 con 0x5650078a18e0 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc3141ae9b0 con 0x5650078aae10 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc31cd67740 10 client.0 _ll_register_callbacks cb 0x7fc30400d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:29.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41198/0 (socket says 172.21.15.87:41198) 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 -- 172.21.15.87:0/942303504 learned_addr learned my addr 172.21.15.87:0/942303504 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc312ffd640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc312ffd640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 0x5650078aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 -- 172.21.15.87:0/942303504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 msgr2=0x5650078aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 0x5650078aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 -- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 msgr2=0x5650078b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 -- 172.21.15.87:0/942303504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650078af5e0 con 0x5650078a2290 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc312ffd640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 0x5650078aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc3137fe640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fc304005d00 tx=0x7fc30400dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d450e6f0c5b0379 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.637+0000 7fc2f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30401d470 con 0x5650078a2290 2024-04-22T13:39:29.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc30401d610 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30800b5f0 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc30402c050 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 10 client.17814 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x565007921790 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30800b5f0 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc30401c3d0 con 0x5650078a2290 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 10 client.17814 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 10 client.17814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 20 client.17814 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565007921f40 0x565007924360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 4 client.17814 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56500780d790 con 0x565007921f40 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.638+0000 7fc312ffd640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565007921f40 0x565007924360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.639+0000 7fc2f2ffd640 10 client.17814 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.639+0000 7fc312ffd640 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565007921f40 0x565007924360 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11af60535afce66e server_cookie=42fe79a615d0674d in_seq=0 out_seq=0 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc2f2ffd640 1 -- 172.21.15.87:0/942303504 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56500780d790 con 0x565007921f40 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc2f2ffd640 10 client.17814 handle_command_reply: tid=0 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 client.17814 shutdown 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 msgr2=0x5650078ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fc304005d00 tx=0x7fc30400dd60 comp rx=0 tx=0).stop 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 shutdown_connections 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x565007921f40 0x565007924360 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5650078aae10 0x5650078aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5650078a2290 0x5650078ae8a0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc2f2ffd640 0 client.17814 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 --2- 172.21.15.87:0/942303504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650078a18e0 0x5650078b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 >> 172.21.15.87:0/942303504 conn(0x56500789bfb0 msgr2=0x56500789b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 shutdown_connections 2024-04-22T13:39:29.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 1 -- 172.21.15.87:0/942303504 wait complete. 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.640+0000 7fc31cd67740 20 client.17814 trim_cache size 0 max 16384 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "fee0137f-b261-4f10-a4a2-ddff90730bd4", 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:39:29.643 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 msgr2=0x7fc2fc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 0x7fc2fc0a6210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc30c034fd0 tx=0x7fc30c090030 comp rx=0 tx=0).stop 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 msgr2=0x7fc3141a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc30c005e30 tx=0x7fc30c034670 comp rx=0 tx=0).stop 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 shutdown_connections 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc2fc0a3d50 0x7fc2fc0a6210 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc314074cd0 0x7fc3141a68e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc314107260 0x7fc314074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 --2- 172.21.15.87:0/1870728481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc314102190 0x7fc314074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 >> 172.21.15.87:0/1870728481 conn(0x7fc3140fd990 msgr2=0x7fc3140fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 shutdown_connections 2024-04-22T13:39:29.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.641+0000 7fc31a175640 1 -- 172.21.15.87:0/1870728481 wait complete. 2024-04-22T13:39:29.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:39:29.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:39:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 -- 172.21.15.87:0/64736436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c0107270 msgr2=0x7fc3c010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/64736436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c0107270 0x7fc3c010ee40 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009ca0 tx=0x7fc3b802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 -- 172.21.15.87:0/64736436 shutdown_connections 2024-04-22T13:39:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/64736436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c0107270 0x7fc3c010ee40 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/64736436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0102b50 0x7fc3c0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/64736436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c01021a0 0x7fc3c0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 -- 172.21.15.87:0/64736436 >> 172.21.15.87:0/64736436 conn(0x7fc3c00793e0 msgr2=0x7fc3c00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 -- 172.21.15.87:0/64736436 shutdown_connections 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.753+0000 7fc3c78d2640 1 -- 172.21.15.87:0/64736436 wait complete. 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 Processor -- start 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 -- start start 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0107270 0x7fc3c0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c019cf90 0x7fc3c019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01121c0 con 0x7fc3c01021a0 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0112040 con 0x7fc3c019cf90 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c78d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c0112340 con 0x7fc3c0107270 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41210/0 (socket says 172.21.15.87:41210) 2024-04-22T13:39:29.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 -- 172.21.15.87:0/2298510818 learned_addr learned my addr 172.21.15.87:0/2298510818 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c019cf90 0x7fc3c019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0107270 0x7fc3c0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 -- 172.21.15.87:0/2298510818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0107270 msgr2=0x7fc3c0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0107270 0x7fc3c0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 -- 172.21.15.87:0/2298510818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c019cf90 msgr2=0x7fc3c019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c019cf90 0x7fc3c019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 -- 172.21.15.87:0/2298510818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c019da80 con 0x7fc3c01021a0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400d950 tx=0x7fc3b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.754+0000 7fc3b27fc640 1 -- 172.21.15.87:0/2298510818 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4017070 con 0x7fc3c01021a0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.755+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c019dd70 con 0x7fc3c01021a0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.755+0000 7fc3b27fc640 1 -- 172.21.15.87:0/2298510818 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b4009c30 con 0x7fc3c01021a0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.755+0000 7fc3b27fc640 1 -- 172.21.15.87:0/2298510818 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400f620 con 0x7fc3c01021a0 2024-04-22T13:39:29.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.755+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c01a55b0 con 0x7fc3c01021a0 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3b27fc640 1 -- 172.21.15.87:0/2298510818 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc3b4010460 con 0x7fc3c01021a0 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 Processor -- start 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 -- start start 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3b27fc640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x560804582540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x560804584e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5608044de1e0 con 0x560804579580 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56080447b390 con 0x560804582a80 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.757+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5608044da1a0 con 0x560804579ea0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41212/0 (socket says 172.21.15.87:41212) 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 -- 172.21.15.87:0/2144037493 learned_addr learned my addr 172.21.15.87:0/2144037493 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3b27fc640 1 -- 172.21.15.87:0/2298510818 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc3b40d0400 con 0x7fc3c01021a0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x560804582540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 -- 172.21.15.87:0/2144037493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 msgr2=0x560804582540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x560804582540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 -- 172.21.15.87:0/2144037493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 msgr2=0x560804584e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x560804584e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 -- 172.21.15.87:0/2144037493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608042ba7d0 con 0x560804579580 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004af0 tx=0x7fc3b8032380 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3c566e640 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c370 tx=0x7fc3b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb92f5218bee617d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3ab7fe640 1 -- 172.21.15.87:0/2144037493 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4020030 con 0x560804579580 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3ab7fe640 1 -- 172.21.15.87:0/2144037493 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b4009de0 con 0x560804579580 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.758+0000 7fc3ab7fe640 1 -- 172.21.15.87:0/2144037493 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b409cd70 con 0x560804579580 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/2144037493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 msgr2=0x560804579960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c370 tx=0x7fc3b4005880 comp rx=0 tx=0).stop 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/2144037493 shutdown_connections 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x560804584e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x560804582540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/2144037493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804579960 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/2144037493 >> 172.21.15.87:0/2144037493 conn(0x560804573cb0 msgr2=0x560804574090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/2144037493 shutdown_connections 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/2144037493 wait complete. 2024-04-22T13:39:29.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 -- start start 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.759+0000 7fc3ca5a1740 1 Processor -- start 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x5608045804a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560804585810 con 0x560804579580 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608044972f0 con 0x560804582a80 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560804585990 con 0x560804579ea0 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 10 client.0 _ll_register_callbacks cb 0x5608044a4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41222/0 (socket says 172.21.15.87:41222) 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 -- 172.21.15.87:0/3125493653 learned_addr learned my addr 172.21.15.87:0/3125493653 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x5608045804a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 -- 172.21.15.87:0/3125493653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 msgr2=0x56080457ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 -- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 msgr2=0x5608045804a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x5608045804a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 -- 172.21.15.87:0/3125493653 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608041f03f0 con 0x560804579580 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c0e0 tx=0x7fc3b40cdb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110c5b8bc06b6d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40b5060 con 0x560804579580 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc3b4098a50 con 0x560804579580 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.760+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b4009de0 con 0x560804579580 2024-04-22T13:39:29.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc3b4020070 con 0x560804579580 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 10 client.17826 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5608041b00f0 con 0x560804579580 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b404d070 con 0x560804579580 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc3b40bb170 con 0x560804579580 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 10 client.17826 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 10 client.17826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 20 client.17826 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560804592800 0x560804594bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 4 client.17826 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608044ebc00 con 0x560804592800 2024-04-22T13:39:29.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.761+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560804592800 0x560804594bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3a8ff9640 10 client.17826 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560804592800 0x560804594bf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4312a427ed9a8de server_cookie=bb0a156239de6305 in_seq=0 out_seq=0 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3125493653 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608044ebc00 con 0x560804592800 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3a8ff9640 10 client.17826 handle_command_reply: tid=0 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3ca5a1740 1 client.17826 shutdown 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 msgr2=0x56080457fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.762+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c0e0 tx=0x7fc3b40cdb00 comp rx=0 tx=0).stop 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 shutdown_connections 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560804592800 0x560804594bf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x5608045804a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3a8ff9640 0 client.17826 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3125493653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x56080457fa20 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 >> 172.21.15.87:0/3125493653 conn(0x560804573cb0 msgr2=0x560804575a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 shutdown_connections 2024-04-22T13:39:29.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3125493653 wait complete. 2024-04-22T13:39:29.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.763+0000 7fc3ca5a1740 20 client.17826 trim_cache size 0 max 16384 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 Processor -- start 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 -- start start 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080458e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x56080458afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01b0360 con 0x560804579580 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01adc30 con 0x560804582a80 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01abaa0 con 0x560804579ea0 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41238/0 (socket says 172.21.15.87:41238) 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 -- 172.21.15.87:0/3444373140 learned_addr learned my addr 172.21.15.87:0/3444373140 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x56080458afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080458e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 -- 172.21.15.87:0/3444373140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 msgr2=0x56080458e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080458e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 -- 172.21.15.87:0/3444373140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 msgr2=0x56080458afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x56080458afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 -- 172.21.15.87:0/3444373140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560804578780 con 0x560804579580 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.780+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c370 tx=0x7fc3b409a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea3c87daaf2dcb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3aaffd640 1 -- 172.21.15.87:0/3444373140 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40b5050 con 0x560804579580 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3aaffd640 1 -- 172.21.15.87:0/3444373140 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b404d030 con 0x560804579580 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3aaffd640 1 -- 172.21.15.87:0/3444373140 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40bb170 con 0x560804579580 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3444373140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 msgr2=0x560804582150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fc3b409c370 tx=0x7fc3b409a8f0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3444373140 shutdown_connections 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804582a80 0x56080458afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080458e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3444373140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804579580 0x560804582150 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3444373140 >> 172.21.15.87:0/3444373140 conn(0x560804573cb0 msgr2=0x5608045724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3444373140 shutdown_connections 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3444373140 wait complete. 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.781+0000 7fc3ca5a1740 1 -- start start 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 Processor -- start 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804579580 0x56080457d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01ac0f0 con 0x560804582a80 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01ad560 con 0x560804579580 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c01ac7c0 con 0x560804579ea0 2024-04-22T13:39:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3ca5a1740 10 client.0 _ll_register_callbacks cb 0x7fc3b409c1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41250/0 (socket says 172.21.15.87:41250) 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 -- 172.21.15.87:0/3502854604 learned_addr learned my addr 172.21.15.87:0/3502854604 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804579580 0x56080457d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 -- 172.21.15.87:0/3502854604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 msgr2=0x56080457d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 -- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804579580 msgr2=0x56080457d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804579580 0x56080457d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c5e6f640 1 -- 172.21.15.87:0/3502854604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56080457c4c0 con 0x560804582a80 2024-04-22T13:39:29.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.782+0000 7fc3c4e6d640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3c5e6f640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009ca0 tx=0x7fc3b802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241efc4bcc1038ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8048040 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fc3b8003c30 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b404d030 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc3b805a050 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 10 client.17832 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5608045fd540 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b404d030 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc3b804b070 con 0x560804582a80 2024-04-22T13:39:29.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 10 client.17832 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 20 client.17832 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5608045fdce0 0x560804600100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 4 client.17832 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.783+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5608044e5a70 con 0x5608045fdce0 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5608045fdce0 0x560804600100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3a8ff9640 10 client.17832 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3c566e640 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5608045fdce0 0x560804600100 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f116e614685ec8dc server_cookie=1b8676ecc3ba0565 in_seq=0 out_seq=0 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3a8ff9640 1 -- 172.21.15.87:0/3502854604 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5608044e5a70 con 0x5608045fdce0 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3a8ff9640 10 client.17832 handle_command_reply: tid=0 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 client.17832 shutdown 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 msgr2=0x56080457bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009ca0 tx=0x7fc3b802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 shutdown_connections 2024-04-22T13:39:29.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5608045fdce0 0x560804600100 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560804582a80 0x56080457bc90 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3a8ff9640 0 client.17832 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560804579ea0 0x56080457d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 --2- 172.21.15.87:0/3502854604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560804579580 0x56080457d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.784+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 >> 172.21.15.87:0/3502854604 conn(0x560804573cb0 msgr2=0x560804572d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 shutdown_connections 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3ca5a1740 1 -- 172.21.15.87:0/3502854604 wait complete. 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3ca5a1740 20 client.17832 trim_cache size 0 max 16384 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:29.788 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: "fee0137f-b261-4f10-a4a2-ddff90730bd4": { 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "fee0137f-b261-4f10-a4a2-ddff90730bd4", 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:29.789 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 msgr2=0x7fc3ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004af0 tx=0x7fc3b8032380 comp rx=0 tx=0).stop 2024-04-22T13:39:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 msgr2=0x7fc3c0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400d950 tx=0x7fc3b400de10 comp rx=0 tx=0).stop 2024-04-22T13:39:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 shutdown_connections 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c019cf90 0x7fc3c019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3c0107270 0x7fc3c0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 --2- 172.21.15.87:0/2298510818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3c01021a0 0x7fc3c0198720 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 >> 172.21.15.87:0/2298510818 conn(0x7fc3c00793e0 msgr2=0x7fc3c00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 shutdown_connections 2024-04-22T13:39:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:29.785+0000 7fc3c78d2640 1 -- 172.21.15.87:0/2298510818 wait complete. 2024-04-22T13:39:29.792 INFO:tasks.cephfs.filesystem:scrub status for tag:fee0137f-b261-4f10-a4a2-ddff90730bd4 - {'path': '/', 'tag': 'fee0137f-b261-4f10-a4a2-ddff90730bd4', 'options': 'recursive,force'} 2024-04-22T13:39:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:30 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:39:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:30 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:30 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:39:30.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:30 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:30.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:30 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:39:30.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:30 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:30.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:30 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:39:30.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:30 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:30.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:30 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:39:30.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:30 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:30.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:30 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:39:30.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:30 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:39:31.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:31 smithi120 ceph-mon[31101]: pgmap v237: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 286 B/s wr, 6 op/s 2024-04-22T13:39:31.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:31 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:39:31.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:31 smithi132 ceph-mon[31531]: pgmap v237: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 286 B/s wr, 6 op/s 2024-04-22T13:39:31.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:31 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:39:31.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:31 smithi087 ceph-mon[27999]: pgmap v237: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 286 B/s wr, 6 op/s 2024-04-22T13:39:31.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:31 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:39:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:32 smithi120 ceph-mon[31101]: pgmap v238: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:39:32.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:32 smithi132 ceph-mon[31531]: pgmap v238: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:39:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:32 smithi087 ceph-mon[27999]: pgmap v238: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:39:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:34 smithi120 ceph-mon[31101]: pgmap v239: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:34 smithi132 ceph-mon[31531]: pgmap v239: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:34 smithi087 ceph-mon[27999]: pgmap v239: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:36 smithi120 ceph-mon[31101]: pgmap v240: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:39:36.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:36 smithi132 ceph-mon[31531]: pgmap v240: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:39:36.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:36 smithi087 ceph-mon[27999]: pgmap v240: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:39:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:38 smithi120 ceph-mon[31101]: pgmap v241: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:38.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:38 smithi132 ceph-mon[31531]: pgmap v241: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:38.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:38 smithi087 ceph-mon[27999]: pgmap v241: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:39:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 -- 172.21.15.87:0/335035968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 msgr2=0x7f30cc10dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 --2- 172.21.15.87:0/335035968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10dd60 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009ca0 tx=0x7f30c402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 -- 172.21.15.87:0/335035968 shutdown_connections 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 --2- 172.21.15.87:0/335035968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10dd60 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 --2- 172.21.15.87:0/335035968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc101ac0 0x7f30cc105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 --2- 172.21.15.87:0/335035968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 0x7f30cc1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 -- 172.21.15.87:0/335035968 >> 172.21.15.87:0/335035968 conn(0x7f30cc078030 msgr2=0x7f30cc078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 -- 172.21.15.87:0/335035968 shutdown_connections 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.568+0000 7f30d27f4640 1 -- 172.21.15.87:0/335035968 wait complete. 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 Processor -- start 2024-04-22T13:39:39.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 -- start start 2024-04-22T13:39:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 0x7f30cc10e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 0x7f30cc1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc112210 con 0x7f30cc106190 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc112090 con 0x7f30cc10f170 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30d27f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc112390 con 0x7f30cc101110 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55418/0 (socket says 172.21.15.87:55418) 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.569+0000 7f30cb7fe640 1 -- 172.21.15.87:0/2605939216 learned_addr learned my addr 172.21.15.87:0/2605939216 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cbfff640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 0x7f30cc10e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30d0d91640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 0x7f30cc1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 -- 172.21.15.87:0/2605939216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 msgr2=0x7f30cc10e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 0x7f30cc10e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 -- 172.21.15.87:0/2605939216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 msgr2=0x7f30cc1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 0x7f30cc1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 -- 172.21.15.87:0/2605939216 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc1a6f20 con 0x7f30cc106190 2024-04-22T13:39:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30d0d91640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 0x7f30cc1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f30c000d770 tx=0x7f30c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30c97fa640 1 -- 172.21.15.87:0/2605939216 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0017070 con 0x7f30cc106190 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30c97fa640 1 -- 172.21.15.87:0/2605939216 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30c0009c30 con 0x7f30cc106190 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc1a7210 con 0x7f30cc106190 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30c97fa640 1 -- 172.21.15.87:0/2605939216 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c000f620 con 0x7f30cc106190 2024-04-22T13:39:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.570+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc1a7760 con 0x7f30cc106190 2024-04-22T13:39:39.575 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30c97fa640 1 -- 172.21.15.87:0/2605939216 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f30c001a070 con 0x7f30cc106190 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 Processor -- start 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 -- start start 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dbb60 0x559e301dbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301e50e0 0x559e301e74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e301408d0 con 0x559e301dc8d0 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e300dde80 con 0x559e301e50e0 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d53e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e3013c890 con 0x559e301dbb60 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30c97fa640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 0x7f30b40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d0d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d0d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55430/0 (socket says 172.21.15.87:55430) 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d0d91640 1 -- 172.21.15.87:0/564063906 learned_addr learned my addr 172.21.15.87:0/564063906 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30cbfff640 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dbb60 0x559e301dbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.576 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30c97fa640 1 -- 172.21.15.87:0/2605939216 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f30c00d0340 con 0x7f30cc106190 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301e50e0 0x559e301e74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.573+0000 7f30d0d91640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 0x7f30b40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 -- 172.21.15.87:0/564063906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dbb60 msgr2=0x559e301dbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dbb60 0x559e301dbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 -- 172.21.15.87:0/564063906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301e50e0 msgr2=0x559e301e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301e50e0 0x559e301e74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 -- 172.21.15.87:0/564063906 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e300ff320 con 0x559e301dc8d0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009430 tx=0x7f30c4032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e970bdbaabb108ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30b27fc640 1 -- 172.21.15.87:0/564063906 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c40327a0 con 0x559e301dc8d0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30b27fc640 1 -- 172.21.15.87:0/564063906 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30c4032940 con 0x559e301dc8d0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30b27fc640 1 -- 172.21.15.87:0/564063906 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4034830 con 0x559e301dc8d0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.574+0000 7f30d0d91640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 0x7f30b40a6280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30c404c000 tx=0x7f30c4048000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- 172.21.15.87:0/564063906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 msgr2=0x559e301db620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009430 tx=0x7f30c4032f30 comp rx=0 tx=0).stop 2024-04-22T13:39:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- 172.21.15.87:0/564063906 shutdown_connections 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301e50e0 0x559e301e74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dbb60 0x559e301dbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 --2- 172.21.15.87:0/564063906 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301db620 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- 172.21.15.87:0/564063906 >> 172.21.15.87:0/564063906 conn(0x559e301a2c90 msgr2=0x559e301d6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- 172.21.15.87:0/564063906 shutdown_connections 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- 172.21.15.87:0/564063906 wait complete. 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.575+0000 7f30d53e6740 1 -- start start 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 Processor -- start 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 0x559e301e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e301e7e10 con 0x559e301e50e0 2024-04-22T13:39:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e300c8eb0 con 0x559e301dbb60 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e301e7f90 con 0x559e301dc8d0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d53e6740 10 client.0 _ll_register_callbacks cb 0x559e300f5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55436/0 (socket says 172.21.15.87:55436) 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 -- 172.21.15.87:0/1279610634 learned_addr learned my addr 172.21.15.87:0/1279610634 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30cbfff640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 0x559e301e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 -- 172.21.15.87:0/1279610634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 msgr2=0x559e301e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 0x559e301e1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 -- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 msgr2=0x559e301e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30d0d91640 1 -- 172.21.15.87:0/1279610634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e2fe53210 con 0x559e301e50e0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.576+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 0x559e301e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d0d91640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f30c4003f70 tx=0x7f30c4034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c746e7c5631c6c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4039040 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f30c4034ad0 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30c4032940 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f30c4034cd0 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 10 client.17844 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e2ff1f620 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30c4032940 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f30c4043550 con 0x559e301e50e0 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 10 client.17844 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 10 client.17844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 20 client.17844 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e301f4f80 0x559e301f7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 4 client.17844 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.577+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e30154880 con 0x559e301f4f80 2024-04-22T13:39:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.578+0000 7f30cbfff640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e301f4f80 0x559e301f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.578+0000 7f30ab7fe640 10 client.17844 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.578+0000 7f30cbfff640 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e301f4f80 0x559e301f7370 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4246bd96db781b95 server_cookie=d57539e630871b93 in_seq=0 out_seq=0 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30ab7fe640 1 -- 172.21.15.87:0/1279610634 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e30154880 con 0x559e301f4f80 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30ab7fe640 10 client.17844 handle_command_reply: tid=0 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 client.17844 shutdown 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 msgr2=0x559e301e23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f30c4003f70 tx=0x7f30c4034440 comp rx=0 tx=0).stop 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 shutdown_connections 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e301f4f80 0x559e301f7370 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301e50e0 0x559e301e23b0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301dc8d0 0x559e301e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30ab7fe640 0 client.17844 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 --2- 172.21.15.87:0/1279610634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 >> 172.21.15.87:0/1279610634 conn(0x559e301a2c90 msgr2=0x559e301d85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 shutdown_connections 2024-04-22T13:39:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 1 -- 172.21.15.87:0/1279610634 wait complete. 2024-04-22T13:39:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.579+0000 7f30d53e6740 20 client.17844 trim_cache size 0 max 16384 2024-04-22T13:39:39.594 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 Processor -- start 2024-04-22T13:39:39.594 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 -- start start 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dc8d0 0x559e301e0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301fca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1b2430 con 0x559e301dbb60 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1afd00 con 0x559e301dc8d0 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d53e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1adb70 con 0x559e301e50e0 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55446/0 (socket says 172.21.15.87:55446) 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 -- 172.21.15.87:0/961870426 learned_addr learned my addr 172.21.15.87:0/961870426 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dc8d0 0x559e301e0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30d0d91640 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 -- 172.21.15.87:0/961870426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 msgr2=0x559e301fca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301fca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 -- 172.21.15.87:0/961870426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dc8d0 msgr2=0x559e301e0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dc8d0 0x559e301e0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.594+0000 7f30cbfff640 1 -- 172.21.15.87:0/961870426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e301dae30 con 0x559e301dbb60 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30cbfff640 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f30bc005f00 tx=0x7f30bc013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04318e76926bd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30b1ffb640 1 -- 172.21.15.87:0/961870426 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30bc01d430 con 0x559e301dbb60 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30b1ffb640 1 -- 172.21.15.87:0/961870426 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30bc005020 con 0x559e301dbb60 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30b1ffb640 1 -- 172.21.15.87:0/961870426 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30bc004a00 con 0x559e301dbb60 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 -- 172.21.15.87:0/961870426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 msgr2=0x559e301f0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f30bc005f00 tx=0x7f30bc013830 comp rx=0 tx=0).stop 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 -- 172.21.15.87:0/961870426 shutdown_connections 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301fca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dc8d0 0x559e301e0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 --2- 172.21.15.87:0/961870426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dbb60 0x559e301f0d10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 -- 172.21.15.87:0/961870426 >> 172.21.15.87:0/961870426 conn(0x559e301a2c90 msgr2=0x559e301e0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 -- 172.21.15.87:0/961870426 shutdown_connections 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.595+0000 7f30d53e6740 1 -- 172.21.15.87:0/961870426 wait complete. 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 -- start start 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 Processor -- start 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301ed630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1ae1c0 con 0x559e301dc8d0 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1af630 con 0x559e301dbb60 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc1ae890 con 0x559e301e50e0 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30d53e6740 10 client.0 _ll_register_callbacks cb 0x7f30c0010b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55462/0 (socket says 172.21.15.87:55462) 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30cb7fe640 1 -- 172.21.15.87:0/3369679640 learned_addr learned my addr 172.21.15.87:0/3369679640 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.596+0000 7f30cbfff640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301ed630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d0d91640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 -- 172.21.15.87:0/3369679640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 msgr2=0x559e301e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 -- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 msgr2=0x559e301ed630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301ed630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 -- 172.21.15.87:0/3369679640 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e301d8b50 con 0x559e301dc8d0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d0d91640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30cb7fe640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f30c009c2b0 tx=0x7f30c0009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d7218104577d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0021040 con 0x559e301dc8d0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f30c009b770 con 0x559e301dc8d0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30bc005020 con 0x559e301dc8d0 2024-04-22T13:39:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f30c00a4f90 con 0x559e301dc8d0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 10 client.17850 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e3025b110 con 0x559e301dc8d0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30bc005020 con 0x559e301dc8d0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f30c0098070 con 0x559e301dc8d0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 10 client.17850 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 10 client.17850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 20 client.17850 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e3025b880 0x559e3025dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 4 client.17850 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.597+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e300e03d0 con 0x559e3025b880 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.598+0000 7f30cbfff640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e3025b880 0x559e3025dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.598+0000 7f30cbfff640 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e3025b880 0x559e3025dca0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f76ec92b371667c server_cookie=beb0be93909fa3a in_seq=0 out_seq=0 2024-04-22T13:39:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.598+0000 7f30ab7fe640 10 client.17850 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.603+0000 7f30ab7fe640 1 -- 172.21.15.87:0/3369679640 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e300e03d0 con 0x559e3025b880 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.603+0000 7f30ab7fe640 10 client.17850 handle_command_reply: tid=0 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.603+0000 7f30d53e6740 1 client.17850 shutdown 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.603+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 msgr2=0x559e301e89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.603+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f30c009c2b0 tx=0x7f30c0009f20 comp rx=0 tx=0).stop 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 shutdown_connections 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e3025b880 0x559e3025dca0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e301e50e0 0x559e301e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30ab7fe640 0 client.17850 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e301dc8d0 0x559e301e89f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 --2- 172.21.15.87:0/3369679640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e301dbb60 0x559e301ed630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 >> 172.21.15.87:0/3369679640 conn(0x559e301a2c90 msgr2=0x559e301d83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 shutdown_connections 2024-04-22T13:39:39.601 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 1 -- 172.21.15.87:0/3369679640 wait complete. 2024-04-22T13:39:39.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d53e6740 20 client.17850 trim_cache size 0 max 16384 2024-04-22T13:39:39.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:39:39.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:39:39.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:39:39.602 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 msgr2=0x7f30b40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 0x7f30b40a6280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30c404c000 tx=0x7f30c4048000 comp rx=0 tx=0).stop 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 msgr2=0x7f30cc10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f30c000d770 tx=0x7f30c000dc30 comp rx=0 tx=0).stop 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 shutdown_connections 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f30b40a3dc0 0x7f30b40a6280 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc10f170 0x7f30cc1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30cc106190 0x7f30cc10ec30 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 --2- 172.21.15.87:0/2605939216 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30cc101110 0x7f30cc10e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 >> 172.21.15.87:0/2605939216 conn(0x7f30cc078030 msgr2=0x7f30cc0fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 shutdown_connections 2024-04-22T13:39:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:39.604+0000 7f30d27f4640 1 -- 172.21.15.87:0/2605939216 wait complete. 2024-04-22T13:39:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:40 smithi120 ceph-mon[31101]: pgmap v242: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:39:40.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:40 smithi132 ceph-mon[31531]: pgmap v242: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:39:40.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:40 smithi087 ceph-mon[27999]: pgmap v242: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:39:42.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:42 smithi120 ceph-mon[31101]: pgmap v243: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-04-22T13:39:42.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:42 smithi132 ceph-mon[31531]: pgmap v243: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-04-22T13:39:42.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:42 smithi087 ceph-mon[27999]: pgmap v243: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7 op/s 2024-04-22T13:39:44.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:44 smithi120 ceph-mon[31101]: pgmap v244: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:39:44.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:44 smithi132 ceph-mon[31531]: pgmap v244: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:39:44.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:44 smithi087 ceph-mon[27999]: pgmap v244: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:39:45.862 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 34.6 2024-04-22T13:39:45.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:39:45.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 -- 172.21.15.87:0/3581871889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074b40 msgr2=0x7f3d6410e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/3581871889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074b40 0x7f3d6410e2f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009900 tx=0x7f3d5002bcd0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 -- 172.21.15.87:0/3581871889 shutdown_connections 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/3581871889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d6410e830 0x7f3d64110c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/3581871889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074b40 0x7f3d6410e2f0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/3581871889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d64074220 0x7f3d64074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 -- 172.21.15.87:0/3581871889 >> 172.21.15.87:0/3581871889 conn(0x7f3d640793e0 msgr2=0x7f3d640797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 -- 172.21.15.87:0/3581871889 shutdown_connections 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.968+0000 7f3d6afbd640 1 -- 172.21.15.87:0/3581871889 wait complete. 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 Processor -- start 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 -- start start 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d64074b40 0x7f3d641a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:45.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d6410e830 0x7f3d6419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3d641143f0 con 0x7f3d64074220 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64114270 con 0x7f3d64074b40 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6afbd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64114570 con 0x7f3d6410e830 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55480/0 (socket says 172.21.15.87:55480) 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 -- 172.21.15.87:0/891415008 learned_addr learned my addr 172.21.15.87:0/891415008 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d63fff640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d64074b40 0x7f3d641a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d6955a640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d6410e830 0x7f3d6419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 -- 172.21.15.87:0/891415008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d6410e830 msgr2=0x7f3d6419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d6410e830 0x7f3d6419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 -- 172.21.15.87:0/891415008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d64074b40 msgr2=0x7f3d641a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d64074b40 0x7f3d641a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.969+0000 7f3d68d59640 1 -- 172.21.15.87:0/891415008 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6419afc0 con 0x7f3d64074220 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d68d59640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800d770 tx=0x7f3d5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:45.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58017070 con 0x7f3d64074220 2024-04-22T13:39:45.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d580054d0 con 0x7f3d64074220 2024-04-22T13:39:45.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6419b2b0 con 0x7f3d64074220 2024-04-22T13:39:45.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58004c30 con 0x7f3d64074220 2024-04-22T13:39:45.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.970+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d641a74d0 con 0x7f3d64074220 2024-04-22T13:39:45.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.971+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d64075960 con 0x7f3d64074220 2024-04-22T13:39:45.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.972+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3d58004e10 con 0x7f3d64074220 2024-04-22T13:39:45.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.973+0000 7f3d61ffb640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 0x7f3d540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:45.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.973+0000 7f3d63fff640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 0x7f3d540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:45.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.973+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f3d580d0290 con 0x7f3d64074220 2024-04-22T13:39:45.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.973+0000 7f3d63fff640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 0x7f3d540a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d50005e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:45.973 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:45.978+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d5809c970 con 0x7f3d64074220 2024-04-22T13:39:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.192+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d641099b0 con 0x7f3d64074220 2024-04-22T13:39:46.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.193+0000 7f3d61ffb640 1 -- 172.21.15.87:0/891415008 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f3d5809c270 con 0x7f3d64074220 2024-04-22T13:39:46.189 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:39:46.189 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:39:46.189 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:39:46.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 msgr2=0x7f3d540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 0x7f3d540a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d50005e40 comp rx=0 tx=0).stop 2024-04-22T13:39:46.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 msgr2=0x7f3d641a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800d770 tx=0x7f3d5800dc30 comp rx=0 tx=0).stop 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 shutdown_connections 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d540a3d20 0x7f3d540a61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d6410e830 0x7f3d6419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d64074b40 0x7f3d641a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 --2- 172.21.15.87:0/891415008 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d64074220 0x7f3d641a0680 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.195+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 >> 172.21.15.87:0/891415008 conn(0x7f3d640793e0 msgr2=0x7f3d640fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.196+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 shutdown_connections 2024-04-22T13:39:46.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.196+0000 7f3d6afbd640 1 -- 172.21.15.87:0/891415008 wait complete. 2024-04-22T13:39:46.203 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 94.6 --await' 2024-04-22T13:39:46.203 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 94.6 --await 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- 172.21.15.87:0/1205488871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8101a40 msgr2=0x7fdce8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 --2- 172.21.15.87:0/1205488871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8101a40 0x7fdce8105a80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009900 tx=0x7fdcd802bc30 comp rx=0 tx=0).stop 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- 172.21.15.87:0/1205488871 shutdown_connections 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 --2- 172.21.15.87:0/1205488871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 0x7fdce810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 --2- 172.21.15.87:0/1205488871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce8101a40 0x7fdce8105a80 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 --2- 172.21.15.87:0/1205488871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 0x7fdce8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- 172.21.15.87:0/1205488871 >> 172.21.15.87:0/1205488871 conn(0x7fdce80771d0 msgr2=0x7fdce80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- 172.21.15.87:0/1205488871 shutdown_connections 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- 172.21.15.87:0/1205488871 wait complete. 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 Processor -- start 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.315+0000 7fdceca62640 1 -- start start 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 0x7fdce81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 0x7fdce8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81121a0 con 0x7fdce819cf60 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8112020 con 0x7fdce8101090 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8112320 con 0x7fdce8106160 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55506/0 (socket says 172.21.15.87:55506) 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 -- 172.21.15.87:0/3237687829 learned_addr learned my addr 172.21.15.87:0/3237687829 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 0x7fdce8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 -- 172.21.15.87:0/3237687829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 msgr2=0x7fdce8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 0x7fdce8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 -- 172.21.15.87:0/3237687829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 msgr2=0x7fdce81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce659c640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 0x7fdce81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 0x7fdce81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 -- 172.21.15.87:0/3237687829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce819da50 con 0x7fdce819cf60 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdce6d9d640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdce000cad0 tx=0x7fdce000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdcd77fe640 1 -- 172.21.15.87:0/3237687829 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce000ae20 con 0x7fdce819cf60 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce819dd40 con 0x7fdce819cf60 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdcd77fe640 1 -- 172.21.15.87:0/3237687829 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdce0005020 con 0x7fdce819cf60 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdcd77fe640 1 -- 172.21.15.87:0/3237687829 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce000f6b0 con 0x7fdce819cf60 2024-04-22T13:39:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.316+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce81a54b0 con 0x7fdce819cf60 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcd77fe640 1 -- 172.21.15.87:0/3237687829 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdce00051d0 con 0x7fdce819cf60 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 Processor -- start 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 -- start start 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a173da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a0d40b0 con 0x55fd1a1742e0 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a085680 con 0x55fd1a1739c0 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcef64c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a091c60 con 0x55fd1a17cec0 2024-04-22T13:39:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdcd77fe640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 0x7fdcd00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdce659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdce659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55518/0 (socket says 172.21.15.87:55518) 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.319+0000 7fdce659c640 1 -- 172.21.15.87:0/3964261033 learned_addr learned my addr 172.21.15.87:0/3964261033 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce6d9d640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 0x7fdcd00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a173da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdcd77fe640 1 -- 172.21.15.87:0/3237687829 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fdce00d0820 con 0x7fdce819cf60 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 -- 172.21.15.87:0/3964261033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 msgr2=0x55fd1a17f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 -- 172.21.15.87:0/3964261033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 msgr2=0x55fd1a173da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a173da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 -- 172.21.15.87:0/3964261033 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd19eb6c90 con 0x55fd1a1742e0 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce6d9d640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 0x7fdcd00a61e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdce00ce060 tx=0x7fdce0087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdce659c640 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00bb10 tx=0x7fdcdc00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c99b65cacfedca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdcd4ff9640 1 -- 172.21.15.87:0/3964261033 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc017420 con 0x55fd1a1742e0 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdcd4ff9640 1 -- 172.21.15.87:0/3964261033 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcdc017a70 con 0x55fd1a1742e0 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.320+0000 7fdcd4ff9640 1 -- 172.21.15.87:0/3964261033 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00fa10 con 0x55fd1a1742e0 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- 172.21.15.87:0/3964261033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 msgr2=0x55fd1a17c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00bb10 tx=0x7fdcdc00d340 comp rx=0 tx=0).stop 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- 172.21.15.87:0/3964261033 shutdown_connections 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1742e0 0x55fd1a17c980 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 --2- 172.21.15.87:0/3964261033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a173da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- 172.21.15.87:0/3964261033 >> 172.21.15.87:0/3964261033 conn(0x55fd1a16e0f0 msgr2=0x55fd1a16e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- 172.21.15.87:0/3964261033 shutdown_connections 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- 172.21.15.87:0/3964261033 wait complete. 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 -- start start 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.321+0000 7fdcef64c740 1 Processor -- start 2024-04-22T13:39:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a17a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a0d7400 con 0x55fd1a1739c0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a093840 con 0x55fd1a17cec0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fd1a0dc720 con 0x55fd1a1742e0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 10 client.0 _ll_register_callbacks cb 0x55fd1a0e4ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55528/0 (socket says 172.21.15.87:55528) 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 -- 172.21.15.87:0/2228908461 learned_addr learned my addr 172.21.15.87:0/2228908461 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce659c640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a17a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 -- 172.21.15.87:0/2228908461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 msgr2=0x55fd1a17a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a17a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 -- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a17cec0 msgr2=0x55fd1a17a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 -- 172.21.15.87:0/2228908461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd19f17420 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce659c640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a17a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fdcd802c680 tx=0x7fdcd80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba7644b06b7b979 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8047030 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fdcd803a040 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcdc017a70 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdcd8005390 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 10 client.17868 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.322+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fd19daa580 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcdc017a70 con 0x55fd1a1739c0 2024-04-22T13:39:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdcd8003780 con 0x55fd1a1739c0 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 10 client.17868 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 10 client.17868 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 20 client.17868 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a18cb50 0x55fd1a18efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 4 client.17868 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd1a0e5b10 con 0x55fd1a18cb50 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.323+0000 7fdce659c640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a18cb50 0x55fd1a18efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.324+0000 7fdce659c640 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a18cb50 0x55fd1a18efb0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4119e1430fc339b server_cookie=2329ddfe96b40e6f in_seq=0 out_seq=0 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.324+0000 7fdccdffb640 10 client.17868 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.324+0000 7fdccdffb640 1 -- 172.21.15.87:0/2228908461 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd1a0e5b10 con 0x55fd1a18cb50 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.324+0000 7fdccdffb640 10 client.17868 handle_command_reply: tid=0 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.324+0000 7fdcef64c740 1 client.17868 shutdown 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 msgr2=0x55fd1a179f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fdcd802c680 tx=0x7fdcd80045e0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 shutdown_connections 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a18cb50 0x55fd1a18efb0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a17a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdccdffb640 0 client.17868 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 --2- 172.21.15.87:0/2228908461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a179f30 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 >> 172.21.15.87:0/2228908461 conn(0x55fd1a16e0f0 msgr2=0x55fd1a16fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 shutdown_connections 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 1 -- 172.21.15.87:0/2228908461 wait complete. 2024-04-22T13:39:46.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.325+0000 7fdcef64c740 20 client.17868 trim_cache size 0 max 16384 2024-04-22T13:39:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 Processor -- start 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 -- start start 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1742e0 0x55fd1a185160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81adbd0 con 0x55fd1a1739c0 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81aba40 con 0x55fd1a1742e0 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdcef64c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81ad500 con 0x55fd1a17cec0 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55532/0 (socket says 172.21.15.87:55532) 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 -- 172.21.15.87:0/1181813278 learned_addr learned my addr 172.21.15.87:0/1181813278 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce659c640 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1742e0 0x55fd1a185160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 -- 172.21.15.87:0/1181813278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 msgr2=0x55fd1a17a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 -- 172.21.15.87:0/1181813278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1742e0 msgr2=0x55fd1a185160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1742e0 0x55fd1a185160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.342+0000 7fdce5d9b640 1 -- 172.21.15.87:0/1181813278 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd1a17f620 con 0x55fd1a1739c0 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.343+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8004380 tx=0x7fdcd8005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a68fbe3db7d6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.343+0000 7fdccffff640 1 -- 172.21.15.87:0/1181813278 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8047070 con 0x55fd1a1739c0 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.343+0000 7fdccffff640 1 -- 172.21.15.87:0/1181813278 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcd8050020 con 0x55fd1a1739c0 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.343+0000 7fdccffff640 1 -- 172.21.15.87:0/1181813278 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8004a40 con 0x55fd1a1739c0 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- 172.21.15.87:0/1181813278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 msgr2=0x55fd1a17c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8004380 tx=0x7fdcd8005530 comp rx=0 tx=0).stop 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- 172.21.15.87:0/1181813278 shutdown_connections 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a17cec0 0x55fd1a17a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1742e0 0x55fd1a185160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- 172.21.15.87:0/1181813278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a1739c0 0x55fd1a17c660 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- 172.21.15.87:0/1181813278 >> 172.21.15.87:0/1181813278 conn(0x55fd1a16e0f0 msgr2=0x55fd1a16f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- 172.21.15.87:0/1181813278 shutdown_connections 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- 172.21.15.87:0/1181813278 wait complete. 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- start start 2024-04-22T13:39:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 Processor -- start 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a176390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a1768d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81ac760 con 0x55fd1a17cec0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81ac090 con 0x55fd1a1739c0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81ace30 con 0x55fd1a1742e0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdcef64c740 10 client.0 _ll_register_callbacks cb 0x7fdcd8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdce5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55534/0 (socket says 172.21.15.87:55534) 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdce5d9b640 1 -- 172.21.15.87:0/848308777 learned_addr learned my addr 172.21.15.87:0/848308777 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdce659c640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a1768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.344+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a176390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 -- 172.21.15.87:0/848308777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 msgr2=0x55fd1a1768d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a1768d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 -- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 msgr2=0x55fd1a176390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a176390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 -- 172.21.15.87:0/848308777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd1a177750 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdce5d9b640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009850 tx=0x7fdcd8045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4eed03b2199bc8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8047070 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fdcd80049d0 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcd8050020 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdcd80035f0 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 10 client.17874 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fd1a1f79d0 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcd8050020 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdcd8043b00 con 0x55fd1a17cec0 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 10 client.17874 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 10 client.17874 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 20 client.17874 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.345+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a1f8090 0x55fd1a1fa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:46.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.346+0000 7fdcef64c740 4 client.17874 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 94.6, "await": true}] 2024-04-22T13:39:46.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.346+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 94.6, "await": true}) -- 0x55fd1a1795e0 con 0x55fd1a1f8090 2024-04-22T13:39:46.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.346+0000 7fdce659c640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a1f8090 0x55fd1a1fa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:46.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.346+0000 7fdce659c640 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a1f8090 0x55fd1a1fa4b0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64247e8d9f50ac5e server_cookie=cbdb336418ac6880 in_seq=0 out_seq=0 2024-04-22T13:39:46.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.346+0000 7fdccdffb640 10 client.17874 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.359+0000 7fdccdffb640 1 -- 172.21.15.87:0/848308777 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55fd1a1795e0 con 0x55fd1a1f8090 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.359+0000 7fdccdffb640 10 client.17874 handle_command_reply: tid=0 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 client.17874 shutdown 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 msgr2=0x55fd1a176f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009850 tx=0x7fdcd8045040 comp rx=0 tx=0).stop 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 shutdown_connections 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fd1a1f8090 0x55fd1a1fa4b0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fd1a17cec0 0x55fd1a176f50 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fd1a1742e0 0x55fd1a1768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 --2- 172.21.15.87:0/848308777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd1a1739c0 0x55fd1a176390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 >> 172.21.15.87:0/848308777 conn(0x55fd1a16e0f0 msgr2=0x55fd1a16f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdccdffb640 0 client.17874 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 shutdown_connections 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 1 -- 172.21.15.87:0/848308777 wait complete. 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdcef64c740 20 client.17874 trim_cache size 0 max 16384 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 30, 2024-04-22T13:39:46.357 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "f6f7ac25": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 30, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 94.6, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:39:46.358 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:39:46.359 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 msgr2=0x7fdcd00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 0x7fdcd00a61e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdce00ce060 tx=0x7fdce0087040 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 msgr2=0x7fdce819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fdce000cad0 tx=0x7fdce000cf90 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 shutdown_connections 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdcd00a3d20 0x7fdcd00a61e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdce819cf60 0x7fdce819d410 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdce8106160 0x7fdce8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 --2- 172.21.15.87:0/3237687829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdce8101090 0x7fdce81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.360+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 >> 172.21.15.87:0/3237687829 conn(0x7fdce80771d0 msgr2=0x7fdce80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:46.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.361+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 shutdown_connections 2024-04-22T13:39:46.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:46.361+0000 7fdceca62640 1 -- 172.21.15.87:0/3237687829 wait complete. 2024-04-22T13:39:46.368 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: f6f7ac25, quiesce took 0.5 seconds. Will release after: 34.6 2024-04-22T13:39:46.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:46 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/891415008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:46.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:46 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/891415008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:46.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:46 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/891415008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:39:47.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:47 smithi120 ceph-mon[31101]: pgmap v245: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:47.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:47 smithi132 ceph-mon[31531]: pgmap v245: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:47.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:47 smithi087 ceph-mon[27999]: pgmap v245: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:49.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:49 smithi120 ceph-mon[31101]: pgmap v246: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:49.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:49 smithi132 ceph-mon[31531]: pgmap v246: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:49.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:49 smithi087 ceph-mon[27999]: pgmap v246: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:51.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:51 smithi120 ceph-mon[31101]: pgmap v247: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:51.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:51 smithi132 ceph-mon[31531]: pgmap v247: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:51.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:51 smithi087 ceph-mon[27999]: pgmap v247: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 -- 172.21.15.87:0/1228206760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 msgr2=0x7f87d010e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 --2- 172.21.15.87:0/1228206760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d010e2d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009900 tx=0x7f87c002bcd0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 -- 172.21.15.87:0/1228206760 shutdown_connections 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 --2- 172.21.15.87:0/1228206760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 0x7f87d0110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 --2- 172.21.15.87:0/1228206760 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d010e2d0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 --2- 172.21.15.87:0/1228206760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 0x7f87d00746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 -- 172.21.15.87:0/1228206760 >> 172.21.15.87:0/1228206760 conn(0x7f87d00fda20 msgr2=0x7f87d00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 -- 172.21.15.87:0/1228206760 shutdown_connections 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.701+0000 7f87d757e640 1 -- 172.21.15.87:0/1228206760 wait complete. 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 Processor -- start 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 -- start start 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 0x7f87d01a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 0x7f87d019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01143b0 con 0x7f87d0074c20 2024-04-22T13:39:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87d0114230 con 0x7f87d0074300 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d757e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87d0114530 con 0x7f87d010e810 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d4b19640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d4b19640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53892/0 (socket says 172.21.15.87:53892) 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.702+0000 7f87d4b19640 1 -- 172.21.15.87:0/741100661 learned_addr learned my addr 172.21.15.87:0/741100661 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d531a640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 0x7f87d01a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 0x7f87d019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 -- 172.21.15.87:0/741100661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 msgr2=0x7f87d019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 0x7f87d019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 -- 172.21.15.87:0/741100661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 msgr2=0x7f87d01a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 0x7f87d01a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 -- 172.21.15.87:0/741100661 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d019af90 con 0x7f87d0074c20 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d4b19640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005940 tx=0x7f87c002d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87c67fc640 1 -- 172.21.15.87:0/741100661 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0043020 con 0x7f87d0074c20 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87c67fc640 1 -- 172.21.15.87:0/741100661 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c002dd10 con 0x7f87d0074c20 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d019b220 con 0x7f87d0074c20 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87c67fc640 1 -- 172.21.15.87:0/741100661 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0003b80 con 0x7f87d0074c20 2024-04-22T13:39:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.703+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d01a7540 con 0x7f87d0074c20 2024-04-22T13:39:52.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.705+0000 7f87c67fc640 1 -- 172.21.15.87:0/741100661 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f87c0046020 con 0x7f87d0074c20 2024-04-22T13:39:52.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 Processor -- start 2024-04-22T13:39:52.710 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 -- start start 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a1502fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a150e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5624a14678d0 con 0x5624a15038d0 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87c67fc640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 0x7f87bc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624a1404e80 con 0x5624a150c0e0 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87da168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5624a1463890 con 0x5624a1502b60 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53904/0 (socket says 172.21.15.87:53904) 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 -- 172.21.15.87:0/2015192843 learned_addr learned my addr 172.21.15.87:0/2015192843 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d4b19640 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a150e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 0x7f87bc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d531a640 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a1502fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87c67fc640 1 -- 172.21.15.87:0/741100661 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f87c00f29b0 con 0x7f87d0074c20 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 -- 172.21.15.87:0/2015192843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 msgr2=0x5624a1502fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.706+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a1502fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87d5b1b640 1 -- 172.21.15.87:0/2015192843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 msgr2=0x5624a150e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a150e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87d5b1b640 1 -- 172.21.15.87:0/2015192843 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a1426320 con 0x5624a15038d0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f87c8012c30 tx=0x7f87c800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e223b7c4a5606666 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 0x7f87bc0a6280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f87c800fc30 tx=0x7f87c801a000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87bb7fe640 1 -- 172.21.15.87:0/2015192843 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8020070 con 0x5624a15038d0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87bb7fe640 1 -- 172.21.15.87:0/2015192843 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c800aa00 con 0x5624a15038d0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.707+0000 7f87bb7fe640 1 -- 172.21.15.87:0/2015192843 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c80113e0 con 0x5624a15038d0 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- 172.21.15.87:0/2015192843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 msgr2=0x5624a1502620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f87c8012c30 tx=0x7f87c800f730 comp rx=0 tx=0).stop 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- 172.21.15.87:0/2015192843 shutdown_connections 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a150e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a1502fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 --2- 172.21.15.87:0/2015192843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a15038d0 0x5624a1502620 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- 172.21.15.87:0/2015192843 >> 172.21.15.87:0/2015192843 conn(0x5624a14c9c90 msgr2=0x5624a14fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- 172.21.15.87:0/2015192843 shutdown_connections 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- 172.21.15.87:0/2015192843 wait complete. 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 -- start start 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.708+0000 7f87da168740 1 Processor -- start 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 0x5624a150bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1508ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5624a150edb0 con 0x5624a150c0e0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624a13efeb0 con 0x5624a1502b60 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5624a150ef30 con 0x5624a15038d0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 10 client.0 _ll_register_callbacks cb 0x5624a141cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53910/0 (socket says 172.21.15.87:53910) 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 -- 172.21.15.87:0/1427077388 learned_addr learned my addr 172.21.15.87:0/1427077388 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d4b19640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1508ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d531a640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 0x5624a150bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 -- 172.21.15.87:0/1427077388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 msgr2=0x5624a1508ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1508ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 -- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 msgr2=0x5624a150bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 0x5624a150bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 -- 172.21.15.87:0/1427077388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a117a210 con 0x5624a150c0e0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d531a640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 0x5624a150bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f87c801ccc0 tx=0x7f87c801c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea986303caf20d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8020050 con 0x5624a150c0e0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f87c8004a00 con 0x5624a150c0e0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.709+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87c800aa00 con 0x5624a150c0e0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87c80113e0 con 0x5624a150c0e0 2024-04-22T13:39:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 10 client.17886 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5624a1246620 con 0x5624a150c0e0 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c800aa00 con 0x5624a150c0e0 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f87c801b070 con 0x5624a150c0e0 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 10 client.17886 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 10 client.17886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 20 client.17886 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a151c0e0 0x5624a151e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 4 client.17886 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624a147b880 con 0x5624a151c0e0 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.710+0000 7f87d531a640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a151c0e0 0x5624a151e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.711+0000 7f87b8ff9640 10 client.17886 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.711+0000 7f87d531a640 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a151c0e0 0x5624a151e4d0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a47bf942fef70c server_cookie=181bbf0f2aca835d in_seq=0 out_seq=0 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.711+0000 7f87b8ff9640 1 -- 172.21.15.87:0/1427077388 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624a147b880 con 0x5624a151c0e0 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.711+0000 7f87b8ff9640 10 client.17886 handle_command_reply: tid=0 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.711+0000 7f87da168740 1 client.17886 shutdown 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 msgr2=0x5624a1509600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f87c801ccc0 tx=0x7f87c801c660 comp rx=0 tx=0).stop 2024-04-22T13:39:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 shutdown_connections 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a151c0e0 0x5624a151e4d0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a1509600 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1508ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87b8ff9640 0 client.17886 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 --2- 172.21.15.87:0/1427077388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a1502b60 0x5624a150bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 >> 172.21.15.87:0/1427077388 conn(0x5624a14c9c90 msgr2=0x5624a14ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 shutdown_connections 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 1 -- 172.21.15.87:0/1427077388 wait complete. 2024-04-22T13:39:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.712+0000 7f87da168740 20 client.17886 trim_cache size 0 max 16384 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 Processor -- start 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 -- start start 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1517cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a1523b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01b2490 con 0x5624a1502b60 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01afd60 con 0x5624a150c0e0 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87da168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01adbd0 con 0x5624a15038d0 2024-04-22T13:39:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53924/0 (socket says 172.21.15.87:53924) 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 -- 172.21.15.87:0/2101338386 learned_addr learned my addr 172.21.15.87:0/2101338386 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a1523b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 -- 172.21.15.87:0/2101338386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 msgr2=0x5624a1517cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d4b19640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1517cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1517cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 -- 172.21.15.87:0/2101338386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 msgr2=0x5624a1523b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a1523b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 -- 172.21.15.87:0/2101338386 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a1501e30 con 0x5624a1502b60 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a1523b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d4b19640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1517cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.727+0000 7f87d531a640 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00eda0 tx=0x7f87cc00e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa5087367084b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87baffd640 1 -- 172.21.15.87:0/2101338386 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0130d0 con 0x5624a1502b60 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87baffd640 1 -- 172.21.15.87:0/2101338386 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc013270 con 0x5624a1502b60 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87baffd640 1 -- 172.21.15.87:0/2101338386 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc011ab0 con 0x5624a1502b60 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 -- 172.21.15.87:0/2101338386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 msgr2=0x5624a15046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00eda0 tx=0x7f87cc00e3b0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 -- 172.21.15.87:0/2101338386 shutdown_connections 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a150c0e0 0x5624a1523b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a15038d0 0x5624a1517cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 --2- 172.21.15.87:0/2101338386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a1502b60 0x5624a15046d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 -- 172.21.15.87:0/2101338386 >> 172.21.15.87:0/2101338386 conn(0x5624a14c9c90 msgr2=0x5624a1508020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 -- 172.21.15.87:0/2101338386 shutdown_connections 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.728+0000 7f87da168740 1 -- 172.21.15.87:0/2101338386 wait complete. 2024-04-22T13:39:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 -- start start 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 Processor -- start 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a150d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a15038d0 0x5624a150d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01ae220 con 0x5624a150c0e0 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01af690 con 0x5624a15038d0 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87d01ae8f0 con 0x5624a1502b60 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87da168740 10 client.0 _ll_register_callbacks cb 0x7f87c00be870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53940/0 (socket says 172.21.15.87:53940) 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 -- 172.21.15.87:0/3113194403 learned_addr learned my addr 172.21.15.87:0/3113194403 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d4b19640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a15038d0 0x5624a150d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d531a640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a150d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 -- 172.21.15.87:0/3113194403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 msgr2=0x5624a150d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a150d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 -- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a15038d0 msgr2=0x5624a150d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a15038d0 0x5624a150d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d5b1b640 1 -- 172.21.15.87:0/3113194403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a1512d10 con 0x5624a150c0e0 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.729+0000 7f87d531a640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a150d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87d5b1b640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f87c800a830 tx=0x7f87c8011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acb7eaa80a59293 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c803c030 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f87c8033030 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87cc013270 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87c8020070 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 10 client.17892 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5624a1582a00 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc013270 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f87c801b040 con 0x5624a150c0e0 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 10 client.17892 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 10 client.17892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 20 client.17892 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a15831a0 0x5624a15855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 4 client.17892 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:39:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.730+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624a14073d0 con 0x5624a15831a0 2024-04-22T13:39:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.731+0000 7f87d531a640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a15831a0 0x5624a15855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.731+0000 7f87d531a640 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a15831a0 0x5624a15855c0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706486646dd60215 server_cookie=329390cea1c23e5e in_seq=0 out_seq=0 2024-04-22T13:39:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.731+0000 7f87b8ff9640 10 client.17892 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87b8ff9640 1 -- 172.21.15.87:0/3113194403 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5624a14073d0 con 0x5624a15831a0 2024-04-22T13:39:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87b8ff9640 10 client.17892 handle_command_reply: tid=0 2024-04-22T13:39:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 client.17892 shutdown 2024-04-22T13:39:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 msgr2=0x5624a15127d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f87c800a830 tx=0x7f87c8011f10 comp rx=0 tx=0).stop 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 shutdown_connections 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624a15831a0 0x5624a15855c0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624a150c0e0 0x5624a15127d0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624a15038d0 0x5624a150d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87b8ff9640 0 client.17892 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 --2- 172.21.15.87:0/3113194403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624a1502b60 0x5624a150d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.736+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 >> 172.21.15.87:0/3113194403 conn(0x5624a14c9c90 msgr2=0x5624a14fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 shutdown_connections 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87da168740 1 -- 172.21.15.87:0/3113194403 wait complete. 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87da168740 20 client.17892 trim_cache size 0 max 16384 2024-04-22T13:39:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:39:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:39:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:39:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:39:52.737 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 msgr2=0x7f87bc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 0x7f87bc0a6280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f87c800fc30 tx=0x7f87c801a000 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 msgr2=0x7f87d01a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005940 tx=0x7f87c002d910 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 shutdown_connections 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f87bc0a3dc0 0x7f87bc0a6280 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87d010e810 0x7f87d019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87d0074c20 0x7f87d01a0b00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 --2- 172.21.15.87:0/741100661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87d0074300 0x7f87d01a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 >> 172.21.15.87:0/741100661 conn(0x7f87d00fda20 msgr2=0x7f87d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 shutdown_connections 2024-04-22T13:39:52.738 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:39:52.737+0000 7f87d757e640 1 -- 172.21.15.87:0/741100661 wait complete. 2024-04-22T13:39:53.515 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:53 smithi132 ceph-mon[31531]: pgmap v248: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:53.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:53 smithi120 ceph-mon[31101]: pgmap v248: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:53.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:53 smithi087 ceph-mon[27999]: pgmap v248: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:55 smithi120 ceph-mon[31101]: pgmap v249: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:39:55.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:55 smithi132 ceph-mon[31531]: pgmap v249: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:39:55.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:55 smithi087 ceph-mon[27999]: pgmap v249: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:39:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:57 smithi120 ceph-mon[31101]: pgmap v250: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:57.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:57 smithi132 ceph-mon[31531]: pgmap v250: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:57.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:57 smithi087 ceph-mon[27999]: pgmap v250: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:39:59.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:39:59 smithi120 ceph-mon[31101]: pgmap v251: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:59.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:39:59 smithi132 ceph-mon[31531]: pgmap v251: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:59.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:39:59 smithi087 ceph-mon[27999]: pgmap v251: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:39:59.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:39:59.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 -- 172.21.15.87:0/160460397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81021a0 msgr2=0x7f58d8102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/160460397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81021a0 0x7f58d8102580 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009ca0 tx=0x7f58c802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 -- 172.21.15.87:0/160460397 shutdown_connections 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/160460397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/160460397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d8102b50 0x7f58d8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/160460397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81021a0 0x7f58d8102580 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 -- 172.21.15.87:0/160460397 >> 172.21.15.87:0/160460397 conn(0x7f58d80793e0 msgr2=0x7f58d80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 -- 172.21.15.87:0/160460397 shutdown_connections 2024-04-22T13:39:59.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 -- 172.21.15.87:0/160460397 wait complete. 2024-04-22T13:39:59.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.912+0000 7f58dd2f1640 1 Processor -- start 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 -- start start 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d81021a0 0x7f58d819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81122b0 con 0x7f58d81985f0 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8112130 con 0x7f58d8107270 2024-04-22T13:39:59.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58dd2f1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8112430 con 0x7f58d81021a0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51490/0 (socket says 172.21.15.87:51490) 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 -- 172.21.15.87:0/947246474 learned_addr learned my addr 172.21.15.87:0/947246474 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d81021a0 0x7f58d819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d67fc640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 -- 172.21.15.87:0/947246474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d81021a0 msgr2=0x7f58d819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d81021a0 0x7f58d819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 -- 172.21.15.87:0/947246474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 msgr2=0x7f58d819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 -- 172.21.15.87:0/947246474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d8199110 con 0x7f58d81985f0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d67fc640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.913+0000 7f58d77fe640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009fd0 tx=0x7f58d000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.914+0000 7f58bffff640 1 -- 172.21.15.87:0/947246474 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0011830 con 0x7f58d81985f0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.914+0000 7f58bffff640 1 -- 172.21.15.87:0/947246474 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d0016070 con 0x7f58d81985f0 2024-04-22T13:39:59.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.914+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d8199400 con 0x7f58d81985f0 2024-04-22T13:39:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.914+0000 7f58bffff640 1 -- 172.21.15.87:0/947246474 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0010de0 con 0x7f58d81985f0 2024-04-22T13:39:59.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.914+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d81a5600 con 0x7f58d81985f0 2024-04-22T13:39:59.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58bffff640 1 -- 172.21.15.87:0/947246474 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f58d00119d0 con 0x7f58d81985f0 2024-04-22T13:39:59.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 Processor -- start 2024-04-22T13:39:59.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 -- start start 2024-04-22T13:39:59.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a3624c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58bffff640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 0x7f58c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36255540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563a361b11e0 con 0x563a36255a80 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a3614e390 con 0x563a3624cea0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.916+0000 7f58dffb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563a361ad1a0 con 0x563a3624c580 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 0x7f58c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51502/0 (socket says 172.21.15.87:51502) 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 -- 172.21.15.87:0/43913745 learned_addr learned my addr 172.21.15.87:0/43913745 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d67fc640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36255540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58bffff640 1 -- 172.21.15.87:0/947246474 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f58d00cfdd0 con 0x7f58d81985f0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d67fc640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a3624c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 -- 172.21.15.87:0/43913745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 msgr2=0x563a3624c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a3624c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 -- 172.21.15.87:0/43913745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 msgr2=0x563a36255540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36255540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 -- 172.21.15.87:0/43913745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a35f8d7d0 con 0x563a36255a80 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d67fc640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a3624c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 0x7f58c00a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009430 tx=0x7f58c8005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58d77fe640 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f58d009bbc0 tx=0x7f58d0010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8296d9e1f11bb12d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58bd7fa640 1 -- 172.21.15.87:0/43913745 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d009b820 con 0x563a36255a80 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58bd7fa640 1 -- 172.21.15.87:0/43913745 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d009c5c0 con 0x563a36255a80 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.917+0000 7f58bd7fa640 1 -- 172.21.15.87:0/43913745 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d00b5640 con 0x563a36255a80 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- 172.21.15.87:0/43913745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 msgr2=0x563a36257e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f58d009bbc0 tx=0x7f58d0010660 comp rx=0 tx=0).stop 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- 172.21.15.87:0/43913745 shutdown_connections 2024-04-22T13:39:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36257e70 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36255540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 --2- 172.21.15.87:0/43913745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a3624c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- 172.21.15.87:0/43913745 >> 172.21.15.87:0/43913745 conn(0x563a36246cb0 msgr2=0x563a36247090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- 172.21.15.87:0/43913745 shutdown_connections 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- 172.21.15.87:0/43913745 wait complete. 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.918+0000 7f58dffb6740 1 -- start start 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 Processor -- start 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36252b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36253050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563a36258870 con 0x563a36255a80 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a3616a2f0 con 0x563a3624cea0 2024-04-22T13:39:59.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563a362589f0 con 0x563a3624c580 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58dffb6740 10 client.0 _ll_register_callbacks cb 0x563a36177580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51504/0 (socket says 172.21.15.87:51504) 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 -- 172.21.15.87:0/1468685997 learned_addr learned my addr 172.21.15.87:0/1468685997 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36252b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d67fc640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36253050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 -- 172.21.15.87:0/1468685997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 msgr2=0x563a36252b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36252b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 -- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 msgr2=0x563a36253050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36253050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 -- 172.21.15.87:0/1468685997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a35ec33f0 con 0x563a36255a80 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58d77fe640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f58d0098320 tx=0x7f58d00104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd544afb5cbfb1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58baff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.919+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d00b5820 con 0x563a36255a80 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f58d00b0960 con 0x563a36255a80 2024-04-22T13:39:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d009c5c0 con 0x563a36255a80 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f58d009b820 con 0x563a36255a80 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 10 client.17904 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563a35e830f0 con 0x563a36255a80 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d009c5c0 con 0x563a36255a80 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f58d00a46a0 con 0x563a36255a80 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 10 client.17904 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 10 client.17904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 20 client.17904 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36265960 0x563a36267d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 4 client.17904 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a361bec00 con 0x563a36265960 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.920+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36265960 0x563a36267d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.921+0000 7f58baff5640 10 client.17904 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.921+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36265960 0x563a36267d50 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=913be7d7a2f6ab4f server_cookie=7eece9c0ec1b1f3a in_seq=0 out_seq=0 2024-04-22T13:39:59.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.921+0000 7f58baff5640 1 -- 172.21.15.87:0/1468685997 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a361bec00 con 0x563a36265960 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.921+0000 7f58baff5640 10 client.17904 handle_command_reply: tid=0 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 client.17904 shutdown 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 msgr2=0x563a36253590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f58d0098320 tx=0x7f58d00104c0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 shutdown_connections 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36265960 0x563a36267d50 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36255a80 0x563a36253590 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36253050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58baff5640 0 client.17904 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 --2- 172.21.15.87:0/1468685997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36252b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 >> 172.21.15.87:0/1468685997 conn(0x563a36246cb0 msgr2=0x563a36248a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 shutdown_connections 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 1 -- 172.21.15.87:0/1468685997 wait complete. 2024-04-22T13:39:59.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.922+0000 7f58dffb6740 20 client.17904 trim_cache size 0 max 16384 2024-04-22T13:39:59.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 Processor -- start 2024-04-22T13:39:59.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 -- start start 2024-04-22T13:39:59.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a36246cb0 0x563a36255240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36247090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81b04e0 con 0x563a3624cea0 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81addb0 con 0x563a36246cb0 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.939+0000 7f58dffb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81abc20 con 0x563a3624c580 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51518/0 (socket says 172.21.15.87:51518) 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 -- 172.21.15.87:0/3553784931 learned_addr learned my addr 172.21.15.87:0/3553784931 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d67fc640 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36247090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 -- 172.21.15.87:0/3553784931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 msgr2=0x563a36247090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36247090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 -- 172.21.15.87:0/3553784931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a36246cb0 msgr2=0x563a36255240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a36246cb0 0x563a36255240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 -- 172.21.15.87:0/3553784931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3624ccd0 con 0x563a3624cea0 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58d77fe640 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f58d0098890 tx=0x7f58d00cd720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab3098ed5a86128 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58bcff9640 1 -- 172.21.15.87:0/3553784931 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d009c5c0 con 0x563a3624cea0 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58bcff9640 1 -- 172.21.15.87:0/3553784931 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d009bec0 con 0x563a3624cea0 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.940+0000 7f58bcff9640 1 -- 172.21.15.87:0/3553784931 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d00b0960 con 0x563a3624cea0 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- 172.21.15.87:0/3553784931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 msgr2=0x563a362533b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f58d0098890 tx=0x7f58d00cd720 comp rx=0 tx=0).stop 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- 172.21.15.87:0/3553784931 shutdown_connections 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a3624cea0 0x563a362533b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36247090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3553784931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a36246cb0 0x563a36255240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- 172.21.15.87:0/3553784931 >> 172.21.15.87:0/3553784931 conn(0x563a3624e4e0 msgr2=0x563a36248100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- 172.21.15.87:0/3553784931 shutdown_connections 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- 172.21.15.87:0/3553784931 wait complete. 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- start start 2024-04-22T13:39:59.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 Processor -- start 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36268320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36268860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81ac270 con 0x563a36246cb0 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81ad6e0 con 0x563a3624cea0 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58d81ac940 con 0x563a3624c580 2024-04-22T13:39:59.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58dffb6740 10 client.0 _ll_register_callbacks cb 0x7f58cc00c750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58d6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58d6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51528/0 (socket says 172.21.15.87:51528) 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.941+0000 7f58d6ffd640 1 -- 172.21.15.87:0/3148273506 learned_addr learned my addr 172.21.15.87:0/3148273506 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d67fc640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36268320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 -- 172.21.15.87:0/3148273506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 msgr2=0x563a36268320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36268320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 -- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 msgr2=0x563a36268860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d77fe640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36268860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36268860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 -- 172.21.15.87:0/3148273506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a36268e50 con 0x563a36246cb0 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d67fc640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36268320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58d6ffd640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f58c802c630 tx=0x7f58c80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2c82d475d464c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:39:59.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8039040 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f58c8003d70 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d009bec0 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f58c805e050 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58dffb6740 10 client.17910 fetch_fsmap learned FSMap version 23 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563a362cb8c0 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d009bec0 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.942+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f58c80504b0 con 0x563a36246cb0 2024-04-22T13:39:59.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 10 client.17910 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 10 client.17910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 20 client.17910 populate_metadata read hostname 'smithi087' 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36255a80 0x563a362ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 4 client.17910 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a361b8a70 con 0x563a36255a80 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58d67fc640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36255a80 0x563a362ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58d67fc640 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36255a80 0x563a362ce020 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=954ea08b64e42f85 server_cookie=8e89d2223a608194 in_seq=0 out_seq=0 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.943+0000 7f58baff5640 10 client.17910 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58baff5640 1 -- 172.21.15.87:0/3148273506 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563a361b8a70 con 0x563a36255a80 2024-04-22T13:39:59.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58baff5640 10 client.17910 handle_command_reply: tid=0 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 client.17910 shutdown 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 msgr2=0x563a3626af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f58c802c630 tx=0x7f58c80095c0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 shutdown_connections 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563a36255a80 0x563a362ce020 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a3624cea0 0x563a36268860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563a3624c580 0x563a36268320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 --2- 172.21.15.87:0/3148273506 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563a36246cb0 0x563a3626af40 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58baff5640 0 client.17910 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:39:59.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 >> 172.21.15.87:0/3148273506 conn(0x563a3624e4e0 msgr2=0x563a36245d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 shutdown_connections 2024-04-22T13:39:59.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 1 -- 172.21.15.87:0/3148273506 wait complete. 2024-04-22T13:39:59.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dffb6740 20 client.17910 trim_cache size 0 max 16384 2024-04-22T13:39:59.951 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:39:59.951 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:39:59.951 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:39:59.951 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 msgr2=0x7f58c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 0x7f58c00a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009430 tx=0x7f58c8005c50 comp rx=0 tx=0).stop 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 msgr2=0x7f58d8198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009fd0 tx=0x7f58d000b8e0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 shutdown_connections 2024-04-22T13:39:59.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c00a3d20 0x7f58c00a61e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58d81985f0 0x7f58d8198aa0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d8107270 0x7f58d819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 --2- 172.21.15.87:0/947246474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58d81021a0 0x7f58d819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.944+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 >> 172.21.15.87:0/947246474 conn(0x7f58d80793e0 msgr2=0x7f58d80fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.945+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 shutdown_connections 2024-04-22T13:39:59.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:39:59.945+0000 7f58dd2f1640 1 -- 172.21.15.87:0/947246474 wait complete. 2024-04-22T13:39:59.954 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:39:59.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- 172.21.15.87:0/860197112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec41072f0 msgr2=0x7f1ec410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 --2- 172.21.15.87:0/860197112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec41072f0 0x7f1ec410ee70 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009ca0 tx=0x7f1ebc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- 172.21.15.87:0/860197112 shutdown_connections 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 --2- 172.21.15.87:0/860197112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec41072f0 0x7f1ec410ee70 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 --2- 172.21.15.87:0/860197112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec406b2a0 0x7f1ec4106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 --2- 172.21.15.87:0/860197112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec406a8f0 0x7f1ec406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- 172.21.15.87:0/860197112 >> 172.21.15.87:0/860197112 conn(0x7f1ec4077e30 msgr2=0x7f1ec4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- 172.21.15.87:0/860197112 shutdown_connections 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- 172.21.15.87:0/860197112 wait complete. 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 Processor -- start 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.058+0000 7f1ecb214640 1 -- start start 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec41072f0 0x7f1ec4198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 0x7f1ec419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec41121f0 con 0x7f1ec406a8f0 2024-04-22T13:40:00.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4112070 con 0x7f1ec419cfb0 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4112370 con 0x7f1ec41072f0 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51542/0 (socket says 172.21.15.87:51542) 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 -- 172.21.15.87:0/3071685779 learned_addr learned my addr 172.21.15.87:0/3071685779 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec97b1640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 0x7f1ec419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec3fff640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec41072f0 0x7f1ec4198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 -- 172.21.15.87:0/3071685779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec41072f0 msgr2=0x7f1ec4198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec41072f0 0x7f1ec4198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 -- 172.21.15.87:0/3071685779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 msgr2=0x7f1ec419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 0x7f1ec419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 -- 172.21.15.87:0/3071685779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec419daa0 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec97b1640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 0x7f1ec419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec8fb0640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d950 tx=0x7f1eb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8017030 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eb8004020 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec419dd90 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.059+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80047a0 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.060+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec41a5580 con 0x7f1ec406a8f0 2024-04-22T13:40:00.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.061+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec4069230 con 0x7f1ec406a8f0 2024-04-22T13:40:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.061+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1eb8009a30 con 0x7f1ec406a8f0 2024-04-22T13:40:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.062+0000 7f1ec1ffb640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 0x7f1eb40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.062+0000 7f1ec3fff640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 0x7f1eb40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.062+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f1eb80d02d0 con 0x7f1ec406a8f0 2024-04-22T13:40:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.063+0000 7f1ec3fff640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 0x7f1eb40a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005e00 tx=0x7f1eb0005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.067+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1eb809c9b0 con 0x7f1ec406a8f0 2024-04-22T13:40:00.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.299+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ec4199a40 con 0x7f1ec406a8f0 2024-04-22T13:40:00.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.300+0000 7f1ec1ffb640 1 -- 172.21.15.87:0/3071685779 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f1eb8096e20 con 0x7f1ec406a8f0 2024-04-22T13:40:00.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:40:00.296 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:40:00.296 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:40:00.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 msgr2=0x7f1eb40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 0x7f1eb40a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005e00 tx=0x7f1eb0005c50 comp rx=0 tx=0).stop 2024-04-22T13:40:00.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 msgr2=0x7f1ec41986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d950 tx=0x7f1eb800de10 comp rx=0 tx=0).stop 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 shutdown_connections 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1eb40a3d20 0x7f1eb40a61e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ec419cfb0 0x7f1ec419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec41072f0 0x7f1ec4198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 --2- 172.21.15.87:0/3071685779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ec406a8f0 0x7f1ec41986f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 >> 172.21.15.87:0/3071685779 conn(0x7f1ec4077e30 msgr2=0x7f1ec40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 shutdown_connections 2024-04-22T13:40:00.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.302+0000 7f1ecb214640 1 -- 172.21.15.87:0/3071685779 wait complete. 2024-04-22T13:40:00.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3651286952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101072b0 msgr2=0x7f5b1010ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3651286952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101072b0 0x7f5b1010ee80 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009ca0 tx=0x7f5b0802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3651286952 shutdown_connections 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3651286952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101072b0 0x7f5b1010ee80 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3651286952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b10102b90 0x7f5b10106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3651286952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b101021e0 0x7f5b101025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3651286952 >> 172.21.15.87:0/3651286952 conn(0x7f5b100fda60 msgr2=0x7f5b100ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3651286952 shutdown_connections 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.417+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3651286952 wait complete. 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 Processor -- start 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 -- start start 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b101021e0 0x7f5b1019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b101072b0 0x7f5b1019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10112200 con 0x7f5b101985a0 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10112080 con 0x7f5b101021e0 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b14f7e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10112380 con 0x7f5b101072b0 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51554/0 (socket says 172.21.15.87:51554) 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 -- 172.21.15.87:0/3753024557 learned_addr learned my addr 172.21.15.87:0/3753024557 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b101021e0 0x7f5b1019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 -- 172.21.15.87:0/3753024557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b101072b0 msgr2=0x7f5b1019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b101072b0 0x7f5b1019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 -- 172.21.15.87:0/3753024557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b101021e0 msgr2=0x7f5b1019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b101021e0 0x7f5b1019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.418+0000 7f5b0ed9d640 1 -- 172.21.15.87:0/3753024557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b101990c0 con 0x7f5b101985a0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5b0ed9d640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005e30 tx=0x7f5b08034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5aff7fe640 1 -- 172.21.15.87:0/3753024557 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08039040 con 0x7f5b101985a0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b10199350 con 0x7f5b101985a0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5aff7fe640 1 -- 172.21.15.87:0/3753024557 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b08044070 con 0x7f5b101985a0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5aff7fe640 1 -- 172.21.15.87:0/3753024557 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08003e60 con 0x7f5b101985a0 2024-04-22T13:40:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.419+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b101a5550 con 0x7f5b101985a0 2024-04-22T13:40:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.421+0000 7f5aff7fe640 1 -- 172.21.15.87:0/3753024557 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b080324c0 con 0x7f5b101985a0 2024-04-22T13:40:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.421+0000 7f5b17b68740 1 Processor -- start 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.421+0000 7f5b17b68740 1 -- start start 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f3980518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f398051e80 0x55f39805a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f397fb61f0 con 0x55f39805aa00 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f397f534b0 con 0x55f3980514d0 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b17b68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f397fb21b0 con 0x55f398051e80 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5aff7fe640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 0x7f5af80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51568/0 (socket says 172.21.15.87:51568) 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1936124768 learned_addr learned my addr 172.21.15.87:0/1936124768 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f3980518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f398051e80 0x55f39805a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0ed9d640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 0x7f5af80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5aff7fe640 1 -- 172.21.15.87:0/3753024557 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f5b080f29d0 con 0x7f5b101985a0 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1936124768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f398051e80 msgr2=0x55f39805a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f398051e80 0x55f39805a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1936124768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 msgr2=0x55f3980518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f3980518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1936124768 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f397f451b0 con 0x55f39805aa00 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5b04012740 tx=0x7f5b0400d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1ec33a5ccc120b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5afcff9640 1 -- 172.21.15.87:0/1936124768 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0400a650 con 0x55f39805aa00 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.422+0000 7f5b0ed9d640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 0x7f5af80a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5b080be760 tx=0x7f5b080095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5afcff9640 1 -- 172.21.15.87:0/1936124768 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0400aca0 con 0x55f39805aa00 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5afcff9640 1 -- 172.21.15.87:0/1936124768 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04010b60 con 0x55f39805aa00 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 -- 172.21.15.87:0/1936124768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 msgr2=0x55f39805ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5b04012740 tx=0x7f5b0400d340 comp rx=0 tx=0).stop 2024-04-22T13:40:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 -- 172.21.15.87:0/1936124768 shutdown_connections 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f39805ce50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f398051e80 0x55f39805a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1936124768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f3980518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 -- 172.21.15.87:0/1936124768 >> 172.21.15.87:0/1936124768 conn(0x55f39804bba0 msgr2=0x55f39804bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 -- 172.21.15.87:0/1936124768 shutdown_connections 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.423+0000 7f5b17b68740 1 -- 172.21.15.87:0/1936124768 wait complete. 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 -- start start 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 Processor -- start 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f39805a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f398051e80 0x55f398057710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f39805d970 con 0x55f39805aa00 2024-04-22T13:40:00.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f397f6f500 con 0x55f398051e80 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f39805daf0 con 0x55f3980514d0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b17b68740 10 client.0 _ll_register_callbacks cb 0x55f397f3db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51570/0 (socket says 172.21.15.87:51570) 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1242041706 learned_addr learned my addr 172.21.15.87:0/1242041706 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f39805a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f398051e80 0x55f398057710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 msgr2=0x55f39805a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f39805a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1242041706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f398051e80 msgr2=0x55f398057710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f398051e80 0x55f398057710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 -- 172.21.15.87:0/1242041706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f397cc83f0 con 0x55f39805aa00 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5b04005d80 tx=0x7f5b0400df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6961b607f6666aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.424+0000 7f5af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04026040 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f5b040108e0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b0400aca0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5b04010ae0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 10 client.17928 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f397d927d0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0400aca0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5b0402f4e0 con 0x55f39805aa00 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 10 client.17928 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 10 client.17928 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 20 client.17928 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f39806ab20 0x55f39806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 4 client.17928 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f397f55930 con 0x55f39806ab20 2024-04-22T13:40:00.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.425+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f39806ab20 0x55f39806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.426+0000 7f5af5ffb640 10 client.17928 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.426+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f39806ab20 0x55f39806cf10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=525a56e58895e9e9 server_cookie=aae4435434722ed in_seq=0 out_seq=0 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5af5ffb640 1 -- 172.21.15.87:0/1242041706 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f397f55930 con 0x55f39806ab20 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5af5ffb640 10 client.17928 handle_command_reply: tid=0 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 client.17928 shutdown 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 msgr2=0x55f398057d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5b04005d80 tx=0x7f5b0400df20 comp rx=0 tx=0).stop 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 shutdown_connections 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f39806ab20 0x55f39806cf10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f39805aa00 0x55f398057d20 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f398051e80 0x55f398057710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1242041706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f39805a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5af5ffb640 0 client.17928 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:00.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 >> 172.21.15.87:0/1242041706 conn(0x55f39804bba0 msgr2=0x55f39804d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 shutdown_connections 2024-04-22T13:40:00.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 1 -- 172.21.15.87:0/1242041706 wait complete. 2024-04-22T13:40:00.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.427+0000 7f5b17b68740 20 client.17928 trim_cache size 0 max 16384 2024-04-22T13:40:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 Processor -- start 2024-04-22T13:40:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 -- start start 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f398066830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f39805aa00 0x55f398059e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101b0390 con 0x55f398051e80 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101adc60 con 0x55f3980514d0 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b17b68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101abad0 con 0x55f39805aa00 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.440+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51578/0 (socket says 172.21.15.87:51578) 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 -- 172.21.15.87:0/1420372644 learned_addr learned my addr 172.21.15.87:0/1420372644 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f398066830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 -- 172.21.15.87:0/1420372644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f39805aa00 msgr2=0x55f398059e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f39805aa00 0x55f398059e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 -- 172.21.15.87:0/1420372644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 msgr2=0x55f398066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f398066830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 -- 172.21.15.87:0/1420372644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f398051c20 con 0x55f398051e80 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f398066830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000ae00 tx=0x7f5b0000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20a33d62f7d4844 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5af7fff640 1 -- 172.21.15.87:0/1420372644 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00004220 con 0x55f398051e80 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5af7fff640 1 -- 172.21.15.87:0/1420372644 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b00024070 con 0x55f398051e80 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.441+0000 7f5af7fff640 1 -- 172.21.15.87:0/1420372644 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00004dd0 con 0x55f398051e80 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- 172.21.15.87:0/1420372644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 msgr2=0x55f39804dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000ae00 tx=0x7f5b0000a2d0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- 172.21.15.87:0/1420372644 shutdown_connections 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f39805aa00 0x55f398059e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39804dac0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- 172.21.15.87:0/1420372644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f3980514d0 0x55f398066830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- 172.21.15.87:0/1420372644 >> 172.21.15.87:0/1420372644 conn(0x55f39804bba0 msgr2=0x55f398052b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- 172.21.15.87:0/1420372644 shutdown_connections 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- 172.21.15.87:0/1420372644 wait complete. 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- start start 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 Processor -- start 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f3980632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f39805aa00 0x55f39805ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ac120 con 0x55f398051e80 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ad590 con 0x55f39805aa00 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ac7f0 con 0x55f3980514d0 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b17b68740 10 client.0 _ll_register_callbacks cb 0x7f5b04012d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51588/0 (socket says 172.21.15.87:51588) 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b0e59c640 1 -- 172.21.15.87:0/2088538232 learned_addr learned my addr 172.21.15.87:0/2088538232 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f3980632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.442+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f39805aa00 0x55f39805ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 -- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 msgr2=0x55f3980632a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f3980632a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 -- 172.21.15.87:0/2088538232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f39805aa00 msgr2=0x55f39805ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f39805aa00 0x55f39805ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 -- 172.21.15.87:0/2088538232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3980cfe30 con 0x55f398051e80 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b0e59c640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5b00012010 tx=0x7f5b00004390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6459ffad1017e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b000203b0 con 0x55f398051e80 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f5b00020550 con 0x55f398051e80 2024-04-22T13:40:00.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b00024070 con 0x55f398051e80 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5b00020750 con 0x55f398051e80 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 10 client.17934 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f3980d0fd0 con 0x55f398051e80 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b00024070 con 0x55f398051e80 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5b0003b5f0 con 0x55f398051e80 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 10 client.17934 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 10 client.17934 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 20 client.17934 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f3980d1780 0x55f3980d3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 4 client.17934 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.443+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f397fc3c10 con 0x55f3980d1780 2024-04-22T13:40:00.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f3980d1780 0x55f3980d3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b0dd9b640 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f3980d1780 0x55f3980d3ba0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e447b43d2df90ced server_cookie=fddc4a0b9b956201 in_seq=0 out_seq=0 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5af5ffb640 10 client.17934 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5af5ffb640 1 -- 172.21.15.87:0/2088538232 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f397fc3c10 con 0x55f3980d1780 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5af5ffb640 10 client.17934 handle_command_reply: tid=0 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 client.17934 shutdown 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 msgr2=0x55f39805e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5b00012010 tx=0x7f5b00004390 comp rx=0 tx=0).stop 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 shutdown_connections 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55f3980d1780 0x55f3980d3ba0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f39805aa00 0x55f39805ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f398051e80 0x55f39805e610 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5af5ffb640 0 client.17934 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 --2- 172.21.15.87:0/2088538232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f3980514d0 0x55f3980632a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 >> 172.21.15.87:0/2088538232 conn(0x55f39804bba0 msgr2=0x55f39804a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 shutdown_connections 2024-04-22T13:40:00.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.444+0000 7f5b17b68740 1 -- 172.21.15.87:0/2088538232 wait complete. 2024-04-22T13:40:00.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b17b68740 20 client.17934 trim_cache size 0 max 16384 2024-04-22T13:40:00.448 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 msgr2=0x7f5af80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 0x7f5af80a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5b080be760 tx=0x7f5b080095c0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 msgr2=0x7f5b10198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005e30 tx=0x7f5b08034760 comp rx=0 tx=0).stop 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 shutdown_connections 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5af80a3d20 0x7f5af80a61e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b101985a0 0x7f5b10198a50 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b101072b0 0x7f5b1019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 --2- 172.21.15.87:0/3753024557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b101021e0 0x7f5b1019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 >> 172.21.15.87:0/3753024557 conn(0x7f5b100fda60 msgr2=0x7f5b100ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 shutdown_connections 2024-04-22T13:40:00.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.445+0000 7f5b14f7e640 1 -- 172.21.15.87:0/3753024557 wait complete. 2024-04-22T13:40:00.452 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:40:00.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 -- 172.21.15.87:0/4152855628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1014a0 msgr2=0x7f63dc106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 --2- 172.21.15.87:0/4152855628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1014a0 0x7f63dc106bd0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009900 tx=0x7f63d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:00.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 -- 172.21.15.87:0/4152855628 shutdown_connections 2024-04-22T13:40:00.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 --2- 172.21.15.87:0/4152855628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 0x7f63dc10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 --2- 172.21.15.87:0/4152855628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1014a0 0x7f63dc106bd0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 --2- 172.21.15.87:0/4152855628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 -- 172.21.15.87:0/4152855628 >> 172.21.15.87:0/4152855628 conn(0x7f63dc0760e0 msgr2=0x7f63dc0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 -- 172.21.15.87:0/4152855628 shutdown_connections 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.556+0000 7f63e27e6640 1 -- 172.21.15.87:0/4152855628 wait complete. 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 Processor -- start 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 -- start start 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 0x7f63dc19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112280 con 0x7f63dc1985a0 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112100 con 0x7f63dc100af0 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e27e6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc112400 con 0x7f63dc1072b0 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51618/0 (socket says 172.21.15.87:51618) 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:56430/0 (socket says 172.21.15.87:56430) 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 -- 172.21.15.87:0/3200431862 learned_addr learned my addr 172.21.15.87:0/3200431862 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63db7fe640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 0x7f63dc19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 -- 172.21.15.87:0/3200431862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 msgr2=0x7f63dc19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 0x7f63dc19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 -- 172.21.15.87:0/3200431862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 msgr2=0x7f63dc19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63e0d83640 1 -- 172.21.15.87:0/3200431862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc1990c0 con 0x7f63dc1985a0 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.557+0000 7f63dbfff640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:00.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f63d400eaa0 tx=0x7f63d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63d97fa640 1 -- 172.21.15.87:0/3200431862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d400cdb0 con 0x7f63dc1985a0 2024-04-22T13:40:00.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc1993b0 con 0x7f63dc1985a0 2024-04-22T13:40:00.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63d97fa640 1 -- 172.21.15.87:0/3200431862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d4005020 con 0x7f63dc1985a0 2024-04-22T13:40:00.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63d97fa640 1 -- 172.21.15.87:0/3200431862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d40116d0 con 0x7f63dc1985a0 2024-04-22T13:40:00.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.558+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1a5650 con 0x7f63dc1985a0 2024-04-22T13:40:00.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.560+0000 7f63d97fa640 1 -- 172.21.15.87:0/3200431862 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f63d40054a0 con 0x7f63dc1985a0 2024-04-22T13:40:00.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.560+0000 7f63e53d4740 1 Processor -- start 2024-04-22T13:40:00.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.560+0000 7f63e53d4740 1 -- start start 2024-04-22T13:40:00.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63d97fa640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 0x7f63c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c17e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5613a7b711f0 con 0x5613a7c0ce80 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5613a7b0e4b0 con 0x5613a7c15a00 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e53d4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5613a7b6d1b0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 0x7f63c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63d97fa640 1 -- 172.21.15.87:0/3200431862 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f63d40d0840 con 0x7f63dc1985a0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51622/0 (socket says 172.21.15.87:51622) 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 -- 172.21.15.87:0/1908629153 learned_addr learned my addr 172.21.15.87:0/1908629153 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63dbfff640 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63e0d83640 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c17e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 -- 172.21.15.87:0/1908629153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 msgr2=0x5613a7c0c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 -- 172.21.15.87:0/1908629153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 msgr2=0x5613a7c17e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c17e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 -- 172.21.15.87:0/1908629153 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a7b001b0 con 0x5613a7c0ce80 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f63d0005c40 tx=0x7f63d002ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21f0524fc9cc34e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63db7fe640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 0x7f63c40a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d0039000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63c27fc640 1 -- 172.21.15.87:0/1908629153 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0004460 con 0x5613a7c0ce80 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63c27fc640 1 -- 172.21.15.87:0/1908629153 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0005410 con 0x5613a7c0ce80 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.561+0000 7f63c27fc640 1 -- 172.21.15.87:0/1908629153 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d00035f0 con 0x5613a7c0ce80 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- 172.21.15.87:0/1908629153 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 msgr2=0x5613a7c15430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f63d0005c40 tx=0x7f63d002ca20 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- 172.21.15.87:0/1908629153 shutdown_connections 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c17e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c15430 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 --2- 172.21.15.87:0/1908629153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- 172.21.15.87:0/1908629153 >> 172.21.15.87:0/1908629153 conn(0x5613a7c06ba0 msgr2=0x5613a7c06f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- 172.21.15.87:0/1908629153 shutdown_connections 2024-04-22T13:40:00.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- 172.21.15.87:0/1908629153 wait complete. 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.562+0000 7f63e53d4740 1 -- start start 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 Processor -- start 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c0c4d0 0x5613a7c15360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c126f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5613a7c18910 con 0x5613a7c15a00 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5613a7b2a500 con 0x5613a7c0c4d0 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5613a7c18a90 con 0x5613a7c0ce80 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e53d4740 10 client.0 _ll_register_callbacks cb 0x5613a7af8b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51632/0 (socket says 172.21.15.87:51632) 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 -- 172.21.15.87:0/3274749064 learned_addr learned my addr 172.21.15.87:0/3274749064 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63dbfff640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c0c4d0 0x5613a7c15360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63db7fe640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 -- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 msgr2=0x5613a7c126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 -- 172.21.15.87:0/3274749064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c0c4d0 msgr2=0x5613a7c15360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c0c4d0 0x5613a7c15360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 -- 172.21.15.87:0/3274749064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a78833f0 con 0x5613a7c15a00 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63db7fe640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63e0d83640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f63d4098d30 tx=0x7f63d407a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adab76e0f1bff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.563+0000 7f63bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d401f070 con 0x5613a7c15a00 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f63d40ce250 con 0x5613a7c15a00 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d0005410 con 0x5613a7c15a00 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f63d409c8d0 con 0x5613a7c15a00 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 10 client.17946 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5613a794d7d0 con 0x5613a7c15a00 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0005410 con 0x5613a7c15a00 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f63d400f040 con 0x5613a7c15a00 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 10 client.17946 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 10 client.17946 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 20 client.17946 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c25ab0 0x5613a7c27ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 4 client.17946 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613a7b10930 con 0x5613a7c25ab0 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.564+0000 7f63dbfff640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c25ab0 0x5613a7c27ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.565+0000 7f63dbfff640 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c25ab0 0x5613a7c27ea0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f89cd4659e0968a3 server_cookie=163d5ee4554cb404 in_seq=0 out_seq=0 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.565+0000 7f63bb7fe640 10 client.17946 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63bb7fe640 1 -- 172.21.15.87:0/3274749064 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613a7b10930 con 0x5613a7c25ab0 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63bb7fe640 10 client.17946 handle_command_reply: tid=0 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 client.17946 shutdown 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 msgr2=0x5613a7c12d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f63d4098d30 tx=0x7f63d407a1c0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 shutdown_connections 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c25ab0 0x5613a7c27ea0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c15a00 0x5613a7c12d00 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 --2- 172.21.15.87:0/3274749064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c0c4d0 0x5613a7c15360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63bb7fe640 0 client.17946 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 >> 172.21.15.87:0/3274749064 conn(0x5613a7c06ba0 msgr2=0x5613a7c088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 shutdown_connections 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 1 -- 172.21.15.87:0/3274749064 wait complete. 2024-04-22T13:40:00.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.566+0000 7f63e53d4740 20 client.17946 trim_cache size 0 max 16384 2024-04-22T13:40:00.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 Processor -- start 2024-04-22T13:40:00.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 -- start start 2024-04-22T13:40:00.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c14e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b0330 con 0x5613a7c0ce80 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1adc00 con 0x5613a7c15a00 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e53d4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1aba70 con 0x5613a7c0c4d0 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63db7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63db7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51642/0 (socket says 172.21.15.87:51642) 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63db7fe640 1 -- 172.21.15.87:0/2263143635 learned_addr learned my addr 172.21.15.87:0/2263143635 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.579+0000 7f63e0d83640 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c14e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 -- 172.21.15.87:0/2263143635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 msgr2=0x5613a7c217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 -- 172.21.15.87:0/2263143635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 msgr2=0x5613a7c14e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c14e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 -- 172.21.15.87:0/2263143635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a7c18180 con 0x5613a7c0ce80 2024-04-22T13:40:00.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e0d83640 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c14e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63db7fe640 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f63d002c170 tx=0x7f63d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741efb2e6b7f264c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63c1ffb640 1 -- 172.21.15.87:0/2263143635 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0004460 con 0x5613a7c0ce80 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63c1ffb640 1 -- 172.21.15.87:0/2263143635 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d002c5d0 con 0x5613a7c0ce80 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63c1ffb640 1 -- 172.21.15.87:0/2263143635 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d00035f0 con 0x5613a7c0ce80 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 -- 172.21.15.87:0/2263143635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 msgr2=0x5613a7c1e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f63d002c170 tx=0x7f63d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 -- 172.21.15.87:0/2263143635 shutdown_connections 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c14e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0ce80 0x5613a7c1e0c0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 --2- 172.21.15.87:0/2263143635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0c4d0 0x5613a7c217d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 -- 172.21.15.87:0/2263143635 >> 172.21.15.87:0/2263143635 conn(0x5613a7c06ba0 msgr2=0x5613a7c0dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.580+0000 7f63e53d4740 1 -- 172.21.15.87:0/2263143635 shutdown_connections 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 -- 172.21.15.87:0/2263143635 wait complete. 2024-04-22T13:40:00.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 -- start start 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 Processor -- start 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c0eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c0f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1ac0c0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1ad530 con 0x5613a7c15a00 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1ac790 con 0x5613a7c0ce80 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e53d4740 10 client.0 _ll_register_callbacks cb 0x7f63d409c2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51654/0 (socket says 172.21.15.87:51654) 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63dbfff640 1 -- 172.21.15.87:0/4218048182 learned_addr learned my addr 172.21.15.87:0/4218048182 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63db7fe640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c0eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.581+0000 7f63e0d83640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c0f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 -- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 msgr2=0x5613a7c0eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c0eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 -- 172.21.15.87:0/4218048182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 msgr2=0x5613a7c0f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c0f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 -- 172.21.15.87:0/4218048182 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a7c0fca0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63dbfff640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f63cc012610 tx=0x7f63cc012af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303bf11c9db5fc83 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc0205a0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f63cc020740 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d002c5d0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f63cc02f050 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 10 client.17952 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5613a7c90550 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d002c5d0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f63cc01e4b0 con 0x5613a7c0c4d0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 10 client.17952 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 10 client.17952 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 20 client.17952 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c90cd0 0x5613a7c930f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 4 client.17952 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.582+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613a7b7ec10 con 0x5613a7c90cd0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63db7fe640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c90cd0 0x5613a7c930f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63db7fe640 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c90cd0 0x5613a7c930f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82beaddd5e1868f server_cookie=33ae21130b4300cd in_seq=0 out_seq=0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63bb7fe640 10 client.17952 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63bb7fe640 1 -- 172.21.15.87:0/4218048182 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613a7b7ec10 con 0x5613a7c90cd0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63bb7fe640 10 client.17952 handle_command_reply: tid=0 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63e53d4740 1 client.17952 shutdown 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 msgr2=0x5613a7c0e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.583+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f63cc012610 tx=0x7f63cc012af0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 shutdown_connections 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5613a7c90cd0 0x5613a7c930f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613a7c15a00 0x5613a7c0f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5613a7c0ce80 0x5613a7c0eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63bb7fe640 0 client.17952 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 --2- 172.21.15.87:0/4218048182 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5613a7c0c4d0 0x5613a7c0e960 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 >> 172.21.15.87:0/4218048182 conn(0x5613a7c06ba0 msgr2=0x5613a7c07e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 shutdown_connections 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 1 -- 172.21.15.87:0/4218048182 wait complete. 2024-04-22T13:40:00.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e53d4740 20 client.17952 trim_cache size 0 max 16384 2024-04-22T13:40:00.587 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:00.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 msgr2=0x7f63c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 0x7f63c40a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d0039000 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 msgr2=0x7f63dc198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f63d400eaa0 tx=0x7f63d400ef60 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 shutdown_connections 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f63c40a3d20 0x7f63c40a61e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63dc1985a0 0x7f63dc198a50 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63dc1072b0 0x7f63dc19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 --2- 172.21.15.87:0/3200431862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63dc100af0 0x7f63dc19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 >> 172.21.15.87:0/3200431862 conn(0x7f63dc0760e0 msgr2=0x7f63dc0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 shutdown_connections 2024-04-22T13:40:00.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.584+0000 7f63e27e6640 1 -- 172.21.15.87:0/3200431862 wait complete. 2024-04-22T13:40:00.590 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:00 smithi087 ceph-mon[27999]: overall HEALTH_OK 2024-04-22T13:40:00.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:40:00.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:00 smithi120 ceph-mon[31101]: overall HEALTH_OK 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 -- 172.21.15.87:0/4000527777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 msgr2=0x7fab38106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 --2- 172.21.15.87:0/4000527777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab38106200 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fab28009930 tx=0x7fab2802bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 -- 172.21.15.87:0/4000527777 shutdown_connections 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 --2- 172.21.15.87:0/4000527777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38102350 0x7fab3810dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 --2- 172.21.15.87:0/4000527777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38101990 0x7fab38101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 --2- 172.21.15.87:0/4000527777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab38106200 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 -- 172.21.15.87:0/4000527777 >> 172.21.15.87:0/4000527777 conn(0x7fab380760e0 msgr2=0x7fab380764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 -- 172.21.15.87:0/4000527777 shutdown_connections 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 -- 172.21.15.87:0/4000527777 wait complete. 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.695+0000 7fab3db2b640 1 Processor -- start 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 -- start start 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38101990 0x7fab38197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38102350 0x7fab38197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab381110a0 con 0x7fab38105e20 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab38110f20 con 0x7fab38101990 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab3db2b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab38111220 con 0x7fab38102350 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51676/0 (socket says 172.21.15.87:51676) 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 -- 172.21.15.87:0/3728413603 learned_addr learned my addr 172.21.15.87:0/3728413603 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab377fe640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38101990 0x7fab38197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab36ffd640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38102350 0x7fab38197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 -- 172.21.15.87:0/3728413603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38102350 msgr2=0x7fab38197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38102350 0x7fab38197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 -- 172.21.15.87:0/3728413603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38101990 msgr2=0x7fab38197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38101990 0x7fab38197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 -- 172.21.15.87:0/3728413603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3819c5f0 con 0x7fab38105e20 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.696+0000 7fab37fff640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fab3000eb10 tx=0x7fab3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.697+0000 7fab34ff9640 1 -- 172.21.15.87:0/3728413603 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30004b20 con 0x7fab38105e20 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.697+0000 7fab34ff9640 1 -- 172.21.15.87:0/3728413603 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab30004cc0 con 0x7fab38105e20 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.697+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3819c8e0 con 0x7fab38105e20 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.697+0000 7fab34ff9640 1 -- 172.21.15.87:0/3728413603 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30010670 con 0x7fab38105e20 2024-04-22T13:40:00.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.697+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab381a4170 con 0x7fab38105e20 2024-04-22T13:40:00.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab34ff9640 1 -- 172.21.15.87:0/3728413603 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fab30005020 con 0x7fab38105e20 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 Processor -- start 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 -- start start 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45ba8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45bab040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559f45b021f0 con 0x559f45b9d270 2024-04-22T13:40:00.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f45a9f4b0 con 0x559f45b9dc20 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab407f9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f45afe1b0 con 0x559f45ba8c10 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.699+0000 7fab36ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab34ff9640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 0x7fab200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51678/0 (socket says 172.21.15.87:51678) 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 -- 172.21.15.87:0/3066084820 learned_addr learned my addr 172.21.15.87:0/3066084820 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab377fe640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45ba8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab37fff640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 0x7fab200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45bab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 -- 172.21.15.87:0/3066084820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 msgr2=0x559f45bab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45bab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab34ff9640 1 -- 172.21.15.87:0/3728413603 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fab300d0d30 con 0x7fab38105e20 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 -- 172.21.15.87:0/3066084820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 msgr2=0x559f45ba8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45ba8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 -- 172.21.15.87:0/3066084820 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f45a911b0 con 0x559f45b9d270 2024-04-22T13:40:00.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab377fe640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45ba8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab36ffd640 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00af60 tx=0x7fab2c00a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58cfc7fbff9967b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab37fff640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 0x7fab200a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fab300ce570 tx=0x7fab3009f040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab1dffb640 1 -- 172.21.15.87:0/3066084820 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c019030 con 0x559f45b9d270 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab1dffb640 1 -- 172.21.15.87:0/3066084820 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c0110d0 con 0x559f45b9d270 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.700+0000 7fab1dffb640 1 -- 172.21.15.87:0/3066084820 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c004c60 con 0x559f45b9d270 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- 172.21.15.87:0/3066084820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 msgr2=0x559f45b9d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00af60 tx=0x7fab2c00a780 comp rx=0 tx=0).stop 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- 172.21.15.87:0/3066084820 shutdown_connections 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45bab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45ba8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 --2- 172.21.15.87:0/3066084820 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45b9d650 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- 172.21.15.87:0/3066084820 >> 172.21.15.87:0/3066084820 conn(0x559f45b97ba0 msgr2=0x559f45b97f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- 172.21.15.87:0/3066084820 shutdown_connections 2024-04-22T13:40:00.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- 172.21.15.87:0/3066084820 wait complete. 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.701+0000 7fab407f9740 1 -- start start 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 Processor -- start 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9d270 0x559f45ba8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559f45babb60 con 0x559f45b9dc20 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f45abb500 con 0x559f45b9d270 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f45babce0 con 0x559f45ba8c10 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 10 client.0 _ll_register_callbacks cb 0x559f45a89b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51682/0 (socket says 172.21.15.87:51682) 2024-04-22T13:40:00.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 -- 172.21.15.87:0/2485183303 learned_addr learned my addr 172.21.15.87:0/2485183303 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab36ffd640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 -- 172.21.15.87:0/2485183303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 msgr2=0x559f45ba5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 -- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9d270 msgr2=0x559f45ba8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9d270 0x559f45ba8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 -- 172.21.15.87:0/2485183303 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f458143f0 con 0x559f45b9dc20 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab377fe640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fab28009930 tx=0x7fab28004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4f657c45e5df68 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28005030 con 0x559f45b9dc20 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fab280051d0 con 0x559f45b9dc20 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.702+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab2c0110d0 con 0x559f45b9dc20 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fab280053d0 con 0x559f45b9dc20 2024-04-22T13:40:00.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 10 client.17964 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559f458de7d0 con 0x559f45b9dc20 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c0110d0 con 0x559f45b9dc20 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fab28003610 con 0x559f45b9dc20 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 10 client.17964 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 20 client.17964 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45bb8e20 0x559f45bbb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 4 client.17964 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f45aa1930 con 0x559f45bb8e20 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.703+0000 7fab36ffd640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45bb8e20 0x559f45bbb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab16ffd640 10 client.17964 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab36ffd640 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45bb8e20 0x559f45bbb210 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b627ca0503e3bc7b server_cookie=e2539a44b41f5f97 in_seq=0 out_seq=0 2024-04-22T13:40:00.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab16ffd640 1 -- 172.21.15.87:0/2485183303 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f45aa1930 con 0x559f45bb8e20 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab16ffd640 10 client.17964 handle_command_reply: tid=0 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 client.17964 shutdown 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 msgr2=0x559f45ba58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fab28009930 tx=0x7fab28004270 comp rx=0 tx=0).stop 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 shutdown_connections 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45bb8e20 0x559f45bbb210 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45ba58a0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab16ffd640 0 client.17964 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 --2- 172.21.15.87:0/2485183303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9d270 0x559f45ba8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.704+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 >> 172.21.15.87:0/2485183303 conn(0x559f45b97ba0 msgr2=0x559f45b998f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.705+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 shutdown_connections 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.705+0000 7fab407f9740 1 -- 172.21.15.87:0/2485183303 wait complete. 2024-04-22T13:40:00.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.705+0000 7fab407f9740 20 client.17964 trim_cache size 0 max 16384 2024-04-22T13:40:00.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:00 smithi132 ceph-mon[31531]: overall HEALTH_OK 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 Processor -- start 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 -- start start 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45bb4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab381ae090 con 0x559f45b9d270 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab381ab960 con 0x559f45b9dc20 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab407f9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab381a97d0 con 0x559f45ba8c10 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51696/0 (socket says 172.21.15.87:51696) 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 -- 172.21.15.87:0/1859336967 learned_addr learned my addr 172.21.15.87:0/1859336967 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 -- 172.21.15.87:0/1859336967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 msgr2=0x559f45ba7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 -- 172.21.15.87:0/1859336967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 msgr2=0x559f45bb4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab377fe640 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45bb4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45bb4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 -- 172.21.15.87:0/1859336967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f45bbb840 con 0x559f45b9d270 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab377fe640 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45bb4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.718+0000 7fab36ffd640 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fab2c013fd0 tx=0x7fab2c005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9139f66ba3dc9e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab1d7fa640 1 -- 172.21.15.87:0/1859336967 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c019040 con 0x559f45b9d270 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab1d7fa640 1 -- 172.21.15.87:0/1859336967 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c011730 con 0x559f45b9d270 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab1d7fa640 1 -- 172.21.15.87:0/1859336967 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c003850 con 0x559f45b9d270 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- 172.21.15.87:0/1859336967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 msgr2=0x559f45ba10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fab2c013fd0 tx=0x7fab2c005da0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- 172.21.15.87:0/1859336967 shutdown_connections 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45ba8c10 0x559f45ba7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45b9dc20 0x559f45bb4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 --2- 172.21.15.87:0/1859336967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9d270 0x559f45ba10b0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- 172.21.15.87:0/1859336967 >> 172.21.15.87:0/1859336967 conn(0x559f45b97ba0 msgr2=0x559f45b96c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- 172.21.15.87:0/1859336967 shutdown_connections 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- 172.21.15.87:0/1859336967 wait complete. 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.719+0000 7fab407f9740 1 -- start start 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 Processor -- start 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 0x559f45bb12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45ba8c10 0x559f45bacb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fab381a9e20 con 0x559f45b9dc20 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fab381ab290 con 0x559f45ba8c10 2024-04-22T13:40:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab381aa4f0 con 0x559f45b9d270 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab407f9740 10 client.0 _ll_register_callbacks cb 0x7fab2802cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51702/0 (socket says 172.21.15.87:51702) 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 -- 172.21.15.87:0/994537574 learned_addr learned my addr 172.21.15.87:0/994537574 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab36ffd640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 0x559f45bb12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab36ffd640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45ba8c10 0x559f45bacb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 -- 172.21.15.87:0/994537574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 msgr2=0x559f45bb12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 0x559f45bb12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 -- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45ba8c10 msgr2=0x559f45bacb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45ba8c10 0x559f45bacb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 -- 172.21.15.87:0/994537574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f45bad3c0 con 0x559f45b9dc20 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab36ffd640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 0x559f45bb12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:00.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab377fe640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fab28009850 tx=0x7fab2802cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828ad505472573e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.720+0000 7fab16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28059060 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7fab2803f780 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab2c011730 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fab2803f9a0 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 10 client.17970 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559f45c1f280 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c011730 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fab28054bb0 con 0x559f45b9dc20 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 10 client.17970 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 20 client.17970 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45c1fa00 0x559f45c21e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 4 client.17970 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:00.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f45b0fc10 con 0x559f45c1fa00 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.721+0000 7fab36ffd640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45c1fa00 0x559f45c21e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab16ffd640 10 client.17970 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab36ffd640 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45c1fa00 0x559f45c21e20 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c955e1552747bc server_cookie=5a7d9203ea05a493 in_seq=0 out_seq=0 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab16ffd640 1 -- 172.21.15.87:0/994537574 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f45b0fc10 con 0x559f45c1fa00 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab16ffd640 10 client.17970 handle_command_reply: tid=0 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 client.17970 shutdown 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 msgr2=0x559f45bac650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fab28009850 tx=0x7fab2802cd80 comp rx=0 tx=0).stop 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 shutdown_connections 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559f45c1fa00 0x559f45c21e20 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f45ba8c10 0x559f45bacb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559f45b9dc20 0x559f45bac650 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 --2- 172.21.15.87:0/994537574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f45b9d270 0x559f45bb12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab16ffd640 0 client.17970 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:00.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 >> 172.21.15.87:0/994537574 conn(0x559f45b97ba0 msgr2=0x559f45ba5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 shutdown_connections 2024-04-22T13:40:00.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 1 -- 172.21.15.87:0/994537574 wait complete. 2024-04-22T13:40:00.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.722+0000 7fab407f9740 20 client.17970 trim_cache size 0 max 16384 2024-04-22T13:40:00.725 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 msgr2=0x7fab200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 0x7fab200a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fab300ce570 tx=0x7fab3009f040 comp rx=0 tx=0).stop 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 msgr2=0x7fab3819bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fab3000eb10 tx=0x7fab3000c2d0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 shutdown_connections 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fab200a3d20 0x7fab200a61e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab38105e20 0x7fab3819bec0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab38102350 0x7fab38197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 --2- 172.21.15.87:0/3728413603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab38101990 0x7fab38197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:00.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 >> 172.21.15.87:0/3728413603 conn(0x7fab380760e0 msgr2=0x7fab380fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:00.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 shutdown_connections 2024-04-22T13:40:00.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:00.723+0000 7fab3db2b640 1 -- 172.21.15.87:0/3728413603 wait complete. 2024-04-22T13:40:01.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:01 smithi120 ceph-mon[31101]: pgmap v252: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:01.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3071685779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:01 smithi132 ceph-mon[31531]: pgmap v252: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3071685779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:01.730 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:40:01.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:40:01.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:40:01.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:01 smithi087 ceph-mon[27999]: pgmap v252: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:01.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3071685779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:01.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 -- 172.21.15.87:0/2708532787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d80101a90 msgr2=0x7f9d80105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 --2- 172.21.15.87:0/2708532787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d80101a90 0x7f9d80105a80 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009900 tx=0x7f9d7002bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:01.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 -- 172.21.15.87:0/2708532787 shutdown_connections 2024-04-22T13:40:01.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 --2- 172.21.15.87:0/2708532787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d80106160 0x7f9d8010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 --2- 172.21.15.87:0/2708532787 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d80101a90 0x7f9d80105a80 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 --2- 172.21.15.87:0/2708532787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d801010e0 0x7f9d801014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 -- 172.21.15.87:0/2708532787 >> 172.21.15.87:0/2708532787 conn(0x7f9d80077fb0 msgr2=0x7f9d800783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 -- 172.21.15.87:0/2708532787 shutdown_connections 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.849+0000 7f9d874a0640 1 -- 172.21.15.87:0/2708532787 wait complete. 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 Processor -- start 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 -- start start 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d801010e0 0x7f9d80198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d80106160 0x7f9d80198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801110a0 con 0x7f9d8019d010 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80110f20 con 0x7f9d80106160 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d874a0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80111220 con 0x7f9d801010e0 2024-04-22T13:40:01.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51718/0 (socket says 172.21.15.87:51718) 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 -- 172.21.15.87:0/1997439693 learned_addr learned my addr 172.21.15.87:0/1997439693 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d80106160 0x7f9d80198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d8523c640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d801010e0 0x7f9d80198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 -- 172.21.15.87:0/1997439693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d801010e0 msgr2=0x7f9d80198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d801010e0 0x7f9d80198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 -- 172.21.15.87:0/1997439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d80106160 msgr2=0x7f9d80198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d80106160 0x7f9d80198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.850+0000 7f9d85a3d640 1 -- 172.21.15.87:0/1997439693 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8019db00 con 0x7f9d8019d010 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d85a3d640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800cad0 tx=0x7f9d7800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d767fc640 1 -- 172.21.15.87:0/1997439693 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800ae20 con 0x7f9d8019d010 2024-04-22T13:40:01.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d767fc640 1 -- 172.21.15.87:0/1997439693 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d78005020 con 0x7f9d8019d010 2024-04-22T13:40:01.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8019ddf0 con 0x7f9d8019d010 2024-04-22T13:40:01.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d767fc640 1 -- 172.21.15.87:0/1997439693 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800f6b0 con 0x7f9d8019d010 2024-04-22T13:40:01.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.851+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d801a55b0 con 0x7f9d8019d010 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 Processor -- start 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d767fc640 1 -- 172.21.15.87:0/1997439693 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d780051d0 con 0x7f9d8019d010 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 -- start start 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b87fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b8806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56262b7daf00 con 0x56262b876740 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56262b778590 con 0x56262b880250 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d8a168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56262b7d6ec0 con 0x56262b8770f0 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51726/0 (socket says 172.21.15.87:51726) 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.854+0000 7f9d84a3b640 1 -- 172.21.15.87:0/3746741684 learned_addr learned my addr 172.21.15.87:0/3746741684 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d8523c640 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b87fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 -- 172.21.15.87:0/3746741684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 msgr2=0x56262b87fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d767fc640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 0x7f9d6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b87fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 -- 172.21.15.87:0/3746741684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 msgr2=0x56262b8806d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b8806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 -- 172.21.15.87:0/3746741684 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56262b5b57e0 con 0x56262b876740 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d85a3d640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 0x7f9d6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9d8006a410 tx=0x7f9d70009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbeac52814dbf23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d767fc640 1 -- 172.21.15.87:0/1997439693 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f9d780d07a0 con 0x7f9d8019d010 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d6b7fe640 1 -- 172.21.15.87:0/3746741684 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70044070 con 0x56262b876740 2024-04-22T13:40:01.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d6b7fe640 1 -- 172.21.15.87:0/3746741684 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d7002c7e0 con 0x56262b876740 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.855+0000 7f9d6b7fe640 1 -- 172.21.15.87:0/3746741684 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d700046c0 con 0x56262b876740 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d85a3d640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 0x7f9d6c0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9d7809c710 tx=0x7f9d7806f270 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- 172.21.15.87:0/3746741684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 msgr2=0x56262b876b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9d8006a410 tx=0x7f9d70009680 comp rx=0 tx=0).stop 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- 172.21.15.87:0/3746741684 shutdown_connections 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b8806d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b87fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 --2- 172.21.15.87:0/3746741684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b876b20 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- 172.21.15.87:0/3746741684 >> 172.21.15.87:0/3746741684 conn(0x56262b870fb0 msgr2=0x56262b871390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- 172.21.15.87:0/3746741684 shutdown_connections 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- 172.21.15.87:0/3746741684 wait complete. 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 -- start start 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.856+0000 7f9d8a168740 1 Processor -- start 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 0x56262b884280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b880250 0x56262b8847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56262b887170 con 0x56262b876740 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56262b794580 con 0x56262b8770f0 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56262b8872f0 con 0x56262b880250 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 10 client.0 _ll_register_callbacks cb 0x56262b79a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51736/0 (socket says 172.21.15.87:51736) 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 -- 172.21.15.87:0/2166215783 learned_addr learned my addr 172.21.15.87:0/2166215783 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b880250 0x56262b8847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8523c640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 0x56262b884280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 -- 172.21.15.87:0/2166215783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b880250 msgr2=0x56262b8847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b880250 0x56262b8847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 -- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 msgr2=0x56262b884280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 0x56262b884280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 -- 172.21.15.87:0/2166215783 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56262b615f70 con 0x56262b876740 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8523c640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 0x56262b884280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:01.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f9d70005770 tx=0x7f9d700052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9919aab863ca2b6b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70056060 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f9d700035f0 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.857+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d7002c7e0 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9d70044040 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 10 client.17982 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56262b4ad170 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d7002c7e0 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9d70004be0 con 0x56262b876740 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 10 client.17982 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 10 client.17982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 20 client.17982 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:01.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b894600 0x56262b8969f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 4 client.17982 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56262b7e8920 con 0x56262b894600 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.858+0000 7f9d8523c640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b894600 0x56262b8969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.859+0000 7f9d8523c640 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b894600 0x56262b8969f0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47887ef1866b9648 server_cookie=b936d64e9986e95d in_seq=0 out_seq=0 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.859+0000 7f9d68ff9640 10 client.17982 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.859+0000 7f9d68ff9640 1 -- 172.21.15.87:0/2166215783 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56262b7e8920 con 0x56262b894600 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.859+0000 7f9d68ff9640 10 client.17982 handle_command_reply: tid=0 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.859+0000 7f9d8a168740 1 client.17982 shutdown 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 msgr2=0x56262b883d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f9d70005770 tx=0x7f9d700052d0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 shutdown_connections 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b894600 0x56262b8969f0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b880250 0x56262b8847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d68ff9640 0 client.17982 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b8770f0 0x56262b884280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 --2- 172.21.15.87:0/2166215783 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b883d40 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 >> 172.21.15.87:0/2166215783 conn(0x56262b870fb0 msgr2=0x56262b872d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 shutdown_connections 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 1 -- 172.21.15.87:0/2166215783 wait complete. 2024-04-22T13:40:01.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.860+0000 7f9d8a168740 20 client.17982 trim_cache size 0 max 16384 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.877+0000 7f9d8a168740 1 Processor -- start 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.877+0000 7f9d8a168740 1 -- start start 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b881640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b86fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ae0f0 con 0x56262b876740 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ab9c0 con 0x56262b880250 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8a168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801a9830 con 0x56262b8770f0 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51752/0 (socket says 172.21.15.87:51752) 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 -- 172.21.15.87:0/42684825 learned_addr learned my addr 172.21.15.87:0/42684825 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d8523c640 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b881640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 -- 172.21.15.87:0/42684825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 msgr2=0x56262b881640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b881640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 -- 172.21.15.87:0/42684825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 msgr2=0x56262b86fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b86fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 -- 172.21.15.87:0/42684825 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56262b870ae0 con 0x56262b876740 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f9d700052a0 tx=0x7f9d7002ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89335176ebbcdef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d6affd640 1 -- 172.21.15.87:0/42684825 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70056070 con 0x56262b876740 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d6affd640 1 -- 172.21.15.87:0/42684825 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d70009520 con 0x56262b876740 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.878+0000 7f9d6affd640 1 -- 172.21.15.87:0/42684825 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d700046c0 con 0x56262b876740 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- 172.21.15.87:0/42684825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 msgr2=0x56262b886470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f9d700052a0 tx=0x7f9d7002ca60 comp rx=0 tx=0).stop 2024-04-22T13:40:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- 172.21.15.87:0/42684825 shutdown_connections 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b86fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b8770f0 0x56262b881640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 --2- 172.21.15.87:0/42684825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b876740 0x56262b886470 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- 172.21.15.87:0/42684825 >> 172.21.15.87:0/42684825 conn(0x56262b870fb0 msgr2=0x56262b86f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- 172.21.15.87:0/42684825 shutdown_connections 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- 172.21.15.87:0/42684825 wait complete. 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 -- start start 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.879+0000 7f9d8a168740 1 Processor -- start 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 0x56262b899b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b898400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801a9e80 con 0x56262b8770f0 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ab2f0 con 0x56262b880250 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801aa550 con 0x56262b876740 2024-04-22T13:40:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 10 client.0 _ll_register_callbacks cb 0x7f9d7002c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51760/0 (socket says 172.21.15.87:51760) 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 -- 172.21.15.87:0/887623018 learned_addr learned my addr 172.21.15.87:0/887623018 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 0x56262b899b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b898400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 -- 172.21.15.87:0/887623018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 msgr2=0x56262b899b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 0x56262b899b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 -- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 msgr2=0x56262b898400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b898400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 -- 172.21.15.87:0/887623018 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56262b898940 con 0x56262b8770f0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 0x56262b899b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8523c640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x56262b8835a0 tx=0x7f9d7c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d39aae0f57875 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c01d580 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f9d7c01d720 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d70009520 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.880+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9d7c01d920 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 10 client.17988 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56262b8fb5e0 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d70009520 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9d7c0255c0 con 0x56262b8770f0 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 10 client.17988 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 10 client.17988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 20 client.17988 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:01.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b8fbd40 0x56262b8fe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:01.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 4 client.17988 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:40:01.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56262b7e2790 con 0x56262b8fbd40 2024-04-22T13:40:01.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.881+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b8fbd40 0x56262b8fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:01.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.882+0000 7f9d68ff9640 10 client.17988 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:01.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.882+0000 7f9d84a3b640 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b8fbd40 0x56262b8fe160 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77bd9e98a4fc353 server_cookie=c8678128e506aee8 in_seq=0 out_seq=0 2024-04-22T13:40:01.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d68ff9640 1 -- 172.21.15.87:0/887623018 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56262b7e2790 con 0x56262b8fbd40 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d68ff9640 10 client.17988 handle_command_reply: tid=0 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 client.17988 shutdown 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 msgr2=0x56262b897ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x56262b8835a0 tx=0x7f9d7c011810 comp rx=0 tx=0).stop 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 shutdown_connections 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56262b8fbd40 0x56262b8fe160 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56262b880250 0x56262b898400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56262b8770f0 0x56262b897ec0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 --2- 172.21.15.87:0/887623018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262b876740 0x56262b899b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 >> 172.21.15.87:0/887623018 conn(0x56262b870fb0 msgr2=0x56262b882d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d68ff9640 0 client.17988 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 shutdown_connections 2024-04-22T13:40:01.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 1 -- 172.21.15.87:0/887623018 wait complete. 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d8a168740 20 client.17988 trim_cache size 0 max 16384 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "6fd4b6f1-013e-4c19-9e32-6d4b16555ba2", 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:40:01.886 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 msgr2=0x7f9d6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 0x7f9d6c0a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9d7809c710 tx=0x7f9d7806f270 comp rx=0 tx=0).stop 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 msgr2=0x7f9d8019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.883+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800cad0 tx=0x7f9d7800cf90 comp rx=0 tx=0).stop 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 shutdown_connections 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9d6c0a3d20 0x7f9d6c0a61e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d8019d010 0x7f9d8019d4c0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d80106160 0x7f9d80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 --2- 172.21.15.87:0/1997439693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d801010e0 0x7f9d80198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:01.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 >> 172.21.15.87:0/1997439693 conn(0x7f9d80077fb0 msgr2=0x7f9d80100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:01.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 shutdown_connections 2024-04-22T13:40:01.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:01.884+0000 7f9d874a0640 1 -- 172.21.15.87:0/1997439693 wait complete. 2024-04-22T13:40:01.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:40:01.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:40:02.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 -- 172.21.15.87:0/3897055227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c06a920 msgr2=0x7f175c06ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 --2- 172.21.15.87:0/3897055227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c06a920 0x7f175c06ad00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1758009900 tx=0x7f175802bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 -- 172.21.15.87:0/3897055227 shutdown_connections 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 --2- 172.21.15.87:0/3897055227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c1072c0 0x7f175c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 --2- 172.21.15.87:0/3897055227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c06b2d0 0x7f175c106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 --2- 172.21.15.87:0/3897055227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c06a920 0x7f175c06ad00 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 -- 172.21.15.87:0/3897055227 >> 172.21.15.87:0/3897055227 conn(0x7f175c0771d0 msgr2=0x7f175c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 -- 172.21.15.87:0/3897055227 shutdown_connections 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.035+0000 7f1763c26640 1 -- 172.21.15.87:0/3897055227 wait complete. 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 Processor -- start 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 -- start start 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c06a920 0x7f175c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c19cf90 0x7f175c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1121d0 con 0x7f175c1072c0 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f175c112050 con 0x7f175c19cf90 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f1763c26640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f175c112350 con 0x7f175c06a920 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51772/0 (socket says 172.21.15.87:51772) 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 -- 172.21.15.87:0/686095687 learned_addr learned my addr 172.21.15.87:0/686095687 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17619c2640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c06a920 0x7f175c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17621c3640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c19cf90 0x7f175c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 -- 172.21.15.87:0/686095687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c06a920 msgr2=0x7f175c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c06a920 0x7f175c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 -- 172.21.15.87:0/686095687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c19cf90 msgr2=0x7f175c19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c19cf90 0x7f175c19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.036+0000 7f17611c1640 1 -- 172.21.15.87:0/686095687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175c19da80 con 0x7f175c1072c0 2024-04-22T13:40:02.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f17611c1640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f174c00e9f0 tx=0x7f174c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f1752ffd640 1 -- 172.21.15.87:0/686095687 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c00ccc0 con 0x7f175c1072c0 2024-04-22T13:40:02.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f1752ffd640 1 -- 172.21.15.87:0/686095687 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174c0108e0 con 0x7f175c1072c0 2024-04-22T13:40:02.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f1752ffd640 1 -- 172.21.15.87:0/686095687 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c018860 con 0x7f175c1072c0 2024-04-22T13:40:02.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f175c19dd70 con 0x7f175c1072c0 2024-04-22T13:40:02.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.037+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f175c1a5560 con 0x7f175c1072c0 2024-04-22T13:40:02.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.039+0000 7f1752ffd640 1 -- 172.21.15.87:0/686095687 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f174c010430 con 0x7f175c1072c0 2024-04-22T13:40:02.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.039+0000 7f176680e740 1 Processor -- start 2024-04-22T13:40:02.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 -- start start 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3ea3e90 0x5624b3ea6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5624b3dfb1e0 con 0x5624b3e96230 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624b3d98390 con 0x5624b3ea3e90 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f176680e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5624b3df71a0 con 0x5624b3e96b50 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f1752ffd640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 0x7f17480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51782/0 (socket says 172.21.15.87:51782) 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 -- 172.21.15.87:0/4222226222 learned_addr learned my addr 172.21.15.87:0/4222226222 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17611c1640 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3ea3e90 0x5624b3ea6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 0x7f17480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17619c2640 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f1752ffd640 1 -- 172.21.15.87:0/686095687 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f174c0d0a70 con 0x7f175c1072c0 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 -- 172.21.15.87:0/4222226222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 msgr2=0x5624b3ea3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 -- 172.21.15.87:0/4222226222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3ea3e90 msgr2=0x5624b3ea6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3ea3e90 0x5624b3ea6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.040+0000 7f17621c3640 1 -- 172.21.15.87:0/4222226222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624b3bd77d0 con 0x5624b3e96230 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f17621c3640 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f1754012d80 tx=0x7f1754010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58858d00edc3a6b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f17621c3640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 0x7f17480a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1754010c40 tx=0x7f175400c000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f1747fff640 1 -- 172.21.15.87:0/4222226222 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754024070 con 0x5624b3e96230 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f1747fff640 1 -- 172.21.15.87:0/4222226222 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754009d80 con 0x5624b3e96230 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f1747fff640 1 -- 172.21.15.87:0/4222226222 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754005240 con 0x5624b3e96230 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 -- 172.21.15.87:0/4222226222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 msgr2=0x5624b3e96610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f1754012d80 tx=0x7f1754010740 comp rx=0 tx=0).stop 2024-04-22T13:40:02.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 -- 172.21.15.87:0/4222226222 shutdown_connections 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3ea3e90 0x5624b3ea6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 --2- 172.21.15.87:0/4222226222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3e96610 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.041+0000 7f176680e740 1 -- 172.21.15.87:0/4222226222 >> 172.21.15.87:0/4222226222 conn(0x5624b3e90cb0 msgr2=0x5624b3e91090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- 172.21.15.87:0/4222226222 shutdown_connections 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- 172.21.15.87:0/4222226222 wait complete. 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- start start 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 Processor -- start 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5624b3ea6c70 con 0x5624b3ea3e90 2024-04-22T13:40:02.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624b3db42f0 con 0x5624b3e96230 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5624b3ea6df0 con 0x5624b3e96b50 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f176680e740 10 client.0 _ll_register_callbacks cb 0x5624b3dc1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51796/0 (socket says 172.21.15.87:51796) 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f17621c3640 1 -- 172.21.15.87:0/1690851370 learned_addr learned my addr 172.21.15.87:0/1690851370 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.042+0000 7f17611c1640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17619c2640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 -- 172.21.15.87:0/1690851370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 msgr2=0x5624b3ea1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 -- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 msgr2=0x5624b3ea0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 -- 172.21.15.87:0/1690851370 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624b3b0d3f0 con 0x5624b3ea3e90 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17619c2640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:02.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17621c3640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f1754012fd0 tx=0x7f175400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83e248a6eac294d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754024050 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1754009d80 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f1754004cd0 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1754005000 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f176680e740 10 client.18000 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5624b3acd0f0 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754009d80 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.043+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f175400f040 con 0x5624b3ea3e90 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 10 client.18000 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 10 client.18000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 20 client.18000 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3eb3fc0 0x5624b3eb63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 4 client.18000 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624b3e08c00 con 0x5624b3eb3fc0 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f17619c2640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3eb3fc0 0x5624b3eb63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f17457fa640 10 client.18000 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.044+0000 7f17619c2640 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3eb3fc0 0x5624b3eb63b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3f7de5f5c73ebe6 server_cookie=59d69b784998fd15 in_seq=0 out_seq=0 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f17457fa640 1 -- 172.21.15.87:0/1690851370 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624b3e08c00 con 0x5624b3eb3fc0 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f17457fa640 10 client.18000 handle_command_reply: tid=0 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 client.18000 shutdown 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 msgr2=0x5624b3ea19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f1754012fd0 tx=0x7f175400cc70 comp rx=0 tx=0).stop 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 shutdown_connections 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3eb3fc0 0x5624b3eb63b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ea19d0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f17457fa640 0 client.18000 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3ea1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 --2- 172.21.15.87:0/1690851370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 >> 172.21.15.87:0/1690851370 conn(0x5624b3e90cb0 msgr2=0x5624b3e92a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 shutdown_connections 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 1 -- 172.21.15.87:0/1690851370 wait complete. 2024-04-22T13:40:02.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.045+0000 7f176680e740 20 client.18000 trim_cache size 0 max 16384 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 Processor -- start 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 -- start start 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3eafba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1ae190 con 0x5624b3ea3e90 2024-04-22T13:40:02.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1aba60 con 0x5624b3e96230 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f176680e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1a98d0 con 0x5624b3e96b50 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.062+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51800/0 (socket says 172.21.15.87:51800) 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 -- 172.21.15.87:0/3081073668 learned_addr learned my addr 172.21.15.87:0/3081073668 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17619c2640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17611c1640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3eafba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 -- 172.21.15.87:0/3081073668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 msgr2=0x5624b3eafba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3eafba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 -- 172.21.15.87:0/3081073668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 msgr2=0x5624b3ea3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 -- 172.21.15.87:0/3081073668 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624b3eb6970 con 0x5624b3ea3e90 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17611c1640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3eafba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17621c3640 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1754002230 tx=0x7f1754010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71db82ca3d632b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17477fe640 1 -- 172.21.15.87:0/3081073668 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175403d070 con 0x5624b3ea3e90 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17477fe640 1 -- 172.21.15.87:0/3081073668 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754020430 con 0x5624b3ea3e90 2024-04-22T13:40:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.063+0000 7f17477fe640 1 -- 172.21.15.87:0/3081073668 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754024040 con 0x5624b3ea3e90 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- 172.21.15.87:0/3081073668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 msgr2=0x5624b3ebbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1754002230 tx=0x7f1754010cf0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- 172.21.15.87:0/3081073668 shutdown_connections 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3ea3e90 0x5624b3ebbba0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3e96b50 0x5624b3eafba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- 172.21.15.87:0/3081073668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96230 0x5624b3ea3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- 172.21.15.87:0/3081073668 >> 172.21.15.87:0/3081073668 conn(0x5624b3e90cb0 msgr2=0x5624b3e8fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- 172.21.15.87:0/3081073668 shutdown_connections 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- 172.21.15.87:0/3081073668 wait complete. 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- start start 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 Processor -- start 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96b50 0x5624b3eac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 0x5624b3e9d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1a9f20 con 0x5624b3e96230 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1ab390 con 0x5624b3e96b50 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1aa5f0 con 0x5624b3ea3e90 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f176680e740 10 client.0 _ll_register_callbacks cb 0x7f174c098da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f17619c2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f17619c2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51814/0 (socket says 172.21.15.87:51814) 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.064+0000 7f17619c2640 1 -- 172.21.15.87:0/2323879908 learned_addr learned my addr 172.21.15.87:0/2323879908 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17621c3640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 0x5624b3e9d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17611c1640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96b50 0x5624b3eac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 -- 172.21.15.87:0/2323879908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 msgr2=0x5624b3e9d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 0x5624b3e9d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 -- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96b50 msgr2=0x5624b3eac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96b50 0x5624b3eac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 -- 172.21.15.87:0/2323879908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624b3e9da50 con 0x5624b3e96230 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17621c3640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 0x5624b3e9d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17619c2640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1758009340 tx=0x7f1758004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc3549464c2356c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175804e020 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f175802c9f0 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1754020430 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f17580044b0 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f176680e740 10 client.18006 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5624b3f19dd0 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754020430 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.065+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1758004930 con 0x5624b3e96230 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 10 client.18006 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 10 client.18006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 20 client.18006 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3f1a520 0x5624b3f1c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 4 client.18006 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624b3e02a70 con 0x5624b3f1a520 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f17611c1640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3f1a520 0x5624b3f1c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f17457fa640 10 client.18006 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f17611c1640 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3f1a520 0x5624b3f1c940 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a863d1f4414d4672 server_cookie=afd37790895242d0 in_seq=0 out_seq=0 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f17457fa640 1 -- 172.21.15.87:0/2323879908 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5624b3e02a70 con 0x5624b3f1a520 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f17457fa640 10 client.18006 handle_command_reply: tid=0 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 1 client.18006 shutdown 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 msgr2=0x5624b3eac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.066+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1758009340 tx=0x7f1758004280 comp rx=0 tx=0).stop 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 shutdown_connections 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5624b3f1a520 0x5624b3f1c940 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5624b3ea3e90 0x5624b3e9d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624b3e96b50 0x5624b3eac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f17457fa640 0 client.18006 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 --2- 172.21.15.87:0/2323879908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5624b3e96230 0x5624b3eac0b0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 >> 172.21.15.87:0/2323879908 conn(0x5624b3e90cb0 msgr2=0x5624b3e9f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 shutdown_connections 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 1 -- 172.21.15.87:0/2323879908 wait complete. 2024-04-22T13:40:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f176680e740 20 client.18006 trim_cache size 0 max 16384 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "6fd4b6f1-013e-4c19-9e32-6d4b16555ba2": { 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "6fd4b6f1-013e-4c19-9e32-6d4b16555ba2", 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:02.070 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:02.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 msgr2=0x7f17480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 0x7f17480a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1754010c40 tx=0x7f175400c000 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 msgr2=0x7f175c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f174c00e9f0 tx=0x7f174c00eeb0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 shutdown_connections 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f17480a3d20 0x7f17480a61e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c19cf90 0x7f175c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f175c1072c0 0x7f175c198c10 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 --2- 172.21.15.87:0/686095687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f175c06a920 0x7f175c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 >> 172.21.15.87:0/686095687 conn(0x7f175c0771d0 msgr2=0x7f175c0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 shutdown_connections 2024-04-22T13:40:02.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:02.067+0000 7f1763c26640 1 -- 172.21.15.87:0/686095687 wait complete. 2024-04-22T13:40:02.074 INFO:tasks.cephfs.filesystem:scrub status for tag:6fd4b6f1-013e-4c19-9e32-6d4b16555ba2 - {'path': '/', 'tag': '6fd4b6f1-013e-4c19-9e32-6d4b16555ba2', 'options': 'recursive,force'} 2024-04-22T13:40:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:02 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:40:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:02 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:02 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:40:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:02 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:02 smithi087 ceph-mon[27999]: pgmap v253: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:02 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:40:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:02 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:02 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:40:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:02 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:02 smithi120 ceph-mon[31101]: pgmap v253: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:02 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:40:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:02 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:02 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:40:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:02 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:02 smithi132 ceph-mon[31531]: pgmap v253: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:04 smithi120 ceph-mon[31101]: pgmap v254: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-22T13:40:04.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:04 smithi132 ceph-mon[31531]: pgmap v254: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-22T13:40:04.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:04 smithi087 ceph-mon[27999]: pgmap v254: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 -- 172.21.15.87:0/561245843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c04102b80 msgr2=0x7f4c04106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/561245843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c04102b80 0x7f4c04106bc0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009900 tx=0x7f4bf402bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 -- 172.21.15.87:0/561245843 shutdown_connections 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/561245843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041072a0 0x7f4c0410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/561245843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c04102b80 0x7f4c04106bc0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/561245843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041021d0 0x7f4c041025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 -- 172.21.15.87:0/561245843 >> 172.21.15.87:0/561245843 conn(0x7f4c040793e0 msgr2=0x7f4c040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 -- 172.21.15.87:0/561245843 shutdown_connections 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.833+0000 7f4c0be5b640 1 -- 172.21.15.87:0/561245843 wait complete. 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 Processor -- start 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 -- start start 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 0x7f4c041986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041072a0 0x7f4c04198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041121e0 con 0x7f4c0419cfa0 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04112060 con 0x7f4c041021d0 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0be5b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04112360 con 0x7f4c041072a0 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0a3f8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0a3f8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51844/0 (socket says 172.21.15.87:51844) 2024-04-22T13:40:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c0a3f8640 1 -- 172.21.15.87:0/494329559 learned_addr learned my addr 172.21.15.87:0/494329559 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.834+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 0x7f4c041986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c093f6640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041072a0 0x7f4c04198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 -- 172.21.15.87:0/494329559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041072a0 msgr2=0x7f4c04198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041072a0 0x7f4c04198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 -- 172.21.15.87:0/494329559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 msgr2=0x7f4c041986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 0x7f4c041986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 -- 172.21.15.87:0/494329559 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0419da90 con 0x7f4c0419cfa0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 0x7f4c041986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e960 tx=0x7f4bfc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4bfaffd640 1 -- 172.21.15.87:0/494329559 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc004d20 con 0x7f4c0419cfa0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0419dd80 con 0x7f4c0419cfa0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4bfaffd640 1 -- 172.21.15.87:0/494329559 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bfc005020 con 0x7f4c0419cfa0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4bfaffd640 1 -- 172.21.15.87:0/494329559 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0116d0 con 0x7f4c0419cfa0 2024-04-22T13:40:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.835+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c041a5520 con 0x7f4c0419cfa0 2024-04-22T13:40:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.837+0000 7f4bfaffd640 1 -- 172.21.15.87:0/494329559 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4bfc0051d0 con 0x7f4c0419cfa0 2024-04-22T13:40:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 Processor -- start 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 -- start start 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4bfaffd640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 0x7f4bf00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 0x55d7aef95620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d7aeefa8d0 con 0x55d7aef95b60 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7aee97e80 con 0x55d7aef968d0 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.838+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7aeef6890 con 0x55d7aef9f0e0 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4bfaffd640 1 -- 172.21.15.87:0/494329559 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4bfc0d0980 con 0x7f4c0419cfa0 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 0x7f4bf00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51848/0 (socket says 172.21.15.87:51848) 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 -- 172.21.15.87:0/3473712331 learned_addr learned my addr 172.21.15.87:0/3473712331 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 0x55d7aef95620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 -- 172.21.15.87:0/3473712331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 msgr2=0x55d7aefa14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 -- 172.21.15.87:0/3473712331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 msgr2=0x55d7aef95620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 0x55d7aef95620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 -- 172.21.15.87:0/3473712331 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7aeeb9320 con 0x55d7aef95b60 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 0x55d7aef95620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 0x7f4bf00a6280 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4bf402c780 tx=0x7f4bf402c650 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4c093f6640 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406b630 tx=0x7f4bf4045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3193d1acf906d929 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.839+0000 7f4beffff640 1 -- 172.21.15.87:0/3473712331 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf402cbb0 con 0x55d7aef95b60 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4beffff640 1 -- 172.21.15.87:0/3473712331 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf40054d0 con 0x55d7aef95b60 2024-04-22T13:40:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4beffff640 1 -- 172.21.15.87:0/3473712331 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40035f0 con 0x55d7aef95b60 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 -- 172.21.15.87:0/3473712331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 msgr2=0x55d7aef95fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406b630 tx=0x7f4bf4045000 comp rx=0 tx=0).stop 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 -- 172.21.15.87:0/3473712331 shutdown_connections 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef95b60 0x55d7aef95fe0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/3473712331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef968d0 0x55d7aef95620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 -- 172.21.15.87:0/3473712331 >> 172.21.15.87:0/3473712331 conn(0x55d7aef5cc90 msgr2=0x55d7aef90850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 -- 172.21.15.87:0/3473712331 shutdown_connections 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.840+0000 7f4c0ea45740 1 -- 172.21.15.87:0/3473712331 wait complete. 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 -- start start 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 Processor -- start 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aef9ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d7aefa1d50 con 0x55d7aef968d0 2024-04-22T13:40:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7aee82eb0 con 0x55d7aef95b60 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d7aefa1ed0 con 0x55d7aef9f0e0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c0ea45740 10 client.0 _ll_register_callbacks cb 0x55d7aeeafb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51850/0 (socket says 172.21.15.87:51850) 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.841+0000 7f4c093f6640 1 -- 172.21.15.87:0/1072969586 learned_addr learned my addr 172.21.15.87:0/1072969586 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aef9ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 -- 172.21.15.87:0/1072969586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 msgr2=0x55d7aef9c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 -- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 msgr2=0x55d7aef9ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aef9ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 -- 172.21.15.87:0/1072969586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7aec0d210 con 0x55d7aef968d0 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aef9ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009900 tx=0x7f4bf4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f8f9781fa1219a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf405a020 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4bf4005170 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf40054d0 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4bf4065050 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c0ea45740 10 client.18018 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d7aecd9620 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.842+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf40054d0 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4bf4043450 con 0x55d7aef968d0 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 10 client.18018 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 10 client.18018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 20 client.18018 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7aefaef50 0x55d7aefb1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 4 client.18018 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7aef0e880 con 0x55d7aefaef50 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7aefaef50 0x55d7aefb1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4bed7fa640 10 client.18018 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.843+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7aefaef50 0x55d7aefb1340 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df3e226c5396ee58 server_cookie=e9a8a534d750745 in_seq=0 out_seq=0 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4bed7fa640 1 -- 172.21.15.87:0/1072969586 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7aef0e880 con 0x55d7aefaef50 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4bed7fa640 10 client.18018 handle_command_reply: tid=0 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 client.18018 shutdown 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 msgr2=0x55d7aef9be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009900 tx=0x7f4bf4005fd0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 shutdown_connections 2024-04-22T13:40:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7aefaef50 0x55d7aefb1340 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be60 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1072969586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aef9ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4bed7fa640 0 client.18018 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.844+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 >> 172.21.15.87:0/1072969586 conn(0x55d7aef5cc90 msgr2=0x55d7aef925a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.845+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 shutdown_connections 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.845+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1072969586 wait complete. 2024-04-22T13:40:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.845+0000 7f4c0ea45740 20 client.18018 trim_cache size 0 max 16384 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.859+0000 7f4c0ea45740 1 Processor -- start 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.859+0000 7f4c0ea45740 1 -- start start 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aefaaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b0340 con 0x55d7aef968d0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041adc10 con 0x55d7aef95b60 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041aba80 con 0x55d7aef9f0e0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51860/0 (socket says 172.21.15.87:51860) 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 -- 172.21.15.87:0/1207309324 learned_addr learned my addr 172.21.15.87:0/1207309324 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aefaaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 -- 172.21.15.87:0/1207309324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 msgr2=0x55d7aef9e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 -- 172.21.15.87:0/1207309324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 msgr2=0x55d7aefaaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aefaaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 -- 172.21.15.87:0/1207309324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7aef8bf30 con 0x55d7aef968d0 2024-04-22T13:40:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4c093f6640 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4005860 tx=0x7f4bf4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a141bd7040f3c4f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4bef7fe640 1 -- 172.21.15.87:0/1207309324 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf405a070 con 0x55d7aef968d0 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4bef7fe640 1 -- 172.21.15.87:0/1207309324 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf4043450 con 0x55d7aef968d0 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.860+0000 7f4bef7fe640 1 -- 172.21.15.87:0/1207309324 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4043e60 con 0x55d7aef968d0 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1207309324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 msgr2=0x55d7aef9be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4005860 tx=0x7f4bf4009780 comp rx=0 tx=0).stop 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1207309324 shutdown_connections 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef9f0e0 0x55d7aef9e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aef9be00 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/1207309324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef95b60 0x55d7aefaaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1207309324 >> 172.21.15.87:0/1207309324 conn(0x55d7aef5cc90 msgr2=0x55d7aef9b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1207309324 shutdown_connections 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- 172.21.15.87:0/1207309324 wait complete. 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.861+0000 7f4c0ea45740 1 -- start start 2024-04-22T13:40:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 Processor -- start 2024-04-22T13:40:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 0x55d7aefa2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041ac0d0 con 0x55d7aef968d0 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041ad540 con 0x55d7aef9f0e0 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041ac7a0 con 0x55d7aef95b60 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0ea45740 10 client.0 _ll_register_callbacks cb 0x7f4c0000b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51862/0 (socket says 172.21.15.87:51862) 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 -- 172.21.15.87:0/368536656 learned_addr learned my addr 172.21.15.87:0/368536656 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c0a3f8640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 0x55d7aefa2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 -- 172.21.15.87:0/368536656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 msgr2=0x55d7aefa2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 0x55d7aefa2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 -- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef9f0e0 msgr2=0x55d7aefa3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c093f6640 1 -- 172.21.15.87:0/368536656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7aefa1840 con 0x55d7aef968d0 2024-04-22T13:40:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.862+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 0x55d7aefa2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c093f6640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4043820 tx=0x7f4bf4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab7cec507f15379 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf405a040 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8455+0+0 (secure 0 0 0) 0x7f4bf40037f0 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf4043450 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4bf4077050 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 10 client.18024 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d7af0154f0 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf4043450 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4bf4056890 con 0x55d7aef968d0 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 10 client.18024 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 10 client.18024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 20 client.18024 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7af015b50 0x55d7af017f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 4 client.18024 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:40:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.863+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7aee9a3d0 con 0x55d7af015b50 2024-04-22T13:40:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.864+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7af015b50 0x55d7af017f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.864+0000 7f4bed7fa640 10 client.18024 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.864+0000 7f4c09bf7640 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7af015b50 0x55d7af017f40 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1eb4dab96c69de3 server_cookie=e3048d9673c86a5b in_seq=0 out_seq=0 2024-04-22T13:40:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4bed7fa640 1 -- 172.21.15.87:0/368536656 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7aee9a3d0 con 0x55d7af015b50 2024-04-22T13:40:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4bed7fa640 10 client.18024 handle_command_reply: tid=0 2024-04-22T13:40:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 client.18024 shutdown 2024-04-22T13:40:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 msgr2=0x55d7aefa3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4043820 tx=0x7f4bf4005fd0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 shutdown_connections 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d7af015b50 0x55d7af017f40 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7aef9f0e0 0x55d7aefa3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4bed7fa640 0 client.18024 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d7aef968d0 0x55d7aefa3310 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 --2- 172.21.15.87:0/368536656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7aef95b60 0x55d7aefa2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 >> 172.21.15.87:0/368536656 conn(0x55d7aef5cc90 msgr2=0x55d7aef8f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 shutdown_connections 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 1 -- 172.21.15.87:0/368536656 wait complete. 2024-04-22T13:40:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0ea45740 20 client.18024 trim_cache size 0 max 16384 2024-04-22T13:40:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:40:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:40:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:40:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 msgr2=0x7f4bf00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 0x7f4bf00a6280 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4bf402c780 tx=0x7f4bf402c650 comp rx=0 tx=0).stop 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 msgr2=0x7f4c0419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.872+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e960 tx=0x7f4bfc00ee20 comp rx=0 tx=0).stop 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 shutdown_connections 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4bf00a3dc0 0x7f4bf00a6280 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c0419cfa0 0x7f4c0419d450 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c041072a0 0x7f4c04198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 --2- 172.21.15.87:0/494329559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c041021d0 0x7f4c041986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 >> 172.21.15.87:0/494329559 conn(0x7f4c040793e0 msgr2=0x7f4c040ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 shutdown_connections 2024-04-22T13:40:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:05.873+0000 7f4c0be5b640 1 -- 172.21.15.87:0/494329559 wait complete. 2024-04-22T13:40:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:06 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:40:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:06 smithi087 ceph-mon[27999]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T13:40:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:06 smithi087 ceph-mon[27999]: pgmap v256: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T13:40:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:06 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:40:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:06 smithi120 ceph-mon[31101]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T13:40:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:06 smithi120 ceph-mon[31101]: pgmap v256: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T13:40:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:06 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:40:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:06 smithi132 ceph-mon[31531]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T13:40:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:06 smithi132 ceph-mon[31531]: pgmap v256: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T13:40:08.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:08 smithi120 ceph-mon[31101]: pgmap v257: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 40 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:08 smithi132 ceph-mon[31531]: pgmap v257: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 40 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:08.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:08 smithi087 ceph-mon[27999]: pgmap v257: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 40 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:09 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:40:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:09 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:40:09.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:09 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:40:10.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:40:10.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:40:10.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:10 smithi120 ceph-mon[31101]: pgmap v258: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 39 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:10.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:10 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:40:10.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:40:10.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:40:10.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:10 smithi132 ceph-mon[31531]: pgmap v258: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 39 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:10.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:10 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:40:10.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:40:10.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:40:10.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:10 smithi087 ceph-mon[27999]: pgmap v258: 97 pgs: 6 active+clean+snaptrim_wait, 89 active+clean, 2 active+clean+snaptrim; 39 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:10.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:10 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:40:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:13 smithi120 ceph-mon[31101]: pgmap v259: 97 pgs: 97 active+clean; 37 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:13 smithi132 ceph-mon[31531]: pgmap v259: 97 pgs: 97 active+clean; 37 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:13.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:13 smithi087 ceph-mon[27999]: pgmap v259: 97 pgs: 97 active+clean; 37 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:40:15.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:15 smithi087 ceph-mon[27999]: pgmap v260: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-22T13:40:15.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:15 smithi087 ceph-mon[27999]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T13:40:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:15 smithi120 ceph-mon[31101]: pgmap v260: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-22T13:40:15.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:15 smithi120 ceph-mon[31101]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T13:40:15.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:15 smithi132 ceph-mon[31531]: pgmap v260: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-04-22T13:40:15.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:15 smithi132 ceph-mon[31531]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T13:40:17.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:17 smithi120 ceph-mon[31101]: pgmap v262: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:40:17.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:17 smithi132 ceph-mon[31531]: pgmap v262: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:40:17.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:17 smithi087 ceph-mon[27999]: pgmap v262: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:40:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 -- 172.21.15.87:0/2677826568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006a780 msgr2=0x7fee7006abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 --2- 172.21.15.87:0/2677826568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006a780 0x7fee7006abe0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fee64009900 tx=0x7fee6402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 -- 172.21.15.87:0/2677826568 shutdown_connections 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 --2- 172.21.15.87:0/2677826568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee7006b120 0x7fee7010edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 --2- 172.21.15.87:0/2677826568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006a780 0x7fee7006abe0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 --2- 172.21.15.87:0/2677826568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee701080e0 0x7fee701084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 -- 172.21.15.87:0/2677826568 >> 172.21.15.87:0/2677826568 conn(0x7fee70076c50 msgr2=0x7fee70077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 -- 172.21.15.87:0/2677826568 shutdown_connections 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.970+0000 7fee76493640 1 -- 172.21.15.87:0/2677826568 wait complete. 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 Processor -- start 2024-04-22T13:40:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 -- start start 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee7006a780 0x7fee70198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee701080e0 0x7fee7019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee701121b0 con 0x7fee7006b120 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee70112030 con 0x7fee7006a780 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee76493640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fee70112330 con 0x7fee701080e0 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee6f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee6f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44336/0 (socket says 172.21.15.87:44336) 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee6f7fe640 1 -- 172.21.15.87:0/873645312 learned_addr learned my addr 172.21.15.87:0/873645312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee6ffff640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee7006a780 0x7fee70198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.971+0000 7fee74a30640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee701080e0 0x7fee7019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 -- 172.21.15.87:0/873645312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee701080e0 msgr2=0x7fee7019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee701080e0 0x7fee7019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 -- 172.21.15.87:0/873645312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee7006a780 msgr2=0x7fee70198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee7006a780 0x7fee70198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 -- 172.21.15.87:0/873645312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7019d5b0 con 0x7fee7006b120 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fee64005940 tx=0x7fee6402d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6d7fa640 1 -- 172.21.15.87:0/873645312 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64043020 con 0x7fee7006b120 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7019d840 con 0x7fee7006b120 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6d7fa640 1 -- 172.21.15.87:0/873645312 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6402dc90 con 0x7fee7006b120 2024-04-22T13:40:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee6d7fa640 1 -- 172.21.15.87:0/873645312 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64003ab0 con 0x7fee7006b120 2024-04-22T13:40:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.972+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee701a5130 con 0x7fee7006b120 2024-04-22T13:40:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.974+0000 7fee6d7fa640 1 -- 172.21.15.87:0/873645312 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee64046020 con 0x7fee7006b120 2024-04-22T13:40:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.974+0000 7fee79167740 1 Processor -- start 2024-04-22T13:40:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 -- start start 2024-04-22T13:40:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6d7fa640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 0x7fee580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c0ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c190e0 0x55c768c1b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c768b748d0 con 0x55c768c108d0 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c768b11e80 con 0x55c768c0fb60 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee79167740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c768b70890 con 0x55c768c190e0 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee74a30640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 0x7fee580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6d7fa640 1 -- 172.21.15.87:0/873645312 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fee640f2750 con 0x7fee7006b120 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44342/0 (socket says 172.21.15.87:44342) 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 -- 172.21.15.87:0/195761910 learned_addr learned my addr 172.21.15.87:0/195761910 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c0ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 -- 172.21.15.87:0/195761910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c190e0 msgr2=0x55c768c1b4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c190e0 0x55c768c1b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 -- 172.21.15.87:0/195761910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 msgr2=0x55c768c0ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c0ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 -- 172.21.15.87:0/195761910 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c768b33320 con 0x55c768c108d0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee74a30640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 0x7fee580a6300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee68005fd0 tx=0x7fee68009840 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.975+0000 7fee6ffff640 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fee6000bc90 tx=0x7fee600107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816e107cbc42c9cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee567fc640 1 -- 172.21.15.87:0/195761910 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60004030 con 0x55c768c108d0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee567fc640 1 -- 172.21.15.87:0/195761910 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee600035f0 con 0x55c768c108d0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee567fc640 1 -- 172.21.15.87:0/195761910 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60011680 con 0x55c768c108d0 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 -- 172.21.15.87:0/195761910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 msgr2=0x55c768c0f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fee6000bc90 tx=0x7fee600107a0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 -- 172.21.15.87:0/195761910 shutdown_connections 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c190e0 0x55c768c1b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c0ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 --2- 172.21.15.87:0/195761910 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c0f620 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 -- 172.21.15.87:0/195761910 >> 172.21.15.87:0/195761910 conn(0x55c768bd6c90 msgr2=0x55c768c0a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 -- 172.21.15.87:0/195761910 shutdown_connections 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.976+0000 7fee79167740 1 -- 172.21.15.87:0/195761910 wait complete. 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 -- start start 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 Processor -- start 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c190e0 0x55c768c162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c768c1be10 con 0x55c768c108d0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c768afceb0 con 0x55c768c190e0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c768c1bf90 con 0x55c768c0fb60 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee79167740 10 client.0 _ll_register_callbacks cb 0x55c768b29b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee6f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee6f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44346/0 (socket says 172.21.15.87:44346) 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee6f7fe640 1 -- 172.21.15.87:0/3366941802 learned_addr learned my addr 172.21.15.87:0/3366941802 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c18a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.977+0000 7fee74a30640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c190e0 0x55c768c162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 -- 172.21.15.87:0/3366941802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 msgr2=0x55c768c18a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c18a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 -- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c190e0 msgr2=0x55c768c162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c190e0 0x55c768c162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 -- 172.21.15.87:0/3366941802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c768887210 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fee64006010 tx=0x7fee64005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8921e9217e1be064 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6402de40 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fee6403e550 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee600035f0 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fee640bf060 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee79167740 10 client.18036 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c768953620 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee600035f0 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.978+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fee640c7140 con 0x55c768c108d0 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 10 client.18036 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 10 client.18036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 20 client.18036 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c290b0 0x55c768c2b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 4 client.18036 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c768b88880 con 0x55c768c290b0 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c290b0 0x55c768c2b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c290b0 0x55c768c2b530 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5760e5fbd7b3cf12 server_cookie=61b4e5cbcfe4427 in_seq=0 out_seq=0 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.979+0000 7fee4f7fe640 10 client.18036 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3366941802 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c768b88880 con 0x55c768c290b0 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee4f7fe640 10 client.18036 handle_command_reply: tid=0 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 client.18036 shutdown 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 msgr2=0x55c768c15da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fee64006010 tx=0x7fee64005400 comp rx=0 tx=0).stop 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 shutdown_connections 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c290b0 0x55c768c2b530 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c190e0 0x55c768c162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c108d0 0x55c768c15da0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee4f7fe640 0 client.18036 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 --2- 172.21.15.87:0/3366941802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c18a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 >> 172.21.15.87:0/3366941802 conn(0x55c768bd6c90 msgr2=0x55c768c0c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 shutdown_connections 2024-04-22T13:40:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 1 -- 172.21.15.87:0/3366941802 wait complete. 2024-04-22T13:40:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.980+0000 7fee79167740 20 client.18036 trim_cache size 0 max 16384 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.995+0000 7fee79167740 1 Processor -- start 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.995+0000 7fee79167740 1 -- start start 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c24d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 0x55c768c18870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee701b01e0 con 0x55c768c190e0 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee701adab0 con 0x55c768c0fb60 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee79167740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fee701ab920 con 0x55c768c108d0 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44354/0 (socket says 172.21.15.87:44354) 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 -- 172.21.15.87:0/192660381 learned_addr learned my addr 172.21.15.87:0/192660381 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 0x55c768c18870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee6ffff640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c24d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 -- 172.21.15.87:0/192660381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 msgr2=0x55c768c18870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 0x55c768c18870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 -- 172.21.15.87:0/192660381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 msgr2=0x55c768c24d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c24d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 -- 172.21.15.87:0/192660381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c768c0ee30 con 0x55c768c190e0 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 0x55c768c18870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee6ffff640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c24d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee74a30640 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fee68010bd0 tx=0x7fee68013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffa25db5fafff0e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee55ffb640 1 -- 172.21.15.87:0/192660381 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee680206f0 con 0x55c768c190e0 2024-04-22T13:40:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee55ffb640 1 -- 172.21.15.87:0/192660381 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee68020d40 con 0x55c768c190e0 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.996+0000 7fee55ffb640 1 -- 172.21.15.87:0/192660381 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68028e50 con 0x55c768c190e0 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- 172.21.15.87:0/192660381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 msgr2=0x55c768c21630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fee68010bd0 tx=0x7fee68013000 comp rx=0 tx=0).stop 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- 172.21.15.87:0/192660381 shutdown_connections 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c21630 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c108d0 0x55c768c18870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 --2- 172.21.15.87:0/192660381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c0fb60 0x55c768c24d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- 172.21.15.87:0/192660381 >> 172.21.15.87:0/192660381 conn(0x55c768bd6c90 msgr2=0x55c768c14670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- 172.21.15.87:0/192660381 shutdown_connections 2024-04-22T13:40:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- 172.21.15.87:0/192660381 wait complete. 2024-04-22T13:40:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.997+0000 7fee79167740 1 -- start start 2024-04-22T13:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 Processor -- start 2024-04-22T13:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c12010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c108d0 0x55c768c12550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fee701abf70 con 0x55c768c190e0 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee701ad3e0 con 0x55c768c108d0 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fee701ac640 con 0x55c768c0fb60 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee79167740 10 client.0 _ll_register_callbacks cb 0x7fee640be730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44364/0 (socket says 172.21.15.87:44364) 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 -- 172.21.15.87:0/3230067826 learned_addr learned my addr 172.21.15.87:0/3230067826 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee6f7fe640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c108d0 0x55c768c12550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c12010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 -- 172.21.15.87:0/3230067826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 msgr2=0x55c768c12010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c12010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 -- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c108d0 msgr2=0x55c768c12550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c108d0 0x55c768c12550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 -- 172.21.15.87:0/3230067826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c768c13320 con 0x55c768c190e0 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c12010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee74a30640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fee68000f80 tx=0x7fee6802adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836558d357c1f850 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.998+0000 7fee4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68028ce0 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fee68004a40 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee68020d40 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fee68004c40 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 10 client.18042 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c768c93c90 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee68020d40 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fee6800f040 con 0x55c768c190e0 2024-04-22T13:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 10 client.18042 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 10 client.18042 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 20 client.18042 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c942b0 0x55c768c966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 4 client.18042 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c768b143d0 con 0x55c768c942b0 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:18.999+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c942b0 0x55c768c966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.000+0000 7fee4f7fe640 10 client.18042 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.000+0000 7fee6ffff640 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c942b0 0x55c768c966d0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9641ab4f89a79926 server_cookie=2ee426e24dd9d85f in_seq=0 out_seq=0 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee4f7fe640 1 -- 172.21.15.87:0/3230067826 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c768b143d0 con 0x55c768c942b0 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee4f7fe640 10 client.18042 handle_command_reply: tid=0 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 client.18042 shutdown 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 msgr2=0x55c768c12af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fee68000f80 tx=0x7fee6802adb0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 shutdown_connections 2024-04-22T13:40:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c768c942b0 0x55c768c966d0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c768c190e0 0x55c768c12af0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c768c108d0 0x55c768c12550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 --2- 172.21.15.87:0/3230067826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c768c0fb60 0x55c768c12010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee4f7fe640 0 client.18042 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 >> 172.21.15.87:0/3230067826 conn(0x55c768bd6c90 msgr2=0x55c768c10ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 shutdown_connections 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 1 -- 172.21.15.87:0/3230067826 wait complete. 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.007+0000 7fee79167740 20 client.18042 trim_cache size 0 max 16384 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:40:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 msgr2=0x7fee580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 0x7fee580a6300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee68005fd0 tx=0x7fee68009840 comp rx=0 tx=0).stop 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 msgr2=0x7fee70198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fee64005940 tx=0x7fee6402d890 comp rx=0 tx=0).stop 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 shutdown_connections 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fee580a3e40 0x7fee580a6300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee701080e0 0x7fee7019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fee7006b120 0x7fee70198b50 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 --2- 172.21.15.87:0/873645312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee7006a780 0x7fee70198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 >> 172.21.15.87:0/873645312 conn(0x7fee70076c50 msgr2=0x7fee700ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 shutdown_connections 2024-04-22T13:40:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:19.008+0000 7fee76493640 1 -- 172.21.15.87:0/873645312 wait complete. 2024-04-22T13:40:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:19 smithi120 ceph-mon[31101]: pgmap v263: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:19 smithi132 ceph-mon[31531]: pgmap v263: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:19.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:19 smithi087 ceph-mon[27999]: pgmap v263: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:20.968 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 -- 172.21.15.87:0/964383866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c101a50 msgr2=0x7f700c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 --2- 172.21.15.87:0/964383866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c101a50 0x7f700c105a90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009900 tx=0x7f6ff402bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 -- 172.21.15.87:0/964383866 shutdown_connections 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 --2- 172.21.15.87:0/964383866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c106170 0x7f700c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 --2- 172.21.15.87:0/964383866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c101a50 0x7f700c105a90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 --2- 172.21.15.87:0/964383866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c1010a0 0x7f700c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 -- 172.21.15.87:0/964383866 >> 172.21.15.87:0/964383866 conn(0x7f700c0771d0 msgr2=0x7f700c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 -- 172.21.15.87:0/964383866 shutdown_connections 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.084+0000 7f70118a5640 1 -- 172.21.15.87:0/964383866 wait complete. 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 Processor -- start 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 -- start start 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c1010a0 0x7f700c197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 0x7f700c1a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f700c111090 con 0x7f700c106170 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f700c110f10 con 0x7f700c1010a0 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f70118a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f700c111210 con 0x7f700c197dc0 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f7003fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f7003fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48794/0 (socket says 172.21.15.87:48794) 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f7003fff640 1 -- 172.21.15.87:0/595735821 learned_addr learned my addr 172.21.15.87:0/595735821 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.085+0000 7f700affd640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c1010a0 0x7f700c197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f700b7fe640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 0x7f700c1a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 -- 172.21.15.87:0/595735821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 msgr2=0x7f700c1a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 0x7f700c1a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 -- 172.21.15.87:0/595735821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c1010a0 msgr2=0x7f700c197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c1010a0 0x7f700c197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 -- 172.21.15.87:0/595735821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c1a6f80 con 0x7f700c106170 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f700b7fe640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 0x7f700c1a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f7003fff640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40045e0 tx=0x7f6ff4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4043020 con 0x7f700c106170 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff402cb70 con 0x7f700c106170 2024-04-22T13:40:21.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700c1a7210 con 0x7f700c106170 2024-04-22T13:40:21.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4003bf0 con 0x7f700c106170 2024-04-22T13:40:21.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.086+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f700c1a7700 con 0x7f700c106170 2024-04-22T13:40:21.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.087+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f700c069230 con 0x7f700c106170 2024-04-22T13:40:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.088+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ff4046020 con 0x7f700c106170 2024-04-22T13:40:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.089+0000 7f70108a3640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 0x7f6ff80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.089+0000 7f700affd640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 0x7f6ff80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.089+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f6ff40f2340 con 0x7f700c106170 2024-04-22T13:40:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.089+0000 7f700affd640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 0x7f6ff80a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0058c0 tx=0x7f6ffc00a2d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.093+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ff404d070 con 0x7f700c106170 2024-04-22T13:40:21.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.310+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f700c1093d0 con 0x7f700c106170 2024-04-22T13:40:21.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.310+0000 7f70108a3640 1 -- 172.21.15.87:0/595735821 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f6ff40beb30 con 0x7f700c106170 2024-04-22T13:40:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:40:21.306 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:40:21.307 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:40:21.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 msgr2=0x7f6ff80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 0x7f6ff80a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0058c0 tx=0x7f6ffc00a2d0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 msgr2=0x7f700c197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40045e0 tx=0x7f6ff4004610 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 shutdown_connections 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6ff80a3d20 0x7f6ff80a61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f700c197dc0 0x7f700c1a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f700c106170 0x7f700c197880 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 --2- 172.21.15.87:0/595735821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f700c1010a0 0x7f700c197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 >> 172.21.15.87:0/595735821 conn(0x7f700c0771d0 msgr2=0x7f700c0feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 shutdown_connections 2024-04-22T13:40:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.312+0000 7f70118a5640 1 -- 172.21.15.87:0/595735821 wait complete. 2024-04-22T13:40:21.319 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id f6f7ac25 --release --await' 2024-04-22T13:40:21.320 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id f6f7ac25 --release --await 2024-04-22T13:40:21.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 -- 172.21.15.87:0/4237041361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f41011a0 msgr2=0x7f89f4101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 --2- 172.21.15.87:0/4237041361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f41011a0 0x7f89f4101600 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009930 tx=0x7f89dc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 -- 172.21.15.87:0/4237041361 shutdown_connections 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 --2- 172.21.15.87:0/4237041361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f4101b40 0x7f89f410dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 --2- 172.21.15.87:0/4237041361 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f41011a0 0x7f89f4101600 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 --2- 172.21.15.87:0/4237041361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4107030 0x7f89f4107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 -- 172.21.15.87:0/4237041361 >> 172.21.15.87:0/4237041361 conn(0x7f89f4076c50 msgr2=0x7f89f4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 -- 172.21.15.87:0/4237041361 shutdown_connections 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.428+0000 7f89f949a640 1 -- 172.21.15.87:0/4237041361 wait complete. 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 Processor -- start 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 -- start start 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 0x7f89f419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4101b40 0x7f89f419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41110e0 con 0x7f89f4107030 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4110f60 con 0x7f89f41011a0 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f949a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4111260 con 0x7f89f4101b40 2024-04-22T13:40:21.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48814/0 (socket says 172.21.15.87:48814) 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f37fe640 1 -- 172.21.15.87:0/1523995718 learned_addr learned my addr 172.21.15.87:0/1523995718 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.429+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 0x7f89f419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4101b40 0x7f89f419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 -- 172.21.15.87:0/1523995718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4101b40 msgr2=0x7f89f419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4101b40 0x7f89f419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 -- 172.21.15.87:0/1523995718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 msgr2=0x7f89f419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 0x7f89f419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 -- 172.21.15.87:0/1523995718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f4198d50 con 0x7f89f4107030 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 0x7f89f419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f37fe640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00eaa0 tx=0x7f89ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f0ff9640 1 -- 172.21.15.87:0/1523995718 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00cdb0 con 0x7f89f4107030 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f0ff9640 1 -- 172.21.15.87:0/1523995718 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89ec005020 con 0x7f89f4107030 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f4199040 con 0x7f89f4107030 2024-04-22T13:40:21.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f0ff9640 1 -- 172.21.15.87:0/1523995718 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0116d0 con 0x7f89f4107030 2024-04-22T13:40:21.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.430+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f41a52e0 con 0x7f89f4107030 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.432+0000 7f89f0ff9640 1 -- 172.21.15.87:0/1523995718 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f89ec0051d0 con 0x7f89f4107030 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.432+0000 7f89fc169740 1 Processor -- start 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.432+0000 7f89fc169740 1 -- start start 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cf310 0x55e6231cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e623134500 con 0x55e6231d8810 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6230da6a0 con 0x55e6231cf310 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89fc169740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6231304c0 con 0x55e6231cfc30 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89f0ff9640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 0x7f89e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48830/0 (socket says 172.21.15.87:48830) 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 -- 172.21.15.87:0/1007516031 learned_addr learned my addr 172.21.15.87:0/1007516031 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89f37fe640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 0x7f89e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cf310 0x55e6231cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89f0ff9640 1 -- 172.21.15.87:0/1523995718 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f89ec0d0a90 con 0x7f89f4107030 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 -- 172.21.15.87:0/1007516031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 msgr2=0x55e6231d82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 -- 172.21.15.87:0/1007516031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cf310 msgr2=0x55e6231cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cf310 0x55e6231cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 -- 172.21.15.87:0/1007516031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6230f95e0 con 0x55e6231d8810 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89f37fe640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 0x7f89e00a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0ce820 tx=0x7f89ec06e4b0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005c40 tx=0x7f89dc0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b58c2d95389955 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.433+0000 7f89e97fa640 1 -- 172.21.15.87:0/1007516031 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc049070 con 0x55e6231d8810 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89e97fa640 1 -- 172.21.15.87:0/1007516031 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89dc005500 con 0x55e6231d8810 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89e97fa640 1 -- 172.21.15.87:0/1007516031 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc004900 con 0x55e6231d8810 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 -- 172.21.15.87:0/1007516031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 msgr2=0x55e6231dac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005c40 tx=0x7f89dc0058d0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 -- 172.21.15.87:0/1007516031 shutdown_connections 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231dac00 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 --2- 172.21.15.87:0/1007516031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cf310 0x55e6231cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 -- 172.21.15.87:0/1007516031 >> 172.21.15.87:0/1007516031 conn(0x55e6231c9a40 msgr2=0x55e6231c9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 -- 172.21.15.87:0/1007516031 shutdown_connections 2024-04-22T13:40:21.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.434+0000 7f89fc169740 1 -- 172.21.15.87:0/1007516031 wait complete. 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 -- start start 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 Processor -- start 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231d5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e6231db5a0 con 0x55e6231cfc30 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6230eda60 con 0x55e6231d8810 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6231db720 con 0x55e6231cf310 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89fc169740 10 client.0 _ll_register_callbacks cb 0x55e622e46110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48844/0 (socket says 172.21.15.87:48844) 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 -- 172.21.15.87:0/3145657167 learned_addr learned my addr 172.21.15.87:0/3145657167 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231d5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 -- 172.21.15.87:0/3145657167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 msgr2=0x55e6231d82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 -- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 msgr2=0x55e6231d5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231d5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 -- 172.21.15.87:0/3145657167 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e622e46920 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.435+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f89e400bfb0 tx=0x7f89e40128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196da0ba72f2d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4005220 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f89e40053c0 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89dc005500 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f89e40055c0 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 10 client.18060 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e622f12d30 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89dc005500 con 0x55e6231cfc30 2024-04-22T13:40:21.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f89e4019030 con 0x55e6231cfc30 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 10 client.18060 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 10 client.18060 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 20 client.18060 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6231e8aa0 0x55e6231eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 4 client.18060 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e623107a10 con 0x55e6231e8aa0 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.436+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6231e8aa0 0x55e6231eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.437+0000 7f89da7fc640 10 client.18060 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.437+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6231e8aa0 0x55e6231eaee0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78ca3d6d67e68e4c server_cookie=4994b7815ee5a202 in_seq=0 out_seq=0 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89da7fc640 1 -- 172.21.15.87:0/3145657167 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e623107a10 con 0x55e6231e8aa0 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89da7fc640 10 client.18060 handle_command_reply: tid=0 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 client.18060 shutdown 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 msgr2=0x55e6231d5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f89e400bfb0 tx=0x7f89e40128e0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 shutdown_connections 2024-04-22T13:40:21.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6231e8aa0 0x55e6231eaee0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cfc30 0x55e6231d5660 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89da7fc640 0 client.18060 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 --2- 172.21.15.87:0/3145657167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 >> 172.21.15.87:0/3145657167 conn(0x55e6231c9a40 msgr2=0x55e6231cb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 shutdown_connections 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 1 -- 172.21.15.87:0/3145657167 wait complete. 2024-04-22T13:40:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.438+0000 7f89fc169740 20 client.18060 trim_cache size 0 max 16384 2024-04-22T13:40:21.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 Processor -- start 2024-04-22T13:40:21.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 -- start start 2024-04-22T13:40:21.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41b02c0 con 0x55e6231d8810 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41adb90 con 0x55e6231cfc30 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89fc169740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41aba00 con 0x55e6231cf310 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48860/0 (socket says 172.21.15.87:48860) 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2450939979 learned_addr learned my addr 172.21.15.87:0/2450939979 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2450939979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 msgr2=0x55e6231d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2450939979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 msgr2=0x55e6231d82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2450939979 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6231d6110 con 0x55e6231d8810 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.455+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:21.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f89dc02c170 tx=0x7f89dc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc01e9a0931e5db4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89e8ff9640 1 -- 172.21.15.87:0/2450939979 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc05d070 con 0x55e6231d8810 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89e8ff9640 1 -- 172.21.15.87:0/2450939979 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89dc02cb00 con 0x55e6231d8810 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89e8ff9640 1 -- 172.21.15.87:0/2450939979 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc049040 con 0x55e6231d8810 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 -- 172.21.15.87:0/2450939979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 msgr2=0x55e6231f07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f89dc02c170 tx=0x7f89dc02c5d0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 -- 172.21.15.87:0/2450939979 shutdown_connections 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231d8810 0x55e6231f07b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231cfc30 0x55e6231d82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 --2- 172.21.15.87:0/2450939979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cf310 0x55e6231d7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.456+0000 7f89fc169740 1 -- 172.21.15.87:0/2450939979 >> 172.21.15.87:0/2450939979 conn(0x55e6231c9a40 msgr2=0x55e6231c8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- 172.21.15.87:0/2450939979 shutdown_connections 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- 172.21.15.87:0/2450939979 wait complete. 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- start start 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 Processor -- start 2024-04-22T13:40:21.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ac050 con 0x55e6231cf310 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ad4c0 con 0x55e6231d8810 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ac720 con 0x55e6231cfc30 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89fc169740 10 client.0 _ll_register_callbacks cb 0x7f89e400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48866/0 (socket says 172.21.15.87:48866) 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2151289706 learned_addr learned my addr 172.21.15.87:0/2151289706 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2151289706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 msgr2=0x55e6231dd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231dd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 msgr2=0x55e6231ddee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89ea7fc640 1 -- 172.21.15.87:0/2151289706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6231de6c0 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.457+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231dd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89ea7fc640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f89dc05b040 tx=0x7f89dc005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948618ea0d855588 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc05d070 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f89dc003a40 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89dc02cb00 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f89dc049040 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 10 client.18066 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e62324ff30 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89dc02cb00 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f89dc04f070 con 0x55e6231cf310 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 10 client.18066 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 10 client.18066 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 20 client.18066 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e623250100 0x55e6232524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 4 client.18066 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f6f7ac25", "release": true, "await": true}] 2024-04-22T13:40:21.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.458+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f6f7ac25", "release": true, "await": true}) -- 0x55e623141f20 con 0x55e623250100 2024-04-22T13:40:21.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.459+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e623250100 0x55e6232524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:21.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.459+0000 7f89da7fc640 10 client.18066 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:21.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.459+0000 7f89f2ffd640 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e623250100 0x55e6232524f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3264f7fa3c4a26e server_cookie=80f588116b9c57bc in_seq=0 out_seq=0 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89da7fc640 1 -- 172.21.15.87:0/2151289706 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e623141f20 con 0x55e623250100 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89da7fc640 10 client.18066 handle_command_reply: tid=0 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 client.18066 shutdown 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 msgr2=0x55e6231e0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f89dc05b040 tx=0x7f89dc005980 comp rx=0 tx=0).stop 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 shutdown_connections 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e623250100 0x55e6232524f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6231d8810 0x55e6231ddee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6231cfc30 0x55e6231dd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89da7fc640 0 client.18066 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 --2- 172.21.15.87:0/2151289706 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6231cf310 0x55e6231e0d70 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 >> 172.21.15.87:0/2151289706 conn(0x55e6231c9a40 msgr2=0x55e6231cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 shutdown_connections 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 1 -- 172.21.15.87:0/2151289706 wait complete. 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.462+0000 7f89fc169740 20 client.18066 trim_cache size 0 max 16384 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 32, 2024-04-22T13:40:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "f6f7ac25": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 32, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 94.6, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:40:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:21.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 msgr2=0x7f89e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 0x7f89e00a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0ce820 tx=0x7f89ec06e4b0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 msgr2=0x7f89f41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00eaa0 tx=0x7f89ec00ef60 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 shutdown_connections 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f89e00a3d20 0x7f89e00a61e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f89f4107030 0x7f89f41985d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f89f4101b40 0x7f89f419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 --2- 172.21.15.87:0/1523995718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89f41011a0 0x7f89f419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 >> 172.21.15.87:0/1523995718 conn(0x7f89f4076c50 msgr2=0x7f89f406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 shutdown_connections 2024-04-22T13:40:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:21.463+0000 7f89f949a640 1 -- 172.21.15.87:0/1523995718 wait complete. 2024-04-22T13:40:21.469 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: f6f7ac25, seconds elapsed: 35.6 2024-04-22T13:40:21.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:21 smithi120 ceph-mon[31101]: pgmap v264: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:21.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:21 smithi132 ceph-mon[31531]: pgmap v264: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:21.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:21 smithi087 ceph-mon[27999]: pgmap v264: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/595735821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:22.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/595735821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/595735821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:23.614 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:23 smithi132 ceph-mon[31531]: pgmap v265: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:23.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:23 smithi120 ceph-mon[31101]: pgmap v265: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:23.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:23 smithi087 ceph-mon[27999]: pgmap v265: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T13:40:24.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:24 smithi120 ceph-mon[31101]: pgmap v266: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:40:24.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:24 smithi132 ceph-mon[31531]: pgmap v266: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:40:24.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:24 smithi087 ceph-mon[27999]: pgmap v266: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:40:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:26 smithi120 ceph-mon[31101]: pgmap v267: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:40:26.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:26 smithi132 ceph-mon[31531]: pgmap v267: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:40:26.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:26 smithi087 ceph-mon[27999]: pgmap v267: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:40:28.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:28 smithi120 ceph-mon[31101]: pgmap v268: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:40:28.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:28 smithi132 ceph-mon[31531]: pgmap v268: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:40:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:28 smithi087 ceph-mon[27999]: pgmap v268: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:40:30.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:30 smithi120 ceph-mon[31101]: pgmap v269: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:30.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:30 smithi132 ceph-mon[31531]: pgmap v269: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:30.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:30 smithi087 ceph-mon[27999]: pgmap v269: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:32.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:40:32.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:40:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 -- 172.21.15.87:0/438727164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc41810e810 msgr2=0x7fc418110c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 --2- 172.21.15.87:0/438727164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc41810e810 0x7fc418110c20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc410009ca0 tx=0x7fc41002c180 comp rx=0 tx=0).stop 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 -- 172.21.15.87:0/438727164 shutdown_connections 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 --2- 172.21.15.87:0/438727164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc41810e810 0x7fc418110c20 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 --2- 172.21.15.87:0/438727164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074c40 0x7fc41810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 --2- 172.21.15.87:0/438727164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418074320 0x7fc418074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 -- 172.21.15.87:0/438727164 >> 172.21.15.87:0/438727164 conn(0x7fc4180fda00 msgr2=0x7fc4180ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 -- 172.21.15.87:0/438727164 shutdown_connections 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.104+0000 7fc41ea11640 1 -- 172.21.15.87:0/438727164 wait complete. 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.105+0000 7fc41ea11640 1 Processor -- start 2024-04-22T13:40:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.105+0000 7fc41ea11640 1 -- start start 2024-04-22T13:40:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.105+0000 7fc41ea11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 0x7fc4181a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.105+0000 7fc41ea11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.105+0000 7fc41ea11640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc41810e810 0x7fc41819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc41ea11640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc418114400 con 0x7fc418074c40 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc41ea11640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc418114280 con 0x7fc418074320 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc41ea11640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc418114580 con 0x7fc41810e810 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56644/0 (socket says 172.21.15.87:56644) 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 -- 172.21.15.87:0/3614675366 learned_addr learned my addr 172.21.15.87:0/3614675366 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc41810e810 0x7fc41819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc417fff640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 0x7fc4181a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 -- 172.21.15.87:0/3614675366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc41810e810 msgr2=0x7fc41819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc41810e810 0x7fc41819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 -- 172.21.15.87:0/3614675366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 msgr2=0x7fc4181a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 0x7fc4181a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 -- 172.21.15.87:0/3614675366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41819b000 con 0x7fc418074c40 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc417fff640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 0x7fc4181a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e9f0 tx=0x7fc40c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4157fa640 1 -- 172.21.15.87:0/3614675366 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00cd80 con 0x7fc418074c40 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41819b2f0 con 0x7fc418074c40 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.106+0000 7fc4157fa640 1 -- 172.21.15.87:0/3614675366 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c0108e0 con 0x7fc418074c40 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.107+0000 7fc4157fa640 1 -- 172.21.15.87:0/3614675366 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c018980 con 0x7fc418074c40 2024-04-22T13:40:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.107+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4181a75b0 con 0x7fc418074c40 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc4157fa640 1 -- 172.21.15.87:0/3614675366 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc40c010430 con 0x7fc418074c40 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc421600740 1 Processor -- start 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc421600740 1 -- start start 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc421600740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546692620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc421600740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.109+0000 7fc421600740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc421600740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c5465f78d0 con 0x55c54669c0e0 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc421600740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c546594e80 con 0x55c546692b60 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc421600740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5465f3890 con 0x55c5466938d0 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc417fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56654/0 (socket says 172.21.15.87:56654) 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc417fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:42378/0 (socket says 172.21.15.87:42378) 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 -- 172.21.15.87:0/3448328979 learned_addr learned my addr 172.21.15.87:0/3448328979 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546692620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4157fa640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 0x7fc4000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 0x7fc4000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4157fa640 1 -- 172.21.15.87:0/3614675366 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fc40c0d0b60 con 0x7fc418074c40 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 -- 172.21.15.87:0/3448328979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 msgr2=0x55c546692620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546692620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 -- 172.21.15.87:0/3448328979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 msgr2=0x55c546692fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 -- 172.21.15.87:0/3448328979 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5465b6320 con 0x55c54669c0e0 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc417fff640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546692620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.110+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc40c09c7e0 tx=0x7fc40c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecd02cccfb31e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc3fe7fc640 1 -- 172.21.15.87:0/3448328979 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c010720 con 0x55c54669c0e0 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 0x7fc4000a6280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc41002c790 tx=0x7fc41002c180 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc3fe7fc640 1 -- 172.21.15.87:0/3448328979 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c09d1e0 con 0x55c54669c0e0 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc3fe7fc640 1 -- 172.21.15.87:0/3448328979 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0bd070 con 0x55c54669c0e0 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc421600740 1 -- 172.21.15.87:0/3448328979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 msgr2=0x55c54669e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.111+0000 7fc421600740 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc40c09c7e0 tx=0x7fc40c00cf20 comp rx=0 tx=0).stop 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 -- 172.21.15.87:0/3448328979 shutdown_connections 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c54669c0e0 0x55c54669e4d0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c546692fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- 172.21.15.87:0/3448328979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546692620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 -- 172.21.15.87:0/3448328979 >> 172.21.15.87:0/3448328979 conn(0x55c546659c90 msgr2=0x55c54668d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 -- 172.21.15.87:0/3448328979 shutdown_connections 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 -- 172.21.15.87:0/3448328979 wait complete. 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 -- start start 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 Processor -- start 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546698da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.112+0000 7fc421600740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54669c0e0 0x55c5466992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c54669edb0 con 0x55c546692b60 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c54657feb0 con 0x55c54669c0e0 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c54669ef30 con 0x55c5466938d0 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 10 client.0 _ll_register_callbacks cb 0x55c5465acb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56670/0 (socket says 172.21.15.87:56670) 2024-04-22T13:40:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 -- 172.21.15.87:0/4176746184 learned_addr learned my addr 172.21.15.87:0/4176746184 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc4177fe640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546698da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc41cfae640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54669c0e0 0x55c5466992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 -- 172.21.15.87:0/4176746184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 msgr2=0x55c546698da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546698da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 -- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54669c0e0 msgr2=0x55c5466992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54669c0e0 0x55c5466992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 -- 172.21.15.87:0/4176746184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54630a210 con 0x55c546692b60 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc417fff640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc408013760 tx=0x7fc408013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2a0aeda5bf64d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc3f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408011e30 con 0x55c546692b60 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fc408005020 con 0x55c546692b60 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c09d1e0 con 0x55c546692b60 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc408005260 con 0x55c546692b60 2024-04-22T13:40:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 10 client.18078 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.113+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c5463d6620 con 0x55c546692b60 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c09d1e0 con 0x55c546692b60 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc40801a030 con 0x55c546692b60 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 10 client.18078 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 10 client.18078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 20 client.18078 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5466ac070 0x55c5466ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 4 client.18078 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c54660b880 con 0x55c5466ac070 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.114+0000 7fc4177fe640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5466ac070 0x55c5466ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.115+0000 7fc3f77fe640 10 client.18078 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.115+0000 7fc4177fe640 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5466ac070 0x55c5466ae4a0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34540b0e21f4986 server_cookie=9fa6d29883ff9a89 in_seq=0 out_seq=0 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.115+0000 7fc3f77fe640 1 -- 172.21.15.87:0/4176746184 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c54660b880 con 0x55c5466ac070 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.115+0000 7fc3f77fe640 10 client.18078 handle_command_reply: tid=0 2024-04-22T13:40:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.115+0000 7fc421600740 1 client.18078 shutdown 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 msgr2=0x55c54669b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc408013760 tx=0x7fc408013c20 comp rx=0 tx=0).stop 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 shutdown_connections 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5466ac070 0x55c5466ae4a0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54669c0e0 0x55c5466992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc3f77fe640 0 client.18078 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5466938d0 0x55c546698da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 --2- 172.21.15.87:0/4176746184 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c546692b60 0x55c54669b9a0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 >> 172.21.15.87:0/4176746184 conn(0x55c546659c90 msgr2=0x55c54668f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 shutdown_connections 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 1 -- 172.21.15.87:0/4176746184 wait complete. 2024-04-22T13:40:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.116+0000 7fc421600740 20 client.18078 trim_cache size 0 max 16384 2024-04-22T13:40:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.130+0000 7fc421600740 1 Processor -- start 2024-04-22T13:40:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 -- start start 2024-04-22T13:40:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c5466b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181b2630 con 0x55c5466938d0 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181aff00 con 0x55c546692b60 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc421600740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181add70 con 0x55c54669c0e0 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56676/0 (socket says 172.21.15.87:56676) 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 -- 172.21.15.87:0/3815084148 learned_addr learned my addr 172.21.15.87:0/3815084148 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c5466b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc417fff640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 -- 172.21.15.87:0/3815084148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 msgr2=0x55c5466b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c5466b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 -- 172.21.15.87:0/3815084148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 msgr2=0x55c5466a7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 -- 172.21.15.87:0/3815084148 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c546691e30 con 0x55c5466938d0 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc417fff640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc41cfae640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c5466b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.131+0000 7fc4177fe640 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc40c098f40 tx=0x7fc40c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680e97418b9ff06a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc3fdffb640 1 -- 172.21.15.87:0/3815084148 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c05f070 con 0x55c5466938d0 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc3fdffb640 1 -- 172.21.15.87:0/3815084148 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c0a5750 con 0x55c5466938d0 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc3fdffb640 1 -- 172.21.15.87:0/3815084148 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c099150 con 0x55c5466938d0 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 -- 172.21.15.87:0/3815084148 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 msgr2=0x55c54669b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc40c098f40 tx=0x7fc40c0bd040 comp rx=0 tx=0).stop 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 -- 172.21.15.87:0/3815084148 shutdown_connections 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c5466b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c54669b4d0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 --2- 172.21.15.87:0/3815084148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 -- 172.21.15.87:0/3815084148 >> 172.21.15.87:0/3815084148 conn(0x55c546659c90 msgr2=0x55c546698d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 -- 172.21.15.87:0/3815084148 shutdown_connections 2024-04-22T13:40:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.132+0000 7fc421600740 1 -- 172.21.15.87:0/3815084148 wait complete. 2024-04-22T13:40:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 -- start start 2024-04-22T13:40:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 Processor -- start 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c546695dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181ae3c0 con 0x55c5466938d0 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181af830 con 0x55c546692b60 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4181aea90 con 0x55c54669c0e0 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc421600740 10 client.0 _ll_register_callbacks cb 0x7fc408009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc4177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56682/0 (socket says 172.21.15.87:56682) 2024-04-22T13:40:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.133+0000 7fc4177fe640 1 -- 172.21.15.87:0/353217523 learned_addr learned my addr 172.21.15.87:0/353217523 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc417fff640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc41cfae640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c546695dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 -- 172.21.15.87:0/353217523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 msgr2=0x55c546695dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c546695dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 -- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 msgr2=0x55c5466a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 -- 172.21.15.87:0/353217523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c546688f30 con 0x55c5466938d0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc41cfae640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c546695dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc4177fe640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc40c09c7e0 tx=0x7fc40c0d0de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96d8c14085b74d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0b5070 con 0x55c5466938d0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fc40c099150 con 0x55c5466938d0 2024-04-22T13:40:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c0a5750 con 0x55c5466938d0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc40c05f040 con 0x55c5466938d0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc421600740 10 client.18084 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c546712500 con 0x55c5466938d0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c0a5750 con 0x55c5466938d0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.134+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc40c09cd40 con 0x55c5466938d0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 10 client.18084 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 10 client.18084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 20 client.18084 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5467127a0 0x55c546714b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 4 client.18084 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c5465973d0 con 0x55c5467127a0 2024-04-22T13:40:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc417fff640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5467127a0 0x55c546714b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc417fff640 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5467127a0 0x55c546714b90 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bf1321f605ff01a server_cookie=206b62b5d28e0d84 in_seq=0 out_seq=0 2024-04-22T13:40:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.135+0000 7fc3f77fe640 10 client.18084 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc3f77fe640 1 -- 172.21.15.87:0/353217523 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c5465973d0 con 0x55c5467127a0 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc3f77fe640 10 client.18084 handle_command_reply: tid=0 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 client.18084 shutdown 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 msgr2=0x55c546695840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc40c09c7e0 tx=0x7fc40c0d0de0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 shutdown_connections 2024-04-22T13:40:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5467127a0 0x55c546714b90 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54669c0e0 0x55c546695dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5466938d0 0x55c546695840 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 --2- 172.21.15.87:0/353217523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c546692b60 0x55c5466a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 >> 172.21.15.87:0/353217523 conn(0x55c546659c90 msgr2=0x55c546693d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc3f77fe640 0 client.18084 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 shutdown_connections 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 1 -- 172.21.15.87:0/353217523 wait complete. 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.141+0000 7fc421600740 20 client.18084 trim_cache size 0 max 16384 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:40:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:40:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:40:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:40:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 msgr2=0x7fc4000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 0x7fc4000a6280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc41002c790 tx=0x7fc41002c180 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 msgr2=0x7fc4181a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e9f0 tx=0x7fc40c00eeb0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 shutdown_connections 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4000a3dc0 0x7fc4000a6280 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc41810e810 0x7fc41819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc418074c40 0x7fc4181a0b70 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 --2- 172.21.15.87:0/3614675366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc418074320 0x7fc4181a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 >> 172.21.15.87:0/3614675366 conn(0x7fc4180fda00 msgr2=0x7fc4180fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 shutdown_connections 2024-04-22T13:40:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:32.142+0000 7fc41ea11640 1 -- 172.21.15.87:0/3614675366 wait complete. 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 -- 172.21.15.87:0/540697715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec1031b0 msgr2=0x7f37ec103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/540697715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec1031b0 0x7f37ec103590 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009900 tx=0x7f37e402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 -- 172.21.15.87:0/540697715 shutdown_connections 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/540697715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec10c580 0x7f37ec10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/540697715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 0x7f37ec10bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/540697715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec1031b0 0x7f37ec103590 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 -- 172.21.15.87:0/540697715 >> 172.21.15.87:0/540697715 conn(0x7f37ec0771d0 msgr2=0x7f37ec0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 -- 172.21.15.87:0/540697715 shutdown_connections 2024-04-22T13:40:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.190+0000 7f37f0c9a640 1 -- 172.21.15.87:0/540697715 wait complete. 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 Processor -- start 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 -- start start 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec1031b0 0x7f37ec198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 0x7f37ec198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1121d0 con 0x7f37ec10c580 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec112050 con 0x7f37ec103ad0 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37f0c9a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec112350 con 0x7f37ec1031b0 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56698/0 (socket says 172.21.15.87:56698) 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 -- 172.21.15.87:0/708291789 learned_addr learned my addr 172.21.15.87:0/708291789 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 0x7f37ec198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ea59c640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec1031b0 0x7f37ec198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 -- 172.21.15.87:0/708291789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec1031b0 msgr2=0x7f37ec198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec1031b0 0x7f37ec198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 -- 172.21.15.87:0/708291789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 msgr2=0x7f37ec198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 0x7f37ec198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.191+0000 7f37ead9d640 1 -- 172.21.15.87:0/708291789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec19d720 con 0x7f37ec10c580 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37ead9d640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00cab0 tx=0x7f37dc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37db7fe640 1 -- 172.21.15.87:0/708291789 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00ae20 con 0x7f37ec10c580 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec19da10 con 0x7f37ec10c580 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37db7fe640 1 -- 172.21.15.87:0/708291789 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc00f900 con 0x7f37ec10c580 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37db7fe640 1 -- 172.21.15.87:0/708291789 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc018a50 con 0x7f37ec10c580 2024-04-22T13:40:32.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.192+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec1a52a0 con 0x7f37ec10c580 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.194+0000 7f37db7fe640 1 -- 172.21.15.87:0/708291789 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f37dc00f450 con 0x7f37ec10c580 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.194+0000 7f37f3969740 1 Processor -- start 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.194+0000 7f37f3969740 1 -- start start 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.194+0000 7f37db7fe640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37db7fe640 1 -- 172.21.15.87:0/708291789 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f37dc0d0a30 con 0x7f37ec10c580 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15ea0 0x55558ae1e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae20e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55558ad7a1e0 con 0x55558ae15580 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55558ad17390 con 0x55558ae1ea80 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37f3969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55558ad761a0 con 0x55558ae15ea0 2024-04-22T13:40:32.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56706/0 (socket says 172.21.15.87:56706) 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 -- 172.21.15.87:0/2813651113 learned_addr learned my addr 172.21.15.87:0/2813651113 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15ea0 0x55558ae1e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ead9d640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae20e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 -- 172.21.15.87:0/2813651113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15ea0 msgr2=0x55558ae1e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15ea0 0x55558ae1e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 -- 172.21.15.87:0/2813651113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 msgr2=0x55558ae20e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae20e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 -- 172.21.15.87:0/2813651113 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55558ab567d0 con 0x55558ae15580 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ead9d640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae20e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 0x7f37d40a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f37ec1006a0 tx=0x7f37e00103d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37ea59c640 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009900 tx=0x7f37e4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7be9d3f0795fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.195+0000 7f37d8ff9640 1 -- 172.21.15.87:0/2813651113 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4004230 con 0x55558ae15580 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37d8ff9640 1 -- 172.21.15.87:0/2813651113 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e40043d0 con 0x55558ae15580 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37d8ff9640 1 -- 172.21.15.87:0/2813651113 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4004920 con 0x55558ae15580 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 -- 172.21.15.87:0/2813651113 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 msgr2=0x55558ae15960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009900 tx=0x7f37e4004040 comp rx=0 tx=0).stop 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 -- 172.21.15.87:0/2813651113 shutdown_connections 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae20e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15ea0 0x55558ae1e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 --2- 172.21.15.87:0/2813651113 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15580 0x55558ae15960 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 -- 172.21.15.87:0/2813651113 >> 172.21.15.87:0/2813651113 conn(0x55558ae0fcb0 msgr2=0x55558ae10090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 -- 172.21.15.87:0/2813651113 shutdown_connections 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.196+0000 7f37f3969740 1 -- 172.21.15.87:0/2813651113 wait complete. 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 -- start start 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 Processor -- start 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae1ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae1bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55558ae21810 con 0x55558ae1ea80 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55558ad332f0 con 0x55558ae15ea0 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55558ae21990 con 0x55558ae15580 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37f3969740 10 client.0 _ll_register_callbacks cb 0x55558ad40580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56714/0 (socket says 172.21.15.87:56714) 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 -- 172.21.15.87:0/1011723139 learned_addr learned my addr 172.21.15.87:0/1011723139 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae1bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae1ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 -- 172.21.15.87:0/1011723139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 msgr2=0x55558ae1ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae1ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 -- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 msgr2=0x55558ae1bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae1bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 -- 172.21.15.87:0/1011723139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55558aa8c3f0 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae1bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae1ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f37dc09c710 tx=0x7f37dc00f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f56d3f57d925473 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.197+0000 7f37d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0a0dc0 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f37dc099080 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e40043d0 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f37dc0a5680 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 10 client.18096 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55558aa4c0f0 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e40043d0 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f37dc020030 con 0x55558ae1ea80 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 10 client.18096 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 20 client.18096 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae2e9e0 0x55558ae30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 4 client.18096 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55558ad87c00 con 0x55558ae2e9e0 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.198+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae2e9e0 0x55558ae30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.199+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae2e9e0 0x55558ae30e10 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f775c9fcd1c2de server_cookie=3a4dda0831141f10 in_seq=0 out_seq=0 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.199+0000 7f37d1ffb640 10 client.18096 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.199+0000 7f37d1ffb640 1 -- 172.21.15.87:0/1011723139 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55558ad87c00 con 0x55558ae2e9e0 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37d1ffb640 10 client.18096 handle_command_reply: tid=0 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 client.18096 shutdown 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 msgr2=0x55558ae1c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f37dc09c710 tx=0x7f37dc00f880 comp rx=0 tx=0).stop 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 shutdown_connections 2024-04-22T13:40:32.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae2e9e0 0x55558ae30e10 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1c4a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae1bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37d1ffb640 0 client.18096 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 --2- 172.21.15.87:0/1011723139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae1ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 >> 172.21.15.87:0/1011723139 conn(0x55558ae0fcb0 msgr2=0x55558ae11a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 shutdown_connections 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 1 -- 172.21.15.87:0/1011723139 wait complete. 2024-04-22T13:40:32.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.200+0000 7f37f3969740 20 client.18096 trim_cache size 0 max 16384 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 Processor -- start 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 -- start start 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae2a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae10090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1b0290 con 0x55558ae1ea80 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1adb60 con 0x55558ae15ea0 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37f3969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1ab9d0 con 0x55558ae15580 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56730/0 (socket says 172.21.15.87:56730) 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37ead9d640 1 -- 172.21.15.87:0/1943590270 learned_addr learned my addr 172.21.15.87:0/1943590270 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.217+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae10090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 -- 172.21.15.87:0/1943590270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 msgr2=0x55558ae2a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae2a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae2a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 -- 172.21.15.87:0/1943590270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 msgr2=0x55558ae10090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae10090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 -- 172.21.15.87:0/1943590270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55558ae0fae0 con 0x55558ae1ea80 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ea59c640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae2a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37ead9d640 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f37dc098e70 tx=0x7f37dc09cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3112b512c213d36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37d3fff640 1 -- 172.21.15.87:0/1943590270 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc07f020 con 0x55558ae1ea80 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37d3fff640 1 -- 172.21.15.87:0/1943590270 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc099080 con 0x55558ae1ea80 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37d3fff640 1 -- 172.21.15.87:0/1943590270 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0a51f0 con 0x55558ae1ea80 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 -- 172.21.15.87:0/1943590270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 msgr2=0x55558ae1b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f37dc098e70 tx=0x7f37dc09cb90 comp rx=0 tx=0).stop 2024-04-22T13:40:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 -- 172.21.15.87:0/1943590270 shutdown_connections 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae1ea80 0x55558ae1b300 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae15ea0 0x55558ae10090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 --2- 172.21.15.87:0/1943590270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae2a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 -- 172.21.15.87:0/1943590270 >> 172.21.15.87:0/1943590270 conn(0x55558ae0fcb0 msgr2=0x55558ae0e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 -- 172.21.15.87:0/1943590270 shutdown_connections 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.218+0000 7f37f3969740 1 -- 172.21.15.87:0/1943590270 wait complete. 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 -- start start 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 Processor -- start 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae26f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae22860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1ac020 con 0x55558ae15ea0 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1ad490 con 0x55558ae1ea80 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec1ac6f0 con 0x55558ae15580 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37f3969740 10 client.0 _ll_register_callbacks cb 0x7f37dc098cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56742/0 (socket says 172.21.15.87:56742) 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 -- 172.21.15.87:0/4008972037 learned_addr learned my addr 172.21.15.87:0/4008972037 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37ead9d640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae22860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37ea59c640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae26f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 -- 172.21.15.87:0/4008972037 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 msgr2=0x55558ae26f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae26f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 -- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 msgr2=0x55558ae22860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae22860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37e9d9b640 1 -- 172.21.15.87:0/4008972037 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55558ae23060 con 0x55558ae15ea0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.219+0000 7f37ead9d640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae22860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37e9d9b640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f37e001a000 tx=0x7f37e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ab1f7c609435ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0005020 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f37e00051c0 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc099080 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f37e00053c0 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 10 client.18102 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55558ae94f10 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc099080 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f37e001f8c0 con 0x55558ae15ea0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 10 client.18102 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 20 client.18102 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae956f0 0x55558ae97b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 4 client.18102 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.220+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55558ad81a70 con 0x55558ae956f0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37ea59c640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae956f0 0x55558ae97b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37ea59c640 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae956f0 0x55558ae97b10 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1344ddf7cfaa410 server_cookie=d354c1977593e58e in_seq=0 out_seq=0 2024-04-22T13:40:32.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37d1ffb640 10 client.18102 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37d1ffb640 1 -- 172.21.15.87:0/4008972037 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55558ad81a70 con 0x55558ae956f0 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37d1ffb640 10 client.18102 handle_command_reply: tid=0 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 client.18102 shutdown 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 msgr2=0x55558ae22320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f37e001a000 tx=0x7f37e0004730 comp rx=0 tx=0).stop 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 shutdown_connections 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55558ae956f0 0x55558ae97b10 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55558ae1ea80 0x55558ae22860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55558ae15ea0 0x55558ae22320 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37d1ffb640 0 client.18102 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 --2- 172.21.15.87:0/4008972037 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55558ae15580 0x55558ae26f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 >> 172.21.15.87:0/4008972037 conn(0x55558ae0fcb0 msgr2=0x55558ae1d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 shutdown_connections 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 1 -- 172.21.15.87:0/4008972037 wait complete. 2024-04-22T13:40:32.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.221+0000 7f37f3969740 20 client.18102 trim_cache size 0 max 16384 2024-04-22T13:40:32.225 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:32.225 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:40:32.225 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:40:32.225 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 msgr2=0x7f37d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 0x7f37d40a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f37ec1006a0 tx=0x7f37e00103d0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 msgr2=0x7f37ec19cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00cab0 tx=0x7f37dc00cf70 comp rx=0 tx=0).stop 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 shutdown_connections 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37ec10c580 0x7f37ec19cff0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37ec103ad0 0x7f37ec198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 --2- 172.21.15.87:0/708291789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37ec1031b0 0x7f37ec198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 >> 172.21.15.87:0/708291789 conn(0x7f37ec0771d0 msgr2=0x7f37ec0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 shutdown_connections 2024-04-22T13:40:32.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.222+0000 7f37f0c9a640 1 -- 172.21.15.87:0/708291789 wait complete. 2024-04-22T13:40:32.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:40:32.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- 172.21.15.87:0/1796001512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc06a640 msgr2=0x7f09cc06aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 --2- 172.21.15.87:0/1796001512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc06a640 0x7f09cc06aaa0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009930 tx=0x7f09bc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- 172.21.15.87:0/1796001512 shutdown_connections 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 --2- 172.21.15.87:0/1796001512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc10ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 --2- 172.21.15.87:0/1796001512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc06a640 0x7f09cc06aaa0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 --2- 172.21.15.87:0/1796001512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc108010 0x7f09cc1083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- 172.21.15.87:0/1796001512 >> 172.21.15.87:0/1796001512 conn(0x7f09cc077e30 msgr2=0x7f09cc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- 172.21.15.87:0/1796001512 shutdown_connections 2024-04-22T13:40:32.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- 172.21.15.87:0/1796001512 wait complete. 2024-04-22T13:40:32.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 Processor -- start 2024-04-22T13:40:32.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.373+0000 7f09d29b2640 1 -- start start 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc06a640 0x7f09cc101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc112270 con 0x7f09cc108010 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1120f0 con 0x7f09cc06a640 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1123f0 con 0x7f09cc06afe0 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56762/0 (socket says 172.21.15.87:56762) 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 -- 172.21.15.87:0/3396188732 learned_addr learned my addr 172.21.15.87:0/3396188732 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09cb7fe640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09cbfff640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc06a640 0x7f09cc101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 -- 172.21.15.87:0/3396188732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 msgr2=0x7f09cc101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 -- 172.21.15.87:0/3396188732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc06a640 msgr2=0x7f09cc101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc06a640 0x7f09cc101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 -- 172.21.15.87:0/3396188732 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc1a6d70 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09cb7fe640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d0f4f640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f09c400eaa0 tx=0x7f09c400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c400cdb0 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c4005020 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc1a7060 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.374+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c40116d0 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.375+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1a7580 con 0x7f09cc108010 2024-04-22T13:40:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.375+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc069230 con 0x7f09cc108010 2024-04-22T13:40:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.377+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f09c40051d0 con 0x7f09cc108010 2024-04-22T13:40:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.377+0000 7f09c97fa640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.377+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f09c40d0e00 con 0x7f09cc108010 2024-04-22T13:40:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.377+0000 7f09cbfff640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.378+0000 7f09cbfff640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 0x7f09b40a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009890 tx=0x7f09c0008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.381+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09c4016030 con 0x7f09cc108010 2024-04-22T13:40:32.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:32 smithi087 ceph-mon[27999]: pgmap v270: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:32.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.597+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09cc10a470 con 0x7f09cc108010 2024-04-22T13:40:32.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.597+0000 7f09c97fa640 1 -- 172.21.15.87:0/3396188732 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f09c409d4e0 con 0x7f09cc108010 2024-04-22T13:40:32.593 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:40:32.594 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:40:32.594 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:40:32.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.599+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 msgr2=0x7f09b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.599+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 0x7f09b40a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009890 tx=0x7f09c0008040 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.599+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 msgr2=0x7f09cc101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.599+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f09c400eaa0 tx=0x7f09c400ef60 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 shutdown_connections 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09cc108010 0x7f09cc101c80 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f09cc06afe0 0x7f09cc101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 --2- 172.21.15.87:0/3396188732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09cc06a640 0x7f09cc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 >> 172.21.15.87:0/3396188732 conn(0x7f09cc077e30 msgr2=0x7f09cc0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 shutdown_connections 2024-04-22T13:40:32.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.600+0000 7f09d29b2640 1 -- 172.21.15.87:0/3396188732 wait complete. 2024-04-22T13:40:32.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:40:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:32 smithi120 ceph-mon[31101]: pgmap v270: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:32.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/1758991985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c4101a40 msgr2=0x7fb2c4105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/1758991985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c4101a40 0x7fb2c4105a80 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009900 tx=0x7fb2b402bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:32.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/1758991985 shutdown_connections 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/1758991985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/1758991985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c4101a40 0x7fb2c4105a80 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/1758991985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 0x7fb2c4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/1758991985 >> 172.21.15.87:0/1758991985 conn(0x7fb2c40771d0 msgr2=0x7fb2c40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/1758991985 shutdown_connections 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.711+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/1758991985 wait complete. 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 Processor -- start 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 -- start start 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 0x7fb2c4199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41110a0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c4110f20 con 0x7fb2c4106160 2024-04-22T13:40:32.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2cbcc1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c4111220 con 0x7fb2c4101090 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56786/0 (socket says 172.21.15.87:56786) 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 -- 172.21.15.87:0/3385324233 learned_addr learned my addr 172.21.15.87:0/3385324233 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 0x7fb2c4199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 -- 172.21.15.87:0/3385324233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 msgr2=0x7fb2c4199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 0x7fb2c4199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 -- 172.21.15.87:0/3385324233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 msgr2=0x7fb2c4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 -- 172.21.15.87:0/3385324233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c41a4e30 con 0x7fb2c419a0e0 2024-04-22T13:40:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2ca25e640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00eaa0 tx=0x7fb2bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.712+0000 7fb2baffd640 1 -- 172.21.15.87:0/3385324233 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00cdb0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.713+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c41a5120 con 0x7fb2c419a0e0 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.713+0000 7fb2baffd640 1 -- 172.21.15.87:0/3385324233 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2bc005020 con 0x7fb2c419a0e0 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.713+0000 7fb2baffd640 1 -- 172.21.15.87:0/3385324233 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0116d0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.713+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c41a55f0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.718 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:32 smithi132 ceph-mon[31531]: pgmap v270: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T13:40:32.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2baffd640 1 -- 172.21.15.87:0/3385324233 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb2bc0051d0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 Processor -- start 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 -- start start 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e102a00 0x55ce3e104e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ce3e05e1f0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ce3dffb4b0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2ce994740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ce3e05a1b0 con 0x55ce3e102a00 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.715+0000 7fb2baffd640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 0x7fb2b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2ca25e640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 0x7fb2b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56800/0 (socket says 172.21.15.87:56800) 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2644162963 learned_addr learned my addr 172.21.15.87:0/2644162963 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c925c640 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e102a00 0x55ce3e104e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c925c640 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2baffd640 1 -- 172.21.15.87:0/3385324233 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fb2bc0d09c0 con 0x7fb2c419a0e0 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2644162963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e102a00 msgr2=0x55ce3e104e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e102a00 0x55ce3e104e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2644162963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 msgr2=0x55ce3e0f98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0f98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2644162963 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3dfed1b0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2ca25e640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 0x7fb2b00a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0ce200 tx=0x7fb2bc0bd040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009c90 tx=0x7fb2c00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98ac4da44dbd43c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2affff640 1 -- 172.21.15.87:0/2644162963 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0012450 con 0x55ce3e0f9e80 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2affff640 1 -- 172.21.15.87:0/2644162963 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0012aa0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.716+0000 7fb2affff640 1 -- 172.21.15.87:0/2644162963 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000f9a0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- 172.21.15.87:0/2644162963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 msgr2=0x55ce3e102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009c90 tx=0x7fb2c00117b0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- 172.21.15.87:0/2644162963 shutdown_connections 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e102a00 0x55ce3e104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e102430 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2644162963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0f98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- 172.21.15.87:0/2644162963 >> 172.21.15.87:0/2644162963 conn(0x55ce3e0f3ba0 msgr2=0x55ce3e0f3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- 172.21.15.87:0/2644162963 shutdown_connections 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- 172.21.15.87:0/2644162963 wait complete. 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.717+0000 7fb2ce994740 1 -- start start 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 Processor -- start 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e0ffc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55ce3e105970 con 0x55ce3e0f94d0 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ce3e017500 con 0x55ce3e102a00 2024-04-22T13:40:32.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ce3e105af0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ce994740 10 client.0 _ll_register_callbacks cb 0x55ce3dfe5b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56814/0 (socket says 172.21.15.87:56814) 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 -- 172.21.15.87:0/3374485881 learned_addr learned my addr 172.21.15.87:0/3374485881 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e0ffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 -- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 msgr2=0x55ce3e0ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 -- 172.21.15.87:0/3374485881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 msgr2=0x55ce3e0ffc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e0ffc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 -- 172.21.15.87:0/3374485881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3dd703f0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40096c0 tx=0x7fb2b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3cdb26ffbc84e1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.718+0000 7fb2ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4005030 con 0x55ce3e0f94d0 2024-04-22T13:40:32.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fb2b40051d0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c0012aa0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb2b40053d0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 10 client.18120 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ce3de3a7d0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0012aa0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb2b4040040 con 0x55ce3e0f94d0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 10 client.18120 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 10 client.18120 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 20 client.18120 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e112b60 0x55ce3e114f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 4 client.18120 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce3dffd930 con 0x55ce3e112b60 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.719+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e112b60 0x55ce3e114f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.720+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e112b60 0x55ce3e114f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d6ada410e94b664 server_cookie=33d28a22b78c8f93 in_seq=0 out_seq=0 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.720+0000 7fb2ad7fa640 10 client.18120 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:32.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.720+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3374485881 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce3dffd930 con 0x55ce3e112b60 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.720+0000 7fb2ad7fa640 10 client.18120 handle_command_reply: tid=0 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.720+0000 7fb2ce994740 1 client.18120 shutdown 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 msgr2=0x55ce3e102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40096c0 tx=0x7fb2b4009260 comp rx=0 tx=0).stop 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 shutdown_connections 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e112b60 0x55ce3e114f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e0ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3374485881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f94d0 0x55ce3e102320 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ad7fa640 0 client.18120 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 >> 172.21.15.87:0/3374485881 conn(0x55ce3e0f3ba0 msgr2=0x55ce3e0f58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 shutdown_connections 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 1 -- 172.21.15.87:0/3374485881 wait complete. 2024-04-22T13:40:32.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.721+0000 7fb2ce994740 20 client.18120 trim_cache size 0 max 16384 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 Processor -- start 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 -- start start 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f94d0 0x55ce3e10e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e102320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41b03c0 con 0x55ce3e0f9e80 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41adc90 con 0x55ce3e102a00 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2ce994740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41abb00 con 0x55ce3e0f94d0 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56830/0 (socket says 172.21.15.87:56830) 2024-04-22T13:40:32.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2193688312 learned_addr learned my addr 172.21.15.87:0/2193688312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c925c640 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f94d0 0x55ce3e10e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c925c640 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2193688312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f94d0 msgr2=0x55ce3e10e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f94d0 0x55ce3e10e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2193688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 msgr2=0x55ce3e102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.734+0000 7fb2c9a5d640 1 -- 172.21.15.87:0/2193688312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3e0ffa10 con 0x55ce3e0f9e80 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb2c001c040 tx=0x7fb2c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41d45286add31af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2af7fe640 1 -- 172.21.15.87:0/2193688312 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0032070 con 0x55ce3e0f9e80 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2af7fe640 1 -- 172.21.15.87:0/2193688312 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0013e50 con 0x55ce3e0f9e80 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2af7fe640 1 -- 172.21.15.87:0/2193688312 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c002c520 con 0x55ce3e0f9e80 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 -- 172.21.15.87:0/2193688312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 msgr2=0x55ce3e101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb2c001c040 tx=0x7fb2c0005f90 comp rx=0 tx=0).stop 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 -- 172.21.15.87:0/2193688312 shutdown_connections 2024-04-22T13:40:32.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e102a00 0x55ce3e102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e0f9e80 0x55ce3e101de0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 --2- 172.21.15.87:0/2193688312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f94d0 0x55ce3e10e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.735+0000 7fb2ce994740 1 -- 172.21.15.87:0/2193688312 >> 172.21.15.87:0/2193688312 conn(0x55ce3e0f3ba0 msgr2=0x55ce3e0f5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- 172.21.15.87:0/2193688312 shutdown_connections 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- 172.21.15.87:0/2193688312 wait complete. 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- start start 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 Processor -- start 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0fc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41ac150 con 0x55ce3e102a00 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41ad5c0 con 0x55ce3e0f94d0 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c41ac820 con 0x55ce3e0f9e80 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2ce994740 10 client.0 _ll_register_callbacks cb 0x7fb2b40040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2c925c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2c925c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56834/0 (socket says 172.21.15.87:56834) 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2c925c640 1 -- 172.21.15.87:0/3210490897 learned_addr learned my addr 172.21.15.87:0/3210490897 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.736+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 -- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 msgr2=0x55ce3e0fc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0fc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 -- 172.21.15.87:0/3210490897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 msgr2=0x55ce3e0fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 -- 172.21.15.87:0/3210490897 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3e178160 con 0x55ce3e102a00 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2c925c640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40041e0 tx=0x7fb2b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3273a063c1f103 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4003a90 con 0x55ce3e102a00 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fb2b4043660 con 0x55ce3e102a00 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c0013e50 con 0x55ce3e102a00 2024-04-22T13:40:32.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb2b4043860 con 0x55ce3e102a00 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 10 client.18126 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ce3e179440 con 0x55ce3e102a00 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0013e50 con 0x55ce3e102a00 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb2b4055a90 con 0x55ce3e102a00 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 10 client.18126 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 10 client.18126 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 20 client.18126 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e179ad0 0x55ce3e17bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 4 client.18126 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.737+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce3e06bc10 con 0x55ce3e179ad0 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e179ad0 0x55ce3e17bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2c9a5d640 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e179ad0 0x55ce3e17bef0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f72dff507cf5ce73 server_cookie=2e29dc8ec3f75f57 in_seq=0 out_seq=0 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ad7fa640 10 client.18126 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ad7fa640 1 -- 172.21.15.87:0/3210490897 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce3e06bc10 con 0x55ce3e179ad0 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ad7fa640 10 client.18126 handle_command_reply: tid=0 2024-04-22T13:40:32.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 client.18126 shutdown 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 msgr2=0x55ce3e0fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40041e0 tx=0x7fb2b4009260 comp rx=0 tx=0).stop 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 shutdown_connections 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55ce3e179ad0 0x55ce3e17bef0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55ce3e102a00 0x55ce3e0fcea0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce3e0f9e80 0x55ce3e0fc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ad7fa640 0 client.18126 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 --2- 172.21.15.87:0/3210490897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce3e0f94d0 0x55ce3e0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 >> 172.21.15.87:0/3210490897 conn(0x55ce3e0f3ba0 msgr2=0x55ce3e0f4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.738+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 shutdown_connections 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2ce994740 1 -- 172.21.15.87:0/3210490897 wait complete. 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2ce994740 20 client.18126 trim_cache size 0 max 16384 2024-04-22T13:40:32.741 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:32.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 msgr2=0x7fb2b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 0x7fb2b00a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0ce200 tx=0x7fb2bc0bd040 comp rx=0 tx=0).stop 2024-04-22T13:40:32.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 msgr2=0x7fb2c41a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00eaa0 tx=0x7fb2bc00ef60 comp rx=0 tx=0).stop 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 shutdown_connections 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b00a3d50 0x7fb2b00a6210 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c419a0e0 0x7fb2c41a46e0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c4106160 0x7fb2c4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 --2- 172.21.15.87:0/3385324233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c4101090 0x7fb2c4199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 >> 172.21.15.87:0/3385324233 conn(0x7fb2c40771d0 msgr2=0x7fb2c40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 shutdown_connections 2024-04-22T13:40:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.739+0000 7fb2cbcc1640 1 -- 172.21.15.87:0/3385324233 wait complete. 2024-04-22T13:40:32.746 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 -- 172.21.15.87:0/849179108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164101090 msgr2=0x7f4164101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 --2- 172.21.15.87:0/849179108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164101090 0x7f4164101470 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f415c009ca0 tx=0x7f415c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 -- 172.21.15.87:0/849179108 shutdown_connections 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 --2- 172.21.15.87:0/849179108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164106160 0x7f416410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 --2- 172.21.15.87:0/849179108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4164101a40 0x7f4164105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 --2- 172.21.15.87:0/849179108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164101090 0x7f4164101470 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.849+0000 7f416bc74640 1 -- 172.21.15.87:0/849179108 >> 172.21.15.87:0/849179108 conn(0x7f4164077f50 msgr2=0x7f4164078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- 172.21.15.87:0/849179108 shutdown_connections 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- 172.21.15.87:0/849179108 wait complete. 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 Processor -- start 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- start start 2024-04-22T13:40:32.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 0x7f4164198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416419d010 0x7f416419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f41641121d0 con 0x7f4164106160 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4164112050 con 0x7f416419d010 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416bc74640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4164112350 con 0x7f4164101090 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416920f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416920f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56840/0 (socket says 172.21.15.87:56840) 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.850+0000 7f416920f640 1 -- 172.21.15.87:0/2555446227 learned_addr learned my addr 172.21.15.87:0/2555446227 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416a211640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416419d010 0x7f416419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f4169a10640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 0x7f4164198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 -- 172.21.15.87:0/2555446227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 msgr2=0x7f4164198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 0x7f4164198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 -- 172.21.15.87:0/2555446227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416419d010 msgr2=0x7f416419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416419d010 0x7f416419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 -- 172.21.15.87:0/2555446227 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416419db00 con 0x7f4164106160 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f4169a10640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 0x7f4164198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416920f640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f415800d770 tx=0x7f415800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f4162ffd640 1 -- 172.21.15.87:0/2555446227 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158017070 con 0x7f4164106160 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f4162ffd640 1 -- 172.21.15.87:0/2555446227 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4158009c30 con 0x7f4164106160 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416419ddf0 con 0x7f4164106160 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f4162ffd640 1 -- 172.21.15.87:0/2555446227 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415800f620 con 0x7f4164106160 2024-04-22T13:40:32.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.851+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41641a55e0 con 0x7f4164106160 2024-04-22T13:40:32.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.853+0000 7f4162ffd640 1 -- 172.21.15.87:0/2555446227 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f415801a070 con 0x7f4164106160 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 Processor -- start 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 -- start start 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 0x55c630428790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630428d60 0x55c630431890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c63038d1f0 con 0x55c630431e60 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c63032a4b0 con 0x55c6304283b0 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416e86f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c6303891b0 con 0x55c630428d60 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f4162ffd640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 0x7f414c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416920f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416a211640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 0x7f414c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416920f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56854/0 (socket says 172.21.15.87:56854) 2024-04-22T13:40:32.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416920f640 1 -- 172.21.15.87:0/640359136 learned_addr learned my addr 172.21.15.87:0/640359136 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f416a211640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 0x55c630428790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f4169a10640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630428d60 0x55c630431890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.854+0000 7f4162ffd640 1 -- 172.21.15.87:0/2555446227 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f41580d0330 con 0x7f4164106160 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 -- 172.21.15.87:0/640359136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630428d60 msgr2=0x55c630431890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630428d60 0x55c630431890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 -- 172.21.15.87:0/640359136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 msgr2=0x55c630428790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 0x55c630428790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 -- 172.21.15.87:0/640359136 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c63031c1b0 con 0x55c630431e60 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416a211640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 0x55c630428790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416a211640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 0x7f414c0a61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f415400bfd0 tx=0x7f415400b7c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f416920f640 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f415809c2a0 tx=0x7f415806ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43549587bf35a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f414bfff640 1 -- 172.21.15.87:0/640359136 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158021070 con 0x55c630431e60 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f414bfff640 1 -- 172.21.15.87:0/640359136 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4158009de0 con 0x55c630431e60 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.855+0000 7f414bfff640 1 -- 172.21.15.87:0/640359136 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158070b80 con 0x55c630431e60 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- 172.21.15.87:0/640359136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 msgr2=0x55c6304322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f415809c2a0 tx=0x7f415806ee00 comp rx=0 tx=0).stop 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- 172.21.15.87:0/640359136 shutdown_connections 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c6304322c0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630428d60 0x55c630431890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 --2- 172.21.15.87:0/640359136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c6304283b0 0x55c630428790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- 172.21.15.87:0/640359136 >> 172.21.15.87:0/640359136 conn(0x55c630422ba0 msgr2=0x55c630422f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- 172.21.15.87:0/640359136 shutdown_connections 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- 172.21.15.87:0/640359136 wait complete. 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 -- start start 2024-04-22T13:40:32.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.856+0000 7f416e86f740 1 Processor -- start 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630435560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c630435aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c630438d40 con 0x55c6304283b0 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c630346500 con 0x55c630428d60 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c630438ec0 con 0x55c630431e60 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 10 client.0 _ll_register_callbacks cb 0x55c630314b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56860/0 (socket says 172.21.15.87:56860) 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 -- 172.21.15.87:0/3546410503 learned_addr learned my addr 172.21.15.87:0/3546410503 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416a211640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c630435aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416920f640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630435560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 -- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 msgr2=0x55c630435aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c630435aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 -- 172.21.15.87:0/3546410503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 msgr2=0x55c630435560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630435560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 -- 172.21.15.87:0/3546410503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c63009f3f0 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f4169a10640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f415c009ca0 tx=0x7f415c0326f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d559490adc332983 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f41497fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c045070 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4158009de0 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f415c003970 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f415c051050 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 10 client.18138 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.857+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c6301697d0 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4158009de0 con 0x55c6304283b0 2024-04-22T13:40:32.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f415c034440 con 0x55c6304283b0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 10 client.18138 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 10 client.18138 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 20 client.18138 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c630445ee0 0x55c630448370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 4 client.18138 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c63032c930 con 0x55c630445ee0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.858+0000 7f416920f640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c630445ee0 0x55c630448370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f416920f640 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c630445ee0 0x55c630448370 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d545b31450e278d server_cookie=192c3b7f693bccd4 in_seq=0 out_seq=0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f41497fa640 10 client.18138 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f41497fa640 1 -- 172.21.15.87:0/3546410503 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c63032c930 con 0x55c630445ee0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f41497fa640 10 client.18138 handle_command_reply: tid=0 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f416e86f740 1 client.18138 shutdown 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 msgr2=0x55c6304381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.859+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f415c009ca0 tx=0x7f415c0326f0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 shutdown_connections 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c630445ee0 0x55c630448370 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c630435aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630435560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f41497fa640 0 client.18138 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 --2- 172.21.15.87:0/3546410503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c6304381d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 >> 172.21.15.87:0/3546410503 conn(0x55c630422ba0 msgr2=0x55c6304248f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 shutdown_connections 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 1 -- 172.21.15.87:0/3546410503 wait complete. 2024-04-22T13:40:32.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.860+0000 7f416e86f740 20 client.18138 trim_cache size 0 max 16384 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 Processor -- start 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 -- start start 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c6304381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f41641ae180 con 0x55c6304283b0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41641aba50 con 0x55c630428d60 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f416e86f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41641a98e0 con 0x55c630431e60 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56864/0 (socket says 172.21.15.87:56864) 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 -- 172.21.15.87:0/1553372398 learned_addr learned my addr 172.21.15.87:0/1553372398 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 -- 172.21.15.87:0/1553372398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 msgr2=0x55c6304381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c6304381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 -- 172.21.15.87:0/1553372398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 msgr2=0x55c630437c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 -- 172.21.15.87:0/1553372398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c630434570 con 0x55c6304283b0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f4169a10640 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f415c009bf0 tx=0x7f415c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0fa951d62d4bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.873+0000 7f414b7fe640 1 -- 172.21.15.87:0/1553372398 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c04e030 con 0x55c6304283b0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f414b7fe640 1 -- 172.21.15.87:0/1553372398 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c03a040 con 0x55c6304283b0 2024-04-22T13:40:32.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f414b7fe640 1 -- 172.21.15.87:0/1553372398 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c045040 con 0x55c6304283b0 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- 172.21.15.87:0/1553372398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 msgr2=0x55c630441c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f415c009bf0 tx=0x7f415c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- 172.21.15.87:0/1553372398 shutdown_connections 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c630431e60 0x55c6304381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 --2- 172.21.15.87:0/1553372398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c6304283b0 0x55c630441c40 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- 172.21.15.87:0/1553372398 >> 172.21.15.87:0/1553372398 conn(0x55c630422ba0 msgr2=0x55c630431610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- 172.21.15.87:0/1553372398 shutdown_connections 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- 172.21.15.87:0/1553372398 wait complete. 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 -- start start 2024-04-22T13:40:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.874+0000 7f416e86f740 1 Processor -- start 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6304283b0 0x55c630437130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f41641a9f30 con 0x55c630431e60 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41641ab380 con 0x55c630428d60 2024-04-22T13:40:32.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41641aa5e0 con 0x55c6304283b0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416e86f740 10 client.0 _ll_register_callbacks cb 0x55c630314b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56878/0 (socket says 172.21.15.87:56878) 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 -- 172.21.15.87:0/1505787358 learned_addr learned my addr 172.21.15.87:0/1505787358 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f4169a10640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6304283b0 0x55c630437130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416920f640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 -- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6304283b0 msgr2=0x55c630437130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6304283b0 0x55c630437130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 -- 172.21.15.87:0/1505787358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 msgr2=0x55c630437670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 -- 172.21.15.87:0/1505787358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6304ab4b0 con 0x55c630431e60 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f416a211640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f4154010660 tx=0x7f415400b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702ad8dfd95f0c0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.875+0000 7f41497fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:32.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154005020 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f41540051c0 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f415c03a040 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f41540053c0 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 10 client.18144 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c6304ac750 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c03a040 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f415401f8a0 con 0x55c630431e60 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 10 client.18144 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 10 client.18144 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 20 client.18144 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c6304acf00 0x55c6304af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 4 client.18144 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c63039ac10 con 0x55c6304acf00 2024-04-22T13:40:32.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.876+0000 7f4169a10640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c6304acf00 0x55c6304af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f41497fa640 10 client.18144 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f4169a10640 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c6304acf00 0x55c6304af320 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85e5628a95bdb06c server_cookie=57e7fd9f1fd90dba in_seq=0 out_seq=0 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f41497fa640 1 -- 172.21.15.87:0/1505787358 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c63039ac10 con 0x55c6304acf00 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f41497fa640 10 client.18144 handle_command_reply: tid=0 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 client.18144 shutdown 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 msgr2=0x55c630437bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f4154010660 tx=0x7f415400b2e0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 shutdown_connections 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c6304acf00 0x55c6304af320 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c630431e60 0x55c630437bb0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c630428d60 0x55c630437670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f41497fa640 0 client.18144 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 --2- 172.21.15.87:0/1505787358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6304283b0 0x55c630437130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 >> 172.21.15.87:0/1505787358 conn(0x55c630422ba0 msgr2=0x55c63042f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 shutdown_connections 2024-04-22T13:40:32.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 1 -- 172.21.15.87:0/1505787358 wait complete. 2024-04-22T13:40:32.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.877+0000 7f416e86f740 20 client.18144 trim_cache size 0 max 16384 2024-04-22T13:40:32.880 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 msgr2=0x7f414c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 0x7f414c0a61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f415400bfd0 tx=0x7f415400b7c0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 msgr2=0x7f4164198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f415800d770 tx=0x7f415800dc30 comp rx=0 tx=0).stop 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 shutdown_connections 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f414c0a3d20 0x7f414c0a61e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416419d010 0x7f416419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4164106160 0x7f4164198c90 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 --2- 172.21.15.87:0/2555446227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4164101090 0x7f4164198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 >> 172.21.15.87:0/2555446227 conn(0x7f4164077f50 msgr2=0x7f41640ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 shutdown_connections 2024-04-22T13:40:32.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.878+0000 7f416bc74640 1 -- 172.21.15.87:0/2555446227 wait complete. 2024-04-22T13:40:32.884 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 -- 172.21.15.87:0/732327242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1072b0 msgr2=0x7f45bc10eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/732327242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1072b0 0x7f45bc10eec0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009ca0 tx=0x7f45b402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 -- 172.21.15.87:0/732327242 shutdown_connections 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/732327242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1072b0 0x7f45bc10eec0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/732327242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc1014a0 0x7f45bc106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/732327242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc100af0 0x7f45bc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 -- 172.21.15.87:0/732327242 >> 172.21.15.87:0/732327242 conn(0x7f45bc0760e0 msgr2=0x7f45bc0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 -- 172.21.15.87:0/732327242 shutdown_connections 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 -- 172.21.15.87:0/732327242 wait complete. 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.989+0000 7f45c3a4e640 1 Processor -- start 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 -- start start 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 0x7f45bc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 0x7f45bc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc112250 con 0x7f45bc1985d0 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1120d0 con 0x7f45bc100af0 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1123d0 con 0x7f45bc1072b0 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56890/0 (socket says 172.21.15.87:56890) 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 -- 172.21.15.87:0/1592516952 learned_addr learned my addr 172.21.15.87:0/1592516952 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 0x7f45bc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 0x7f45bc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 -- 172.21.15.87:0/1592516952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 msgr2=0x7f45bc19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 0x7f45bc19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 -- 172.21.15.87:0/1592516952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 msgr2=0x7f45bc19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 0x7f45bc19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 -- 172.21.15.87:0/1592516952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45bc1990f0 con 0x7f45bc1985d0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 0x7f45bc19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 0x7f45bc19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c1feb640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005e30 tx=0x7f45b4034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45b27fc640 1 -- 172.21.15.87:0/1592516952 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4039040 con 0x7f45bc1985d0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.990+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45bc199380 con 0x7f45bc1985d0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.991+0000 7f45b27fc640 1 -- 172.21.15.87:0/1592516952 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b4044070 con 0x7f45bc1985d0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.991+0000 7f45b27fc640 1 -- 172.21.15.87:0/1592516952 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4003e60 con 0x7f45bc1985d0 2024-04-22T13:40:32.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.991+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45bc1a55d0 con 0x7f45bc1985d0 2024-04-22T13:40:32.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45b27fc640 1 -- 172.21.15.87:0/1592516952 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f45b40324c0 con 0x7f45bc1985d0 2024-04-22T13:40:32.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 Processor -- start 2024-04-22T13:40:32.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 -- start start 2024-04-22T13:40:32.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9e4d0 0x5573d1e9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1ea7a00 0x5573d1ea9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5573d1e031f0 con 0x5573d1e9ee80 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573d1da04b0 con 0x5573d1ea7a00 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c6644740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5573d1dff1b0 con 0x5573d1e9e4d0 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45b27fc640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 0x7f45a80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.993+0000 7f45c17ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56900/0 (socket says 172.21.15.87:56900) 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 -- 172.21.15.87:0/2187847492 learned_addr learned my addr 172.21.15.87:0/2187847492 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c1feb640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 0x7f45a80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45b27fc640 1 -- 172.21.15.87:0/1592516952 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f45b40f2a40 con 0x7f45bc1985d0 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9e4d0 0x5573d1e9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1ea7a00 0x5573d1ea9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 -- 172.21.15.87:0/2187847492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9e4d0 msgr2=0x5573d1e9e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9e4d0 0x5573d1e9e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 -- 172.21.15.87:0/2187847492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1ea7a00 msgr2=0x5573d1ea9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1ea7a00 0x5573d1ea9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 -- 172.21.15.87:0/2187847492 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d1d921b0 con 0x5573d1e9ee80 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c1feb640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 0x7f45a80a61e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f45b40f0390 tx=0x7f45b402c0f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f45b800bd40 tx=0x7f45b800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447044e63bb6e948 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45a77fe640 1 -- 172.21.15.87:0/2187847492 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8015070 con 0x5573d1e9ee80 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45a77fe640 1 -- 172.21.15.87:0/2187847492 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b800be30 con 0x5573d1e9ee80 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.994+0000 7f45a77fe640 1 -- 172.21.15.87:0/2187847492 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b80106e0 con 0x5573d1e9ee80 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- 172.21.15.87:0/2187847492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 msgr2=0x5573d1ea7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f45b800bd40 tx=0x7f45b800f730 comp rx=0 tx=0).stop 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- 172.21.15.87:0/2187847492 shutdown_connections 2024-04-22T13:40:32.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1ea7a00 0x5573d1ea9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea7430 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 --2- 172.21.15.87:0/2187847492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9e4d0 0x5573d1e9e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- 172.21.15.87:0/2187847492 >> 172.21.15.87:0/2187847492 conn(0x5573d1e98ba0 msgr2=0x5573d1e98f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- 172.21.15.87:0/2187847492 shutdown_connections 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- 172.21.15.87:0/2187847492 wait complete. 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 -- start start 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.995+0000 7f45c6644740 1 Processor -- start 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1ea7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 0x5573d1ea4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5573d1eaa970 con 0x5573d1e9ee80 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573d1dbc500 con 0x5573d1e9e4d0 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5573d1eaaaf0 con 0x5573d1ea7a00 2024-04-22T13:40:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 10 client.0 _ll_register_callbacks cb 0x5573d1d8ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56902/0 (socket says 172.21.15.87:56902) 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 -- 172.21.15.87:0/1133400790 learned_addr learned my addr 172.21.15.87:0/1133400790 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 0x5573d1ea4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 -- 172.21.15.87:0/1133400790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 msgr2=0x5573d1ea4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 0x5573d1ea4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 -- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 msgr2=0x5573d1ea7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1ea7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 -- 172.21.15.87:0/1133400790 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d1b153f0 con 0x5573d1e9ee80 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 0x5573d1ea4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f45b8019000 tx=0x7f45b8005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bce6aac2a428022 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:33.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8029060 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f45b8005480 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.996+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45b800be30 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f45b8015050 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 10 client.18156 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5573d1bdf7d0 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b800be30 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f45b8004ad0 con 0x5573d1e9ee80 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 10 client.18156 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 10 client.18156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 20 client.18156 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1eb7bc0 0x5573d1eb9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 4 client.18156 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573d1da2930 con 0x5573d1eb7bc0 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.997+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1eb7bc0 0x5573d1eb9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1eb7bc0 0x5573d1eb9fb0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533ae51932436032 server_cookie=8b7b95678f2f8109 in_seq=0 out_seq=0 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45a4ff9640 10 client.18156 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45a4ff9640 1 -- 172.21.15.87:0/1133400790 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573d1da2930 con 0x5573d1eb7bc0 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45a4ff9640 10 client.18156 handle_command_reply: tid=0 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 client.18156 shutdown 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 msgr2=0x5573d1ea46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f45b8019000 tx=0x7f45b8005040 comp rx=0 tx=0).stop 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 shutdown_connections 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1eb7bc0 0x5573d1eb9fb0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1ea7a00 0x5573d1ea4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1e9ee80 0x5573d1ea46e0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45a4ff9640 0 client.18156 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 --2- 172.21.15.87:0/1133400790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1ea7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.998+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 >> 172.21.15.87:0/1133400790 conn(0x5573d1e98ba0 msgr2=0x5573d1e9a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:33.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.999+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 shutdown_connections 2024-04-22T13:40:33.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.999+0000 7f45c6644740 1 -- 172.21.15.87:0/1133400790 wait complete. 2024-04-22T13:40:33.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:32.999+0000 7f45c6644740 20 client.18156 trim_cache size 0 max 16384 2024-04-22T13:40:33.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 Processor -- start 2024-04-22T13:40:33.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 -- start start 2024-04-22T13:40:33.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1eb3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1ea6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1b0470 con 0x5573d1ea7a00 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1add40 con 0x5573d1e9e4d0 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c6644740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1abbb0 con 0x5573d1e9ee80 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56914/0 (socket says 172.21.15.87:56914) 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 -- 172.21.15.87:0/1273123990 learned_addr learned my addr 172.21.15.87:0/1273123990 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1eb3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1ea6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 -- 172.21.15.87:0/1273123990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 msgr2=0x5573d1ea6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1ea6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 -- 172.21.15.87:0/1273123990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 msgr2=0x5573d1eb3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1eb3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 -- 172.21.15.87:0/1273123990 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573d1e98380 con 0x5573d1ea7a00 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c17ea640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1ea6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.012+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00ef60 tx=0x7f45ac00e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f7523840920882 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45a6ffd640 1 -- 172.21.15.87:0/1273123990 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac004230 con 0x5573d1ea7a00 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45a6ffd640 1 -- 172.21.15.87:0/1273123990 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac0043d0 con 0x5573d1ea7a00 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45a6ffd640 1 -- 172.21.15.87:0/1273123990 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac01d860 con 0x5573d1ea7a00 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- 172.21.15.87:0/1273123990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 msgr2=0x5573d1ea7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00ef60 tx=0x7f45ac00e980 comp rx=0 tx=0).stop 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- 172.21.15.87:0/1273123990 shutdown_connections 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1ea7350 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1ea6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 --2- 172.21.15.87:0/1273123990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1eb3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- 172.21.15.87:0/1273123990 >> 172.21.15.87:0/1273123990 conn(0x5573d1e98ba0 msgr2=0x5573d1eb2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- 172.21.15.87:0/1273123990 shutdown_connections 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- 172.21.15.87:0/1273123990 wait complete. 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.013+0000 7f45c6644740 1 -- start start 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 Processor -- start 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1f15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1f156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1ac200 con 0x5573d1ea7a00 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1ad670 con 0x5573d1e9e4d0 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1ac8d0 con 0x5573d1e9ee80 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c6644740 10 client.0 _ll_register_callbacks cb 0x7f45b8011d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56928/0 (socket says 172.21.15.87:56928) 2024-04-22T13:40:33.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 -- 172.21.15.87:0/2843475429 learned_addr learned my addr 172.21.15.87:0/2843475429 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1f156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1f15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 -- 172.21.15.87:0/2843475429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 msgr2=0x5573d1f156a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1f156a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 -- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 msgr2=0x5573d1f15160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1f15160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 -- 172.21.15.87:0/2843475429 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573d1ebaa90 con 0x5573d1ea7a00 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1f156a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45c0fe9640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005bc0 tx=0x7f45ac00e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8f8c5d8254c533 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.014+0000 7f45a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac01dba0 con 0x5573d1ea7a00 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f45ac01dd40 con 0x5573d1ea7a00 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45ac0043d0 con 0x5573d1ea7a00 2024-04-22T13:40:33.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f45ac02c440 con 0x5573d1ea7a00 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 10 client.18162 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5573d1f1d880 con 0x5573d1ea7a00 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac0043d0 con 0x5573d1ea7a00 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f45ac02a550 con 0x5573d1ea7a00 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 10 client.18162 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 20 client.18162 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1f1dee0 0x5573d1f20300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 4 client.18162 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.015+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573d1e10c10 con 0x5573d1f1dee0 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1f1dee0 0x5573d1f20300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c17ea640 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1f1dee0 0x5573d1f20300 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79c1b8ffd3ac3f23 server_cookie=7c5c31fb203bed02 in_seq=0 out_seq=0 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45a4ff9640 10 client.18162 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45a4ff9640 1 -- 172.21.15.87:0/2843475429 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573d1e10c10 con 0x5573d1f1dee0 2024-04-22T13:40:33.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45a4ff9640 10 client.18162 handle_command_reply: tid=0 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 client.18162 shutdown 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 msgr2=0x5573d1eba2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005bc0 tx=0x7f45ac00e8e0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 shutdown_connections 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5573d1f1dee0 0x5573d1f20300 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5573d1ea7a00 0x5573d1eba2b0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573d1e9ee80 0x5573d1f156a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 --2- 172.21.15.87:0/2843475429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573d1e9e4d0 0x5573d1f15160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 >> 172.21.15.87:0/2843475429 conn(0x5573d1e98ba0 msgr2=0x5573d1eb3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45a4ff9640 0 client.18162 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 shutdown_connections 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 1 -- 172.21.15.87:0/2843475429 wait complete. 2024-04-22T13:40:33.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.016+0000 7f45c6644740 20 client.18162 trim_cache size 0 max 16384 2024-04-22T13:40:33.022 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 msgr2=0x7f45a80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 0x7f45a80a61e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f45b40f0390 tx=0x7f45b402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 msgr2=0x7f45bc198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005e30 tx=0x7f45b4034760 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 shutdown_connections 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45a80a3d20 0x7f45a80a61e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45bc1985d0 0x7f45bc198a80 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45bc1072b0 0x7f45bc19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 --2- 172.21.15.87:0/1592516952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45bc100af0 0x7f45bc19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 >> 172.21.15.87:0/1592516952 conn(0x7f45bc0760e0 msgr2=0x7f45bc0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 shutdown_connections 2024-04-22T13:40:33.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:33.017+0000 7f45c3a4e640 1 -- 172.21.15.87:0/1592516952 wait complete. 2024-04-22T13:40:33.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:33 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3396188732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:33.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:33 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3396188732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:33.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:33 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3396188732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:40:34.025 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:40:34.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:40:34.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:40:34.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3819253368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c601010a0 msgr2=0x7f8c60101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3819253368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c601010a0 0x7f8c60101480 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009900 tx=0x7f8c5002bc30 comp rx=0 tx=0).stop 2024-04-22T13:40:34.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3819253368 shutdown_connections 2024-04-22T13:40:34.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3819253368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c60106170 0x7f8c6010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3819253368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60101a50 0x7f8c60105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3819253368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c601010a0 0x7f8c60101480 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3819253368 >> 172.21.15.87:0/3819253368 conn(0x7f8c60077eb0 msgr2=0x7f8c600782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3819253368 shutdown_connections 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.145+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3819253368 wait complete. 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 Processor -- start 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 -- start start 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601010a0 0x7f8c6019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 0x7f8c60198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60111130 con 0x7f8c60106170 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60110fb0 con 0x7f8c601985e0 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c66ceb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601112b0 con 0x7f8c601010a0 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56936/0 (socket says 172.21.15.87:56936) 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 -- 172.21.15.87:0/3398222485 learned_addr learned my addr 172.21.15.87:0/3398222485 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c64a87640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601010a0 0x7f8c6019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c65288640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 0x7f8c60198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 -- 172.21.15.87:0/3398222485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601010a0 msgr2=0x7f8c6019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601010a0 0x7f8c6019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 -- 172.21.15.87:0/3398222485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 msgr2=0x7f8c60198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 0x7f8c60198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 -- 172.21.15.87:0/3398222485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c60199100 con 0x7f8c60106170 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c65288640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 0x7f8c60198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.146+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400b450 tx=0x7f8c5400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.147+0000 7f8c5dffb640 1 -- 172.21.15.87:0/3398222485 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54015020 con 0x7f8c60106170 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.147+0000 7f8c5dffb640 1 -- 172.21.15.87:0/3398222485 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c54011d40 con 0x7f8c60106170 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.147+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c601993f0 con 0x7f8c60106170 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.147+0000 7f8c5dffb640 1 -- 172.21.15.87:0/3398222485 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400f340 con 0x7f8c60106170 2024-04-22T13:40:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.147+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c601a5640 con 0x7f8c60106170 2024-04-22T13:40:34.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.149+0000 7f8c5dffb640 1 -- 172.21.15.87:0/3398222485 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8c54018020 con 0x7f8c60106170 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 Processor -- start 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 -- start start 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d416b4ff00 con 0x55d416bf4e10 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d416aed590 con 0x55d416beb8e0 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c699b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d416b4bec0 con 0x55d416bec290 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c5dffb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 0x7f8c480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c65288640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c5ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56948/0 (socket says 172.21.15.87:56948) 2024-04-22T13:40:34.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c5ffff640 1 -- 172.21.15.87:0/323643522 learned_addr learned my addr 172.21.15.87:0/323643522 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c65288640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 0x7f8c480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.150+0000 7f8c64a87640 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5dffb640 1 -- 172.21.15.87:0/3398222485 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f8c540d0200 con 0x7f8c60106170 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 -- 172.21.15.87:0/323643522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 msgr2=0x55d416bf4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 -- 172.21.15.87:0/323643522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 msgr2=0x55d416bebcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bebcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 -- 172.21.15.87:0/323643522 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d41692a7e0 con 0x55d416bf4e10 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c65288640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 0x7f8c480a61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009fd0 tx=0x7f8c58009970 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009fd0 tx=0x7f8c540108a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcaab3d5a361f0b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c46ffd640 1 -- 172.21.15.87:0/323643522 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54096e60 con 0x55d416bf4e10 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c46ffd640 1 -- 172.21.15.87:0/323643522 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c5409c9f0 con 0x55d416bf4e10 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.151+0000 7f8c46ffd640 1 -- 172.21.15.87:0/323643522 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54010500 con 0x55d416bf4e10 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- 172.21.15.87:0/323643522 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 msgr2=0x55d416bf7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009fd0 tx=0x7f8c540108a0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- 172.21.15.87:0/323643522 shutdown_connections 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf7260 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 --2- 172.21.15.87:0/323643522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bebcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- 172.21.15.87:0/323643522 >> 172.21.15.87:0/323643522 conn(0x55d416be5fb0 msgr2=0x55d416be6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- 172.21.15.87:0/323643522 shutdown_connections 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- 172.21.15.87:0/323643522 wait complete. 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 -- start start 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.152+0000 7f8c699b9740 1 Processor -- start 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416bec290 0x55d416bf23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 0x55d416bf28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d416bf7d20 con 0x55d416beb8e0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d416b09580 con 0x55d416bec290 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d416bf7ea0 con 0x55d416bf4e10 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 10 client.0 _ll_register_callbacks cb 0x55d416b0f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56962/0 (socket says 172.21.15.87:56962) 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 -- 172.21.15.87:0/1706416797 learned_addr learned my addr 172.21.15.87:0/1706416797 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416bec290 0x55d416bf23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c65288640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 0x55d416bf28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 -- 172.21.15.87:0/1706416797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 msgr2=0x55d416bf28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 0x55d416bf28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 -- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416bec290 msgr2=0x55d416bf23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416bec290 0x55d416bf23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 -- 172.21.15.87:0/1706416797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d41698af70 con 0x55d416beb8e0 2024-04-22T13:40:34.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c65288640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 0x55d416bf28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c64a87640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8c50000c00 tx=0x7f8c5002ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bb7f18983eaeca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c50005d50 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f8c50005230 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c5409c9f0 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8c5002c5d0 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 10 client.18174 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.153+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d416822170 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c5409c9f0 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8c50003680 con 0x55d416beb8e0 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 10 client.18174 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 20 client.18174 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:34.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c05310 0x55d416c07760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 4 client.18174 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d416b5d920 con 0x55d416c05310 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.154+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c05310 0x55d416c07760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c05310 0x55d416c07760 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df829972706f9e0 server_cookie=f646a183d5150ae9 in_seq=0 out_seq=0 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c3ffff640 10 client.18174 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c3ffff640 1 -- 172.21.15.87:0/1706416797 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d416b5d920 con 0x55d416c05310 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c3ffff640 10 client.18174 handle_command_reply: tid=0 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 client.18174 shutdown 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 msgr2=0x55d416bf1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8c50000c00 tx=0x7f8c5002ca70 comp rx=0 tx=0).stop 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 shutdown_connections 2024-04-22T13:40:34.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c05310 0x55d416c07760 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bf4e10 0x55d416bf28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c3ffff640 0 client.18174 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416bec290 0x55d416bf23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 --2- 172.21.15.87:0/1706416797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416beb8e0 0x55d416bf1e60 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.155+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 >> 172.21.15.87:0/1706416797 conn(0x55d416be5fb0 msgr2=0x55d416be7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.156+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 shutdown_connections 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.156+0000 7f8c699b9740 1 -- 172.21.15.87:0/1706416797 wait complete. 2024-04-22T13:40:34.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.156+0000 7f8c699b9740 20 client.18174 trim_cache size 0 max 16384 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 Processor -- start 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 -- start start 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416c00be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601b0480 con 0x55d416bf4e10 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601add50 con 0x55d416beb8e0 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c699b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601abbc0 con 0x55d416bec290 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56978/0 (socket says 172.21.15.87:56978) 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 -- 172.21.15.87:0/2448450027 learned_addr learned my addr 172.21.15.87:0/2448450027 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416c00be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 -- 172.21.15.87:0/2448450027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 msgr2=0x55d416c00be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416c00be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c64a87640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 -- 172.21.15.87:0/2448450027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 msgr2=0x55d416bf4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 -- 172.21.15.87:0/2448450027 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d416beac00 con 0x55d416bf4e10 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416c00be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.173+0000 7f8c65288640 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009dd0 tx=0x7f8c5801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729460fb0c863754 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c467fc640 1 -- 172.21.15.87:0/2448450027 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58026070 con 0x55d416bf4e10 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c467fc640 1 -- 172.21.15.87:0/2448450027 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c58005020 con 0x55d416bf4e10 2024-04-22T13:40:34.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c467fc640 1 -- 172.21.15.87:0/2448450027 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58020650 con 0x55d416bf4e10 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 -- 172.21.15.87:0/2448450027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 msgr2=0x55d416bf2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009dd0 tx=0x7f8c5801f000 comp rx=0 tx=0).stop 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 -- 172.21.15.87:0/2448450027 shutdown_connections 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416bf2530 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416c00be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2448450027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 -- 172.21.15.87:0/2448450027 >> 172.21.15.87:0/2448450027 conn(0x55d416be5fb0 msgr2=0x55d416bf1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 -- 172.21.15.87:0/2448450027 shutdown_connections 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.174+0000 7f8c699b9740 1 -- 172.21.15.87:0/2448450027 wait complete. 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 -- start start 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 Processor -- start 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601ac210 con 0x55d416bf4e10 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601ad680 con 0x55d416beb8e0 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601ac8e0 con 0x55d416bec290 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c699b9740 10 client.0 _ll_register_callbacks cb 0x7f8c50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56992/0 (socket says 172.21.15.87:56992) 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 -- 172.21.15.87:0/2615880461 learned_addr learned my addr 172.21.15.87:0/2615880461 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c64a87640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 -- 172.21.15.87:0/2615880461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 msgr2=0x55d416bf9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 -- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 msgr2=0x55d416bf9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c65288640 1 -- 172.21.15.87:0/2615880461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d416c6ade0 con 0x55d416bf4e10 2024-04-22T13:40:34.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.175+0000 7f8c5ffff640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c65288640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f8c58011ee0 tx=0x7f8c58011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34d55e74753c510 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58026070 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f8c58014260 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c58005020 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8c58040050 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 10 client.18180 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d416c6c040 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c58005020 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8c58032410 con 0x55d416bf4e10 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 10 client.18180 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 10 client.18180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 20 client.18180 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:34.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c6c730 0x55d416c6eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 4 client.18180 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:40:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.176+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d416b57790 con 0x55d416c6c730 2024-04-22T13:40:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.177+0000 7f8c64a87640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c6c730 0x55d416c6eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.177+0000 7f8c3ffff640 10 client.18180 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.177+0000 7f8c64a87640 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c6c730 0x55d416c6eb50 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d2577eef9708aa server_cookie=b9c82e3ba388dd1a in_seq=0 out_seq=0 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.178+0000 7f8c3ffff640 1 -- 172.21.15.87:0/2615880461 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d416b57790 con 0x55d416c6c730 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.178+0000 7f8c3ffff640 10 client.18180 handle_command_reply: tid=0 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.178+0000 7f8c699b9740 1 client.18180 shutdown 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 msgr2=0x55d416c6a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f8c58011ee0 tx=0x7f8c58011f10 comp rx=0 tx=0).stop 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 shutdown_connections 2024-04-22T13:40:34.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d416c6c730 0x55d416c6eb50 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c3ffff640 0 client.18180 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d416bf4e10 0x55d416c6a8a0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d416bec290 0x55d416bf9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 --2- 172.21.15.87:0/2615880461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d416beb8e0 0x55d416bf9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 >> 172.21.15.87:0/2615880461 conn(0x55d416be5fb0 msgr2=0x55d416be7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 shutdown_connections 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 1 -- 172.21.15.87:0/2615880461 wait complete. 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c699b9740 20 client.18180 trim_cache size 0 max 16384 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:40:34.181 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "6d0b89a4-cb28-479c-aa39-49dbfa4cddb4", 2024-04-22T13:40:34.182 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:40:34.182 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 msgr2=0x7f8c480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 0x7f8c480a61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009fd0 tx=0x7f8c58009970 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 msgr2=0x7f8c6019e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400b450 tx=0x7f8c5400b910 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 shutdown_connections 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8c480a3d20 0x7f8c480a61e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c601985e0 0x7f8c60198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c60106170 0x7f8c6019e970 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.179+0000 7f8c66ceb640 1 --2- 172.21.15.87:0/3398222485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601010a0 0x7f8c6019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.180+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 >> 172.21.15.87:0/3398222485 conn(0x7f8c60077eb0 msgr2=0x7f8c6006a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.180+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 shutdown_connections 2024-04-22T13:40:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.180+0000 7f8c66ceb640 1 -- 172.21.15.87:0/3398222485 wait complete. 2024-04-22T13:40:34.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:40:34.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:40:34.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- 172.21.15.87:0/2542718343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540106160 msgr2=0x7f854010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 --2- 172.21.15.87:0/2542718343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540106160 0x7f854010dd30 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8538009ca0 tx=0x7f853802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- 172.21.15.87:0/2542718343 shutdown_connections 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 --2- 172.21.15.87:0/2542718343 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540106160 0x7f854010dd30 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 --2- 172.21.15.87:0/2542718343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85401021d0 0x7f8540105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 --2- 172.21.15.87:0/2542718343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8540101820 0x7f8540101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- 172.21.15.87:0/2542718343 >> 172.21.15.87:0/2542718343 conn(0x7f8540077360 msgr2=0x7f85400fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- 172.21.15.87:0/2542718343 shutdown_connections 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- 172.21.15.87:0/2542718343 wait complete. 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 Processor -- start 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.292+0000 7f8545959640 1 -- start start 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 0x7f854010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 0x7f85401a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f85401121e0 con 0x7f8540101820 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8540112060 con 0x7f854010f140 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8540112360 con 0x7f8540106160 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57014/0 (socket says 172.21.15.87:57014) 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 -- 172.21.15.87:0/2749002579 learned_addr learned my addr 172.21.15.87:0/2749002579 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853e7fc640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 0x7f854010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 0x7f85401a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 -- 172.21.15.87:0/2749002579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 msgr2=0x7f854010ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 0x7f854010ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 -- 172.21.15.87:0/2749002579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 msgr2=0x7f85401a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 0x7f85401a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 -- 172.21.15.87:0/2749002579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85401a6f30 con 0x7f8540101820 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 0x7f85401a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853e7fc640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 0x7f854010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f853effd640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f853400c920 tx=0x7f853400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8544957640 1 -- 172.21.15.87:0/2749002579 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534007c70 con 0x7f8540101820 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85401a7220 con 0x7f8540101820 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8544957640 1 -- 172.21.15.87:0/2749002579 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8534007e10 con 0x7f8540101820 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8544957640 1 -- 172.21.15.87:0/2749002579 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534018820 con 0x7f8540101820 2024-04-22T13:40:34.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.293+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85401a7770 con 0x7f8540101820 2024-04-22T13:40:34.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8544957640 1 -- 172.21.15.87:0/2749002579 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f853400f450 con 0x7f8540101820 2024-04-22T13:40:34.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 Processor -- start 2024-04-22T13:40:34.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 -- start start 2024-04-22T13:40:34.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 0x55685554a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55685554aa80 0x55685554ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5568554a61e0 con 0x556855541580 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556855443390 con 0x55685554aa80 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8548620740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5568554a21a0 con 0x556855541ea0 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8544957640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 0x7f85280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57028/0 (socket says 172.21.15.87:57028) 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 -- 172.21.15.87:0/870767930 learned_addr learned my addr 172.21.15.87:0/870767930 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f8544957640 1 -- 172.21.15.87:0/2749002579 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f85340d0990 con 0x7f8540101820 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 0x7f85280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853e7fc640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55685554aa80 0x55685554ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853effd640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 0x55685554a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 -- 172.21.15.87:0/870767930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 msgr2=0x55685554a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 0x55685554a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.296+0000 7f853f7fe640 1 -- 172.21.15.87:0/870767930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55685554aa80 msgr2=0x55685554ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f853f7fe640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55685554aa80 0x55685554ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f853f7fe640 1 -- 172.21.15.87:0/870767930 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568552827d0 con 0x556855541580 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f853effd640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 0x55685554a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f853f7fe640 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f8538009430 tx=0x7f8538032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa49c7eefaa5d3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f8525ffb640 1 -- 172.21.15.87:0/870767930 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380327a0 con 0x556855541580 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 0x7f85280a61e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f853804c000 tx=0x7f8538048000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f8525ffb640 1 -- 172.21.15.87:0/870767930 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538032940 con 0x556855541580 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.297+0000 7f8525ffb640 1 -- 172.21.15.87:0/870767930 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853804d930 con 0x556855541580 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- 172.21.15.87:0/870767930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 msgr2=0x556855541960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f8538009430 tx=0x7f8538032f30 comp rx=0 tx=0).stop 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- 172.21.15.87:0/870767930 shutdown_connections 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55685554aa80 0x55685554ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541ea0 0x55685554a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- 172.21.15.87:0/870767930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x556855541960 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- 172.21.15.87:0/870767930 >> 172.21.15.87:0/870767930 conn(0x55685553bcb0 msgr2=0x55685553c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- 172.21.15.87:0/870767930 shutdown_connections 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- 172.21.15.87:0/870767930 wait complete. 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- start start 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 Processor -- start 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x556855547b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x556855548060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55685554d870 con 0x55685554aa80 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55685545f2f0 con 0x556855541ea0 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55685554d9f0 con 0x556855541580 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.298+0000 7f8548620740 10 client.0 _ll_register_callbacks cb 0x55685546c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57044/0 (socket says 172.21.15.87:57044) 2024-04-22T13:40:34.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 -- 172.21.15.87:0/2132652263 learned_addr learned my addr 172.21.15.87:0/2132652263 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 -- 172.21.15.87:0/2132652263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 msgr2=0x556855547b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x556855547b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 -- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 msgr2=0x556855548060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x556855548060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 -- 172.21.15.87:0/2132652263 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568551b83f0 con 0x55685554aa80 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f853f7fe640 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f853804f040 tx=0x7f853804de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e466165e451a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f851effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853805a060 con 0x55685554aa80 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f8538003d40 con 0x55685554aa80 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8538032940 con 0x55685554aa80 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8538039040 con 0x55685554aa80 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f8548620740 10 client.18192 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:34.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5568551780f0 con 0x55685554aa80 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.299+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538032940 con 0x55685554aa80 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8538034440 con 0x55685554aa80 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 10 client.18192 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 10 client.18192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 20 client.18192 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55685555a8d0 0x55685555ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 4 client.18192 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568554b3c00 con 0x55685555a8d0 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f853effd640 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55685555a8d0 0x55685555ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f851effd640 10 client.18192 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.300+0000 7f853effd640 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55685555a8d0 0x55685555ccc0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c4ee125ec411f4 server_cookie=15e2ee0273646c53 in_seq=0 out_seq=0 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f851effd640 1 -- 172.21.15.87:0/2132652263 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568554b3c00 con 0x55685555a8d0 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f851effd640 10 client.18192 handle_command_reply: tid=0 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 client.18192 shutdown 2024-04-22T13:40:34.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 msgr2=0x5568555485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f853804f040 tx=0x7f853804de70 comp rx=0 tx=0).stop 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 shutdown_connections 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55685555a8d0 0x55685555ccc0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x5568555485a0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x556855548060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f851effd640 0 client.18192 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 --2- 172.21.15.87:0/2132652263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x556855547b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 >> 172.21.15.87:0/2132652263 conn(0x55685553bcb0 msgr2=0x55685553da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 shutdown_connections 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 1 -- 172.21.15.87:0/2132652263 wait complete. 2024-04-22T13:40:34.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.301+0000 7f8548620740 20 client.18192 trim_cache size 0 max 16384 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.318+0000 7f8548620740 1 Processor -- start 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.318+0000 7f8548620740 1 -- start start 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.318+0000 7f8548620740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x55685554a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f8548620740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685553d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f8548620740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f8548620740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f85401b24f0 con 0x55685554aa80 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f8548620740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f85401afdc0 con 0x556855541ea0 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f8548620740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85401adc30 con 0x556855541580 2024-04-22T13:40:34.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57056/0 (socket says 172.21.15.87:57056) 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 -- 172.21.15.87:0/53353128 learned_addr learned my addr 172.21.15.87:0/53353128 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 -- 172.21.15.87:0/53353128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 msgr2=0x55685554a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x55685554a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 -- 172.21.15.87:0/53353128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 msgr2=0x55685553d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685553d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 -- 172.21.15.87:0/53353128 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556855546070 con 0x55685554aa80 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f853f7fe640 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f853802c6d0 tx=0x7f853804d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f968688285d2c33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f85257fa640 1 -- 172.21.15.87:0/53353128 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538047050 con 0x55685554aa80 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f85257fa640 1 -- 172.21.15.87:0/53353128 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538003d40 con 0x55685554aa80 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.319+0000 7f85257fa640 1 -- 172.21.15.87:0/53353128 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380326e0 con 0x55685554aa80 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- 172.21.15.87:0/53353128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 msgr2=0x55685555fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f853802c6d0 tx=0x7f853804d400 comp rx=0 tx=0).stop 2024-04-22T13:40:34.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- 172.21.15.87:0/53353128 shutdown_connections 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55685554aa80 0x55685555fce0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685553d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 --2- 172.21.15.87:0/53353128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556855541580 0x55685554a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- 172.21.15.87:0/53353128 >> 172.21.15.87:0/53353128 conn(0x55685553bcb0 msgr2=0x55685553d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- 172.21.15.87:0/53353128 shutdown_connections 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- 172.21.15.87:0/53353128 wait complete. 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 -- start start 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.320+0000 7f8548620740 1 Processor -- start 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685555e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55685554aa80 0x5568555b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f85401ae280 con 0x556855541580 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f85401af6f0 con 0x556855541ea0 2024-04-22T13:40:34.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85401ae950 con 0x55685554aa80 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 10 client.0 _ll_register_callbacks cb 0x7f8538009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57072/0 (socket says 172.21.15.87:57072) 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 -- 172.21.15.87:0/1491483749 learned_addr learned my addr 172.21.15.87:0/1491483749 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 -- 172.21.15.87:0/1491483749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55685554aa80 msgr2=0x5568555b9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853f7fe640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55685554aa80 0x5568555b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853e7fc640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685555e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55685554aa80 0x5568555b9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 -- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 msgr2=0x55685555e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685555e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 -- 172.21.15.87:0/1491483749 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568555ba040 con 0x556855541580 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853e7fc640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685555e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:34.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f853effd640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f85340ce720 tx=0x7f853409d130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beae36e144741277 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f851effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340a55e0 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f853400fa90 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8538003d40 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f85340ce3a0 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 10 client.18198 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.321+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5568555c1440 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538003d40 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f85340bb020 con 0x556855541580 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 10 client.18198 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 10 client.18198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 20 client.18198 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5568555c1b10 0x5568555c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:34.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 4 client.18198 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568554ada70 con 0x5568555c1b10 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f853e7fc640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5568555c1b10 0x5568555c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f851effd640 10 client.18198 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.322+0000 7f853e7fc640 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5568555c1b10 0x5568555c3f30 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=548023cc7f5d5be3 server_cookie=d9f45f050d467680 in_seq=0 out_seq=0 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f851effd640 1 -- 172.21.15.87:0/1491483749 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5568554ada70 con 0x5568555c1b10 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f851effd640 10 client.18198 handle_command_reply: tid=0 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 client.18198 shutdown 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 msgr2=0x55685555df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f85340ce720 tx=0x7f853409d130 comp rx=0 tx=0).stop 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 shutdown_connections 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5568555c1b10 0x5568555c3f30 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55685554aa80 0x5568555b9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556855541ea0 0x55685555e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f851effd640 0 client.18198 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 --2- 172.21.15.87:0/1491483749 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556855541580 0x55685555df40 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 >> 172.21.15.87:0/1491483749 conn(0x55685553bcb0 msgr2=0x556855548350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 shutdown_connections 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 1 -- 172.21.15.87:0/1491483749 wait complete. 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8548620740 20 client.18198 trim_cache size 0 max 16384 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:40:34.326 INFO:teuthology.orchestra.run.smithi087.stdout: "6d0b89a4-cb28-479c-aa39-49dbfa4cddb4": { 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "6d0b89a4-cb28-479c-aa39-49dbfa4cddb4", 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:40:34.327 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 msgr2=0x7f85280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 0x7f85280a61e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f853804c000 tx=0x7f8538048000 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 msgr2=0x7f854010e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f853400c920 tx=0x7f853400cde0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 shutdown_connections 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f85280a3d20 0x7f85280a61e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f854010f140 0x7f85401a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8540106160 0x7f854010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 --2- 172.21.15.87:0/2749002579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8540101820 0x7f854010e6c0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.323+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 >> 172.21.15.87:0/2749002579 conn(0x7f8540077360 msgr2=0x7f85401042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:34.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.324+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 shutdown_connections 2024-04-22T13:40:34.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:40:34.324+0000 7f8545959640 1 -- 172.21.15.87:0/2749002579 wait complete. 2024-04-22T13:40:34.331 INFO:tasks.cephfs.filesystem:scrub status for tag:6d0b89a4-cb28-479c-aa39-49dbfa4cddb4 - {'path': '/', 'tag': '6d0b89a4-cb28-479c-aa39-49dbfa4cddb4', 'options': 'recursive,force'} 2024-04-22T13:40:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:34 smithi120 ceph-mon[31101]: pgmap v271: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T13:40:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:34 smithi132 ceph-mon[31531]: pgmap v271: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T13:40:34.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:34 smithi087 ceph-mon[27999]: pgmap v271: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T13:40:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:35 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:40:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:35 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:35 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:40:35.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:35 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:35 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:40:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:35 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:35 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:40:35.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:35 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:35.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:35 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:40:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:35 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:35 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:40:35.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:35 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:40:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:36 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:40:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:36 smithi120 ceph-mon[31101]: pgmap v272: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:40:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:36 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:40:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:36 smithi132 ceph-mon[31531]: pgmap v272: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:40:36.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:36 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:40:36.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:36 smithi087 ceph-mon[27999]: pgmap v272: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:40:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:38 smithi120 ceph-mon[31101]: pgmap v273: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:38.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:38 smithi132 ceph-mon[31531]: pgmap v273: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:38.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:38 smithi087 ceph-mon[27999]: pgmap v273: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:40 smithi120 ceph-mon[31101]: pgmap v274: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:40 smithi132 ceph-mon[31531]: pgmap v274: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:40 smithi087 ceph-mon[27999]: pgmap v274: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:42.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:42 smithi120 ceph-mon[31101]: pgmap v275: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:42.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:42 smithi132 ceph-mon[31531]: pgmap v275: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:42.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:42 smithi087 ceph-mon[27999]: pgmap v275: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:40:44.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:44 smithi120 ceph-mon[31101]: pgmap v276: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:44.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:44 smithi132 ceph-mon[31531]: pgmap v276: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:44.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:44 smithi087 ceph-mon[27999]: pgmap v276: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 -- 172.21.15.87:0/1537079339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c06b120 msgr2=0x7fbb8c10edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 --2- 172.21.15.87:0/1537079339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c06b120 0x7fbb8c10edf0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009ca0 tx=0x7fbb8402c180 comp rx=0 tx=0).stop 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 -- 172.21.15.87:0/1537079339 shutdown_connections 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 --2- 172.21.15.87:0/1537079339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c06b120 0x7fbb8c10edf0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 --2- 172.21.15.87:0/1537079339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 0x7fbb8c06abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 --2- 172.21.15.87:0/1537079339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.238+0000 7fbb94021640 1 -- 172.21.15.87:0/1537079339 >> 172.21.15.87:0/1537079339 conn(0x7fbb8c076c50 msgr2=0x7fbb8c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- 172.21.15.87:0/1537079339 shutdown_connections 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- 172.21.15.87:0/1537079339 wait complete. 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 Processor -- start 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- start start 2024-04-22T13:40:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 0x7fbb8c101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 0x7fbb8c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c112190 con 0x7fbb8c1080e0 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c112010 con 0x7fbb8c06b120 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.239+0000 7fbb94021640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c112310 con 0x7fbb8c06a780 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43768/0 (socket says 172.21.15.87:43768) 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 -- 172.21.15.87:0/1301265992 learned_addr learned my addr 172.21.15.87:0/1301265992 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 0x7fbb8c101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb915bc640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 0x7fbb8c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 -- 172.21.15.87:0/1301265992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 msgr2=0x7fbb8c101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 0x7fbb8c101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 -- 172.21.15.87:0/1301265992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 msgr2=0x7fbb8c1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 0x7fbb8c1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 -- 172.21.15.87:0/1301265992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c1a6ab0 con 0x7fbb8c1080e0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb915bc640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 0x7fbb8c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb925be640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbb8402c790 tx=0x7fbb84003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb82ffd640 1 -- 172.21.15.87:0/1301265992 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84039040 con 0x7fbb8c1080e0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb82ffd640 1 -- 172.21.15.87:0/1301265992 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb84044070 con 0x7fbb8c1080e0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8c1a6d40 con 0x7fbb8c1080e0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb82ffd640 1 -- 172.21.15.87:0/1301265992 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84032850 con 0x7fbb8c1080e0 2024-04-22T13:40:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.240+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb8c1a7230 con 0x7fbb8c1080e0 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb82ffd640 1 -- 172.21.15.87:0/1301265992 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb84004240 con 0x7fbb8c1080e0 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 Processor -- start 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 -- start start 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb82ffd640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 0x7fbb780a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f58600a8d0 0x55f586009620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f586009b60 0x55f586009fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.243+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f585f6e8d0 con 0x55f5860130e0 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f585f0be80 con 0x55f58600a8d0 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f585f6a890 con 0x55f586009b60 2024-04-22T13:40:45.246 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb915bc640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 0x7fbb780a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb82ffd640 1 -- 172.21.15.87:0/1301265992 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fbb840f2bd0 con 0x7fbb8c1080e0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43780/0 (socket says 172.21.15.87:43780) 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 -- 172.21.15.87:0/1558399572 learned_addr learned my addr 172.21.15.87:0/1558399572 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb915bc640 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f586009b60 0x55f586009fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f58600a8d0 0x55f586009620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 -- 172.21.15.87:0/1558399572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f586009b60 msgr2=0x55f586009fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f586009b60 0x55f586009fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 -- 172.21.15.87:0/1558399572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f58600a8d0 msgr2=0x55f586009620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f58600a8d0 0x55f586009620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 -- 172.21.15.87:0/1558399572 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f585f2d320 con 0x55f5860130e0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb915bc640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 0x7fbb780a6280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005d00 tx=0x7fbb7c015840 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb925be640 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbb840c7400 tx=0x7fbb840bea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3013508247c8c1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb77fff640 1 -- 172.21.15.87:0/1558399572 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84003e10 con 0x55f5860130e0 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb77fff640 1 -- 172.21.15.87:0/1558399572 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb840bf3c0 con 0x55f5860130e0 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.244+0000 7fbb77fff640 1 -- 172.21.15.87:0/1558399572 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840bb330 con 0x55f5860130e0 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 -- 172.21.15.87:0/1558399572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 msgr2=0x55f5860154d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbb840c7400 tx=0x7fbb840bea90 comp rx=0 tx=0).stop 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 -- 172.21.15.87:0/1558399572 shutdown_connections 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860154d0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f586009b60 0x55f586009fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/1558399572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f58600a8d0 0x55f586009620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 -- 172.21.15.87:0/1558399572 >> 172.21.15.87:0/1558399572 conn(0x55f585fd0c90 msgr2=0x55f586004850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 -- 172.21.15.87:0/1558399572 shutdown_connections 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.245+0000 7fbb96c0b740 1 -- 172.21.15.87:0/1558399572 wait complete. 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 -- start start 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 Processor -- start 2024-04-22T13:40:45.248 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f586012a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f585f6e8d0 con 0x55f5860130e0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f585ef6eb0 con 0x55f586009b60 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f586015bd0 con 0x55f58600a8d0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb96c0b740 10 client.0 _ll_register_callbacks cb 0x55f585f23b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43782/0 (socket says 172.21.15.87:43782) 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb925be640 1 -- 172.21.15.87:0/777045224 learned_addr learned my addr 172.21.15.87:0/777045224 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.246+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f586012a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb915bc640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 -- 172.21.15.87:0/777045224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 msgr2=0x55f58600fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 -- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 msgr2=0x55f586012a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f586012a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 -- 172.21.15.87:0/777045224 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f585c81210 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb915bc640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb925be640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbb840bb120 tx=0x7fbb840bebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d142ae661c88029f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb757fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840d8060 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fbb840bef20 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb840bf3c0 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb840d8200 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb96c0b740 10 client.18210 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f585d4d620 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.247+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb840bf3c0 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb8403c030 con 0x55f5860130e0 2024-04-22T13:40:45.250 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 10 client.18210 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 10 client.18210 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 20 client.18210 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f586022b80 0x55f586024fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 4 client.18210 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f585f82880 con 0x55f586022b80 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f586022b80 0x55f586024fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb757fa640 10 client.18210 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.248+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f586022b80 0x55f586024fa0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa73239450811976 server_cookie=6b10e39417583ad9 in_seq=0 out_seq=0 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb757fa640 1 -- 172.21.15.87:0/777045224 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f585f82880 con 0x55f586022b80 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb757fa640 10 client.18210 handle_command_reply: tid=0 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 client.18210 shutdown 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 msgr2=0x55f5860102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.251 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbb840bb120 tx=0x7fbb840bebc0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 shutdown_connections 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f586022b80 0x55f586024fa0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f5860102d0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/777045224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f586012a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb757fa640 0 client.18210 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 >> 172.21.15.87:0/777045224 conn(0x55f585fd0c90 msgr2=0x55f5860065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 shutdown_connections 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.249+0000 7fbb96c0b740 1 -- 172.21.15.87:0/777045224 wait complete. 2024-04-22T13:40:45.252 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.250+0000 7fbb96c0b740 20 client.18210 trim_cache size 0 max 16384 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 Processor -- start 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 -- start start 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f5860130e0 0x55f5860063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1b2430 con 0x55f58600a8d0 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1afd00 con 0x55f586009b60 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.264+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1adb70 con 0x55f5860130e0 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43786/0 (socket says 172.21.15.87:43786) 2024-04-22T13:40:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 -- 172.21.15.87:0/611127770 learned_addr learned my addr 172.21.15.87:0/611127770 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb925be640 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f5860130e0 0x55f5860063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 -- 172.21.15.87:0/611127770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f5860130e0 msgr2=0x55f5860063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f5860130e0 0x55f5860063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 -- 172.21.15.87:0/611127770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 msgr2=0x55f58601e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 -- 172.21.15.87:0/611127770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f586025620 con 0x55f58600a8d0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb915bc640 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c102d00 tx=0x7fbb7c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1b6af025a980c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb777fe640 1 -- 172.21.15.87:0/611127770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c005020 con 0x55f58600a8d0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb777fe640 1 -- 172.21.15.87:0/611127770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb7c0047a0 con 0x55f58600a8d0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.265+0000 7fbb777fe640 1 -- 172.21.15.87:0/611127770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c020690 con 0x55f58600a8d0 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- 172.21.15.87:0/611127770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 msgr2=0x55f5860049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c102d00 tx=0x7fbb7c018000 comp rx=0 tx=0).stop 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- 172.21.15.87:0/611127770 shutdown_connections 2024-04-22T13:40:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f5860130e0 0x55f5860063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f58600a8d0 0x55f5860049f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/611127770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- 172.21.15.87:0/611127770 >> 172.21.15.87:0/611127770 conn(0x55f585fd0c90 msgr2=0x55f586005ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- 172.21.15.87:0/611127770 shutdown_connections 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- 172.21.15.87:0/611127770 wait complete. 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 -- start start 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.266+0000 7fbb96c0b740 1 Processor -- start 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1ae1c0 con 0x55f5860130e0 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1af630 con 0x55f586009b60 2024-04-22T13:40:45.267 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1ae890 con 0x55f58600a8d0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 10 client.0 _ll_register_callbacks cb 0x7fbb880097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43802/0 (socket says 172.21.15.87:43802) 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 -- 172.21.15.87:0/420890282 learned_addr learned my addr 172.21.15.87:0/420890282 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb915bc640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 -- 172.21.15.87:0/420890282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 msgr2=0x55f58600c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 -- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 msgr2=0x55f58601b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 -- 172.21.15.87:0/420890282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f586008e30 con 0x55f5860130e0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb925be640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fbb840c7400 tx=0x7fbb840c77f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e110ffb2c1d142a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb757fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840d8060 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fbb84004b40 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.267+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb7c0047a0 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb840bf130 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 10 client.18216 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f586008e30 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb7c0047a0 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb8409dae0 con 0x55f5860130e0 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 10 client.18216 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 10 client.18216 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 20 client.18216 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f5860892e0 0x55f58608b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 4 client.18216 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:40:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f585f0e3d0 con 0x55f5860892e0 2024-04-22T13:40:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.268+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f5860892e0 0x55f58608b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.269+0000 7fbb757fa640 10 client.18216 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.269+0000 7fbb91dbd640 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f5860892e0 0x55f58608b6d0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d8630a9f3d67d5 server_cookie=4bb5d5faef109fd0 in_seq=0 out_seq=0 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb757fa640 1 -- 172.21.15.87:0/420890282 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f585f0e3d0 con 0x55f5860892e0 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb757fa640 10 client.18216 handle_command_reply: tid=0 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 client.18216 shutdown 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 msgr2=0x55f58600ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fbb840c7400 tx=0x7fbb840c77f0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 shutdown_connections 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f5860892e0 0x55f58608b6d0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f5860130e0 0x55f58600ce80 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f58600a8d0 0x55f58600c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb757fa640 0 client.18216 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 --2- 172.21.15.87:0/420890282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f586009b60 0x55f58601b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 >> 172.21.15.87:0/420890282 conn(0x55f585fd0c90 msgr2=0x55f5860038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 shutdown_connections 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 1 -- 172.21.15.87:0/420890282 wait complete. 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.274+0000 7fbb96c0b740 20 client.18216 trim_cache size 0 max 16384 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:40:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:40:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 msgr2=0x7fbb780a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 0x7fbb780a6280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005d00 tx=0x7fbb7c015840 comp rx=0 tx=0).stop 2024-04-22T13:40:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 msgr2=0x7fbb8c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbb8402c790 tx=0x7fbb84003c20 comp rx=0 tx=0).stop 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 shutdown_connections 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb780a3dc0 0x7fbb780a6280 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb8c1080e0 0x7fbb8c101cf0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb8c06b120 0x7fbb8c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 --2- 172.21.15.87:0/1301265992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c06a780 0x7fbb8c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 >> 172.21.15.87:0/1301265992 conn(0x7fbb8c076c50 msgr2=0x7fbb8c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 shutdown_connections 2024-04-22T13:40:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:45.275+0000 7fbb94021640 1 -- 172.21.15.87:0/1301265992 wait complete. 2024-04-22T13:40:46.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:46 smithi120 ceph-mon[31101]: pgmap v277: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8 op/s 2024-04-22T13:40:46.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:46 smithi132 ceph-mon[31531]: pgmap v277: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8 op/s 2024-04-22T13:40:46.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:46 smithi087 ceph-mon[27999]: pgmap v277: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8 op/s 2024-04-22T13:40:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:48 smithi120 ceph-mon[31101]: pgmap v278: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:40:48.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:48 smithi132 ceph-mon[31531]: pgmap v278: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:40:48.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:48 smithi087 ceph-mon[27999]: pgmap v278: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:40:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:50 smithi120 ceph-mon[31101]: pgmap v279: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:50 smithi132 ceph-mon[31531]: pgmap v279: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:50 smithi087 ceph-mon[27999]: pgmap v279: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:52 smithi120 ceph-mon[31101]: pgmap v280: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:52.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:52 smithi132 ceph-mon[31531]: pgmap v280: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:52.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:52 smithi087 ceph-mon[27999]: pgmap v280: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:54 smithi120 ceph-mon[31101]: pgmap v281: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:40:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:54 smithi132 ceph-mon[31531]: pgmap v281: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:40:54.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:54 smithi087 ceph-mon[27999]: pgmap v281: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:40:56.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:56 smithi120 ceph-mon[31101]: pgmap v282: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:56.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:56 smithi132 ceph-mon[31531]: pgmap v282: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:56.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:56 smithi087 ceph-mon[27999]: pgmap v282: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 -- 172.21.15.87:0/1335487652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 msgr2=0x7f464806abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 --2- 172.21.15.87:0/1335487652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f464806abe0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4638009900 tx=0x7f463802bcd0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 -- 172.21.15.87:0/1335487652 shutdown_connections 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 --2- 172.21.15.87:0/1335487652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f464810edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 --2- 172.21.15.87:0/1335487652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f464806abe0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 --2- 172.21.15.87:0/1335487652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 0x7f46481084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.376+0000 7f4650104640 1 -- 172.21.15.87:0/1335487652 >> 172.21.15.87:0/1335487652 conn(0x7f4648076c50 msgr2=0x7f4648077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- 172.21.15.87:0/1335487652 shutdown_connections 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- 172.21.15.87:0/1335487652 wait complete. 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 Processor -- start 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- start start 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f4648198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 0x7f464819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46481121a0 con 0x7f464806a780 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112020 con 0x7f464806b120 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f4650104640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112320 con 0x7f46481080e0 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.377+0000 7f464dea0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45898/0 (socket says 172.21.15.87:45898) 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 -- 172.21.15.87:0/2765338650 learned_addr learned my addr 172.21.15.87:0/2765338650 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464d69f640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f4648198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464e6a1640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 0x7f464819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 -- 172.21.15.87:0/2765338650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 msgr2=0x7f464819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 0x7f464819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 -- 172.21.15.87:0/2765338650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 msgr2=0x7f4648198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f4648198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 -- 172.21.15.87:0/2765338650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464819d5c0 con 0x7f464806a780 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464d69f640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f4648198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f464dea0640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f464400d770 tx=0x7f464400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f463effd640 1 -- 172.21.15.87:0/2765338650 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644017070 con 0x7f464806a780 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f463effd640 1 -- 172.21.15.87:0/2765338650 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46440054d0 con 0x7f464806a780 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f463effd640 1 -- 172.21.15.87:0/2765338650 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644004c30 con 0x7f464806a780 2024-04-22T13:40:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464819d8b0 con 0x7f464806a780 2024-04-22T13:40:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.378+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46481a5140 con 0x7f464806a780 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.380+0000 7f463effd640 1 -- 172.21.15.87:0/2765338650 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4644004e10 con 0x7f464806a780 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 Processor -- start 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 -- start start 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a1268d0 0x56422a125620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f463effd640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 0x7f46340a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56422a08a8d0 con 0x56422a12f0e0 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56422a027e80 con 0x56422a125b60 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f4652dd3740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56422a086890 con 0x56422a1268d0 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464dea0640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 0x7f46340a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464d69f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464d69f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:59938/0 (socket says 172.21.15.87:59938) 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45900/0 (socket says 172.21.15.87:45900) 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464d69f640 1 -- 172.21.15.87:0/4109240815 learned_addr learned my addr 172.21.15.87:0/4109240815 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f463effd640 1 -- 172.21.15.87:0/2765338650 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f46440d0340 con 0x7f464806a780 2024-04-22T13:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.381+0000 7f464e6a1640 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a1268d0 0x56422a125620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 -- 172.21.15.87:0/4109240815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a1268d0 msgr2=0x56422a125620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a1268d0 0x56422a125620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 -- 172.21.15.87:0/4109240815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 msgr2=0x56422a1314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 -- 172.21.15.87:0/4109240815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56422a049320 con 0x56422a125b60 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464e6a1640 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464dea0640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 0x7f46340a6280 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4644098cf0 tx=0x7f4644016310 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f464d69f640 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ce54fc99221d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f4633fff640 1 -- 172.21.15.87:0/4109240815 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638004400 con 0x56422a125b60 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f4633fff640 1 -- 172.21.15.87:0/4109240815 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46380054f0 con 0x56422a125b60 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.382+0000 7f4633fff640 1 -- 172.21.15.87:0/4109240815 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46380046c0 con 0x56422a125b60 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- 172.21.15.87:0/4109240815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 msgr2=0x56422a125fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4638000c00 tx=0x7f4638005e80 comp rx=0 tx=0).stop 2024-04-22T13:40:58.387 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- 172.21.15.87:0/4109240815 shutdown_connections 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a1314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a125fe0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 --2- 172.21.15.87:0/4109240815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a1268d0 0x56422a125620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- 172.21.15.87:0/4109240815 >> 172.21.15.87:0/4109240815 conn(0x56422a0ecc90 msgr2=0x56422a120850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- 172.21.15.87:0/4109240815 shutdown_connections 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- 172.21.15.87:0/4109240815 wait complete. 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 -- start start 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.383+0000 7f4652dd3740 1 Processor -- start 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a125b60 0x56422a12ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a12bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56422a131db0 con 0x56422a12f0e0 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56422a012eb0 con 0x56422a1268d0 2024-04-22T13:40:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56422a131f30 con 0x56422a125b60 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f4652dd3740 10 client.0 _ll_register_callbacks cb 0x56422a03fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45910/0 (socket says 172.21.15.87:45910) 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a12bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464e6a1640 1 -- 172.21.15.87:0/1021501902 learned_addr learned my addr 172.21.15.87:0/1021501902 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a125b60 0x56422a12ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 -- 172.21.15.87:0/1021501902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a125b60 msgr2=0x56422a12ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a125b60 0x56422a12ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 -- 172.21.15.87:0/1021501902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 msgr2=0x56422a12c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464d69f640 1 -- 172.21.15.87:0/1021501902 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564229d9d210 con 0x56422a1268d0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.384+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f464d69f640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a12bda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4638005940 tx=0x7f46380056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782613e0372f63e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463802cb50 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f46380035f0 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46380054f0 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f46380037f0 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f4652dd3740 10 client.27101 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564229e69620 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46380054f0 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.385+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4638034850 con 0x56422a1268d0 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 10 client.27101 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 10 client.27101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 20 client.27101 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a13eef0 0x56422a1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 4 client.27101 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56422a09e880 con 0x56422a13eef0 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a13eef0 0x56422a1412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f46317fa640 10 client.27101 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.386+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a13eef0 0x56422a1412e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5abf87e240c3b5 server_cookie=c01187497d4ecb9a in_seq=0 out_seq=0 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f46317fa640 1 -- 172.21.15.87:0/1021501902 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56422a09e880 con 0x56422a13eef0 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f46317fa640 10 client.27101 handle_command_reply: tid=0 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 client.27101 shutdown 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 msgr2=0x56422a12bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a12bda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4638005940 tx=0x7f46380056a0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 shutdown_connections 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a13eef0 0x56422a1412e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a12f0e0 0x56422a12c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a12bda0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f46317fa640 0 client.27101 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1021501902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a125b60 0x56422a12ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 >> 172.21.15.87:0/1021501902 conn(0x56422a0ecc90 msgr2=0x56422a1225a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 shutdown_connections 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 1 -- 172.21.15.87:0/1021501902 wait complete. 2024-04-22T13:40:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.387+0000 7f4652dd3740 20 client.27101 trim_cache size 0 max 16384 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 Processor -- start 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 -- start start 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a12e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46481b01f0 con 0x56422a1268d0 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46481adac0 con 0x56422a125b60 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f4652dd3740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46481ab930 con 0x56422a12f0e0 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45922/0 (socket says 172.21.15.87:45922) 2024-04-22T13:40:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 -- 172.21.15.87:0/584842718 learned_addr learned my addr 172.21.15.87:0/584842718 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464e6a1640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 -- 172.21.15.87:0/584842718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 msgr2=0x56422a12e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464e6a1640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a12e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a12e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 -- 172.21.15.87:0/584842718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 msgr2=0x56422a13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.402+0000 7f464d69f640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a13ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f464d69f640 1 -- 172.21.15.87:0/584842718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56422a124e30 con 0x56422a1268d0 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f464e6a1640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a13ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f464d69f640 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f463802c210 tx=0x7f4638005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704f9ad4333ef8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f46337fe640 1 -- 172.21.15.87:0/584842718 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46380035f0 con 0x56422a1268d0 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f46337fe640 1 -- 172.21.15.87:0/584842718 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f463802cb50 con 0x56422a1268d0 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f46337fe640 1 -- 172.21.15.87:0/584842718 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638004040 con 0x56422a1268d0 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 -- 172.21.15.87:0/584842718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 msgr2=0x56422a12c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f463802c210 tx=0x7f4638005e80 comp rx=0 tx=0).stop 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 -- 172.21.15.87:0/584842718 shutdown_connections 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a12e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a1268d0 0x56422a12c1c0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 --2- 172.21.15.87:0/584842718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a125b60 0x56422a13ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.403+0000 7f4652dd3740 1 -- 172.21.15.87:0/584842718 >> 172.21.15.87:0/584842718 conn(0x56422a0ecc90 msgr2=0x56422a12a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- 172.21.15.87:0/584842718 shutdown_connections 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- 172.21.15.87:0/584842718 wait complete. 2024-04-22T13:40:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- start start 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 Processor -- start 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a132cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46481abf80 con 0x56422a125b60 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46481ad3f0 con 0x56422a1268d0 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46481ac650 con 0x56422a12f0e0 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f4652dd3740 10 client.0 _ll_register_callbacks cb 0x7f4640007b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f464e6a1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45932/0 (socket says 172.21.15.87:45932) 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.404+0000 7f464e6a1640 1 -- 172.21.15.87:0/1839530879 learned_addr learned my addr 172.21.15.87:0/1839530879 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a132cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464d69f640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 -- 172.21.15.87:0/1839530879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 msgr2=0x56422a132cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a132cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 -- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 msgr2=0x56422a132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a132790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 -- 172.21.15.87:0/1839530879 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56422a133310 con 0x56422a125b60 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464d69f640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f464e6a1640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4640015010 tx=0x7f4640013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2488cd7790b033d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f46317fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464002b030 con 0x56422a125b60 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f464001c040 con 0x56422a125b60 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f463802cb50 con 0x56422a125b60 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4640024650 con 0x56422a125b60 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f4652dd3740 10 client.18234 fetch_fsmap learned FSMap version 23 2024-04-22T13:40:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56422a1a4fd0 con 0x56422a125b60 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.405+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f463802cb50 con 0x56422a125b60 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4640010620 con 0x56422a125b60 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 10 client.18234 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 10 client.18234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 20 client.18234 populate_metadata read hostname 'smithi087' 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a1a56d0 0x56422a1a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 4 client.18234 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56422a02a3d0 con 0x56422a1a56d0 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f464d69f640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a1a56d0 0x56422a1a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f46317fa640 10 client.18234 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.406+0000 7f464d69f640 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a1a56d0 0x56422a1a7af0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b4bb65ce26d78b server_cookie=9fa6db01084f823e in_seq=0 out_seq=0 2024-04-22T13:40:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f46317fa640 1 -- 172.21.15.87:0/1839530879 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56422a02a3d0 con 0x56422a1a56d0 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f46317fa640 10 client.18234 handle_command_reply: tid=0 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 client.18234 shutdown 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 msgr2=0x56422a1373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4640015010 tx=0x7f4640013040 comp rx=0 tx=0).stop 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 shutdown_connections 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56422a1a56d0 0x56422a1a7af0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56422a12f0e0 0x56422a132cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f46317fa640 0 client.18234 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56422a1268d0 0x56422a132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 --2- 172.21.15.87:0/1839530879 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56422a125b60 0x56422a1373d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 >> 172.21.15.87:0/1839530879 conn(0x56422a0ecc90 msgr2=0x56422a11f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 shutdown_connections 2024-04-22T13:40:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.411+0000 7f4652dd3740 1 -- 172.21.15.87:0/1839530879 wait complete. 2024-04-22T13:40:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4652dd3740 20 client.18234 trim_cache size 0 max 16384 2024-04-22T13:40:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:40:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:40:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:40:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 msgr2=0x7f46340a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 0x7f46340a6280 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4644098cf0 tx=0x7f4644016310 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 msgr2=0x7f4648198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f464400d770 tx=0x7f464400dc30 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 shutdown_connections 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46340a3dc0 0x7f46340a6280 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46481080e0 0x7f464819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464806b120 0x7f4648198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 --2- 172.21.15.87:0/2765338650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f464806a780 0x7f4648198620 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:40:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 >> 172.21.15.87:0/2765338650 conn(0x7f4648076c50 msgr2=0x7f46480ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:40:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 shutdown_connections 2024-04-22T13:40:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:40:58.412+0000 7f4650104640 1 -- 172.21.15.87:0/2765338650 wait complete. 2024-04-22T13:40:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:40:58 smithi120 ceph-mon[31101]: pgmap v283: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:40:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:40:58 smithi132 ceph-mon[31531]: pgmap v283: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:40:58.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:40:58 smithi087 ceph-mon[27999]: pgmap v283: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:00.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:00 smithi120 ceph-mon[31101]: pgmap v284: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:00.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:00 smithi132 ceph-mon[31531]: pgmap v284: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:00.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:00 smithi087 ceph-mon[27999]: pgmap v284: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:02 smithi120 ceph-mon[31101]: pgmap v285: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:02 smithi132 ceph-mon[31531]: pgmap v285: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:02.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:02 smithi087 ceph-mon[27999]: pgmap v285: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:41:04.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:41:04.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 -- 172.21.15.87:0/2903078639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc1021a0 msgr2=0x7f9cdc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2903078639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc102580 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009900 tx=0x7f9cd802bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 -- 172.21.15.87:0/2903078639 shutdown_connections 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2903078639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc107270 0x7f9cdc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2903078639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc102b50 0x7f9cdc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2903078639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc102580 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 -- 172.21.15.87:0/2903078639 >> 172.21.15.87:0/2903078639 conn(0x7f9cdc0793e0 msgr2=0x7f9cdc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 -- 172.21.15.87:0/2903078639 shutdown_connections 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.447+0000 7f9ce453d640 1 -- 172.21.15.87:0/2903078639 wait complete. 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 Processor -- start 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 -- start start 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc19cef0 0x7f9cdc19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1121b0 con 0x7f9cdc107270 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc112030 con 0x7f9cdc1021a0 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce453d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc112330 con 0x7f9cdc19cef0 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58974/0 (socket says 172.21.15.87:58974) 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2244192833 learned_addr learned my addr 172.21.15.87:0/2244192833 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc19cef0 0x7f9cdc19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2244192833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc19cef0 msgr2=0x7f9cdc19d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.448+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc19cef0 0x7f9cdc19d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2244192833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc1021a0 msgr2=0x7f9cdc198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2244192833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc19d9e0 con 0x7f9cdc107270 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c970 tx=0x7f9ccc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9cd37fe640 1 -- 172.21.15.87:0/2244192833 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc014070 con 0x7f9cdc107270 2024-04-22T13:41:04.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9cd37fe640 1 -- 172.21.15.87:0/2244192833 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ccc004440 con 0x7f9cdc107270 2024-04-22T13:41:04.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc19dcd0 con 0x7f9cdc107270 2024-04-22T13:41:04.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9cd37fe640 1 -- 172.21.15.87:0/2244192833 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc003e50 con 0x7f9cdc107270 2024-04-22T13:41:04.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.449+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc1a54c0 con 0x7f9cdc107270 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.451+0000 7f9cd37fe640 1 -- 172.21.15.87:0/2244192833 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ccc017070 con 0x7f9cdc107270 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 Processor -- start 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 -- start start 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac61960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac6ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9cd37fe640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 0x7f9cc80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b88abc61e0 con 0x55b88ac61ea0 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b88ab63390 con 0x55b88ac61580 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce720a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b88abc21a0 con 0x55b88ac6aa80 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce22d9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 0x7f9cc80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce22d9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58978/0 (socket says 172.21.15.87:58978) 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce22d9640 1 -- 172.21.15.87:0/1577491354 learned_addr learned my addr 172.21.15.87:0/1577491354 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9cd37fe640 1 -- 172.21.15.87:0/2244192833 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f9ccc0d0350 con 0x7f9cdc107270 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.452+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac61960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac6ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 -- 172.21.15.87:0/1577491354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 msgr2=0x55b88ac6ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac6ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 -- 172.21.15.87:0/1577491354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 msgr2=0x55b88ac61960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac61960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 -- 172.21.15.87:0/1577491354 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88a9a27d0 con 0x55b88ac61ea0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 0x7f9cc80a6380 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4007fd0 tx=0x7f9cd4007c60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009900 tx=0x7f9cd80040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2f5bf34bbfe721 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9cd0bfe640 1 -- 172.21.15.87:0/1577491354 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8045020 con 0x55b88ac61ea0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9cd0bfe640 1 -- 172.21.15.87:0/1577491354 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cd802cc90 con 0x55b88ac61ea0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.453+0000 7f9cd0bfe640 1 -- 172.21.15.87:0/1577491354 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80046c0 con 0x55b88ac61ea0 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- 172.21.15.87:0/1577491354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 msgr2=0x55b88ac6a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009900 tx=0x7f9cd80040f0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- 172.21.15.87:0/1577491354 shutdown_connections 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac6ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6a540 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1577491354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac61960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- 172.21.15.87:0/1577491354 >> 172.21.15.87:0/1577491354 conn(0x55b88ac5bcb0 msgr2=0x55b88ac5c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- 172.21.15.87:0/1577491354 shutdown_connections 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- 172.21.15.87:0/1577491354 wait complete. 2024-04-22T13:41:04.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 -- start start 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.454+0000 7f9ce720a740 1 Processor -- start 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88ac68430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b88ac6d810 con 0x55b88ac61ea0 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b88ab7f2f0 con 0x55b88ac6aa80 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b88ac6d990 con 0x55b88ac61580 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 10 client.0 _ll_register_callbacks cb 0x55b88ab8c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58990/0 (socket says 172.21.15.87:58990) 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2621164225 learned_addr learned my addr 172.21.15.87:0/2621164225 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88ac68430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2621164225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 msgr2=0x55b88ac679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 msgr2=0x55b88ac68430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88ac68430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2621164225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88a8d83f0 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88ac68430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc09c2c0 tx=0x7f9ccc098c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f721535b7d587fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9cc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc09a210 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f9ccc099980 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cd802cc90 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9ccc099b80 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 10 client.18246 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.455+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b88a8980f0 con 0x55b88ac61ea0 2024-04-22T13:41:04.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cd802cc90 con 0x55b88ac61ea0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9ccc099d80 con 0x55b88ac61ea0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 10 client.18246 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 10 client.18246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 20 client.18246 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ac7a8b0 0x55b88ac7ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 4 client.18246 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b88abd3c00 con 0x55b88ac7a8b0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ac7a8b0 0x55b88ac7ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9cc5ffb640 10 client.18246 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.456+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ac7a8b0 0x55b88ac7ccd0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a25956caa56101c server_cookie=58cc3fa2af9c2fcd in_seq=0 out_seq=0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2621164225 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b88abd3c00 con 0x55b88ac7a8b0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9cc5ffb640 10 client.18246 handle_command_reply: tid=0 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 client.18246 shutdown 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 msgr2=0x55b88ac67ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc09c2c0 tx=0x7f9ccc098c50 comp rx=0 tx=0).stop 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 shutdown_connections 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ac7a8b0 0x55b88ac7ccd0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88ac68430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9cc5ffb640 0 client.18246 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac67ef0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2621164225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 >> 172.21.15.87:0/2621164225 conn(0x55b88ac5bcb0 msgr2=0x55b88ac5da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 shutdown_connections 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.457+0000 7f9ce720a740 1 -- 172.21.15.87:0/2621164225 wait complete. 2024-04-22T13:41:04.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.458+0000 7f9ce720a740 20 client.18246 trim_cache size 0 max 16384 2024-04-22T13:41:04.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 Processor -- start 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 -- start start 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac6a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac66b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1b0240 con 0x55b88ac61ea0 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1adb10 con 0x55b88ac61580 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce720a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1ab980 con 0x55b88ac6aa80 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59006/0 (socket says 172.21.15.87:59006) 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/1914106014 learned_addr learned my addr 172.21.15.87:0/1914106014 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/1914106014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 msgr2=0x55b88ac66b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac66b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/1914106014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 msgr2=0x55b88ac6a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac6a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac6a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/1914106014 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88ac5ccb0 con 0x55b88ac61ea0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.475+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0a4a70 tx=0x7f9ccc0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485fc363ecabf7be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9cc7fff640 1 -- 172.21.15.87:0/1914106014 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0a4ff0 con 0x55b88ac61ea0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9cc7fff640 1 -- 172.21.15.87:0/1914106014 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ccc09ca30 con 0x55b88ac61ea0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9cc7fff640 1 -- 172.21.15.87:0/1914106014 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc09bc90 con 0x55b88ac61ea0 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- 172.21.15.87:0/1914106014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 msgr2=0x55b88ac766d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0a4a70 tx=0x7f9ccc0045f0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- 172.21.15.87:0/1914106014 shutdown_connections 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac6aa80 0x55b88ac66b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac766d0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 --2- 172.21.15.87:0/1914106014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac61580 0x55b88ac6a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- 172.21.15.87:0/1914106014 >> 172.21.15.87:0/1914106014 conn(0x55b88ac5bcb0 msgr2=0x55b88ac67970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- 172.21.15.87:0/1914106014 shutdown_connections 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- 172.21.15.87:0/1914106014 wait complete. 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 -- start start 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.476+0000 7f9ce720a740 1 Processor -- start 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac6e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88acdfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1abfd0 con 0x55b88ac61ea0 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1ad440 con 0x55b88ac6aa80 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1ac6a0 con 0x55b88ac61580 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 10 client.0 _ll_register_callbacks cb 0x7f9ccc09c100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:04.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59010/0 (socket says 172.21.15.87:59010) 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2388564979 learned_addr learned my addr 172.21.15.87:0/2388564979 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac6e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88acdfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2388564979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 msgr2=0x55b88ac6e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac6e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 msgr2=0x55b88acdfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88acdfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 -- 172.21.15.87:0/2388564979 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88ac6f6e0 con 0x55b88ac61ea0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce2ada640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88acdfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce1ad8640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc09ae30 tx=0x7f9ccc0bb190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a372569ac160cd01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9cc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0a4fa0 con 0x55b88ac61ea0 2024-04-22T13:41:04.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ccc09ca30 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f9ccc09a8f0 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.477+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9ccc09c330 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 10 client.18252 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b88ace13e0 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ccc09ca30 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9ccc0981f0 con 0x55b88ac61ea0 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 10 client.18252 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 10 client.18252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 20 client.18252 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ace17c0 0x55b88ace3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 4 client.18252 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:41:04.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b88abcda70 con 0x55b88ace17c0 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ace17c0 0x55b88ace3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9ce22d9640 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ace17c0 0x55b88ace3be0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd7dbe0d3e9cfa07 server_cookie=e1f06a7e5600fc50 in_seq=0 out_seq=0 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.478+0000 7f9cc5ffb640 10 client.18252 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9cc5ffb640 1 -- 172.21.15.87:0/2388564979 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b88abcda70 con 0x55b88ace17c0 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9cc5ffb640 10 client.18252 handle_command_reply: tid=0 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 client.18252 shutdown 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 msgr2=0x55b88ac6ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc09ae30 tx=0x7f9ccc0bb190 comp rx=0 tx=0).stop 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 shutdown_connections 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b88ace17c0 0x55b88ace3be0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b88ac6aa80 0x55b88acdfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b88ac61ea0 0x55b88ac6ee50 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9cc5ffb640 0 client.18252 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:04.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 --2- 172.21.15.87:0/2388564979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b88ac61580 0x55b88ac6e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 >> 172.21.15.87:0/2388564979 conn(0x55b88ac5bcb0 msgr2=0x55b88ac5ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 shutdown_connections 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 1 -- 172.21.15.87:0/2388564979 wait complete. 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce720a740 20 client.18252 trim_cache size 0 max 16384 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:41:04.483 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 msgr2=0x7f9cc80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 0x7f9cc80a6380 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4007fd0 tx=0x7f9cd4007c60 comp rx=0 tx=0).stop 2024-04-22T13:41:04.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 msgr2=0x7f9cdc198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c970 tx=0x7f9ccc00ce30 comp rx=0 tx=0).stop 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 shutdown_connections 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cc80a3ec0 0x7f9cc80a6380 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9cdc19cef0 0x7f9cdc19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9cdc107270 0x7f9cdc198b70 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 --2- 172.21.15.87:0/2244192833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cdc1021a0 0x7f9cdc198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.479+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 >> 172.21.15.87:0/2244192833 conn(0x7f9cdc0793e0 msgr2=0x7f9cdc0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.480+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 shutdown_connections 2024-04-22T13:41:04.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.480+0000 7f9ce453d640 1 -- 172.21.15.87:0/2244192833 wait complete. 2024-04-22T13:41:04.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:41:04.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:41:04.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/2078892518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 msgr2=0x7fbd5810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/2078892518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd5810ee40 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009ca0 tx=0x7fbd5002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/2078892518 shutdown_connections 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/2078892518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd5810ee40 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/2078892518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd58102b50 0x7fbd58106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/2078892518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd581021a0 0x7fbd58102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/2078892518 >> 172.21.15.87:0/2078892518 conn(0x7fbd580793a0 msgr2=0x7fbd580797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/2078892518 shutdown_connections 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.591+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/2078892518 wait complete. 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 Processor -- start 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 -- start start 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 0x7fbd581986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd5819cf50 0x7fbd5819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581121e0 con 0x7fbd58107270 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbd58112060 con 0x7fbd5819cf50 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd5d7a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd58112360 con 0x7fbd581021a0 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59028/0 (socket says 172.21.15.87:59028) 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 -- 172.21.15.87:0/3303160744 learned_addr learned my addr 172.21.15.87:0/3303160744 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd56ffd640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 0x7fbd581986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd577fe640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd5819cf50 0x7fbd5819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 -- 172.21.15.87:0/3303160744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 msgr2=0x7fbd581986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 0x7fbd581986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 -- 172.21.15.87:0/3303160744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd5819cf50 msgr2=0x7fbd5819d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd5819cf50 0x7fbd5819d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 -- 172.21.15.87:0/3303160744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5819da40 con 0x7fbd58107270 2024-04-22T13:41:04.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd56ffd640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 0x7fbd581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.592+0000 7fbd4ffff640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c970 tx=0x7fbd4000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.593+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd40007bb0 con 0x7fbd58107270 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.593+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5819dd30 con 0x7fbd58107270 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.593+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd40007d50 con 0x7fbd58107270 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.593+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd40018730 con 0x7fbd58107270 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.593+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd581a54d0 con 0x7fbd58107270 2024-04-22T13:41:04.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.594+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd58069230 con 0x7fbd58107270 2024-04-22T13:41:04.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.595+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbd4000f990 con 0x7fbd58107270 2024-04-22T13:41:04.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.595+0000 7fbd54ff9640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 0x7fbd440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.596+0000 7fbd56ffd640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 0x7fbd440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.596+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fbd400d0c60 con 0x7fbd58107270 2024-04-22T13:41:04.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.596+0000 7fbd56ffd640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 0x7fbd440a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009b40 tx=0x7fbd48005e50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.599+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd40014030 con 0x7fbd58107270 2024-04-22T13:41:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:04 smithi120 ceph-mon[31101]: pgmap v286: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:41:04.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:04 smithi132 ceph-mon[31531]: pgmap v286: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:41:04.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:04 smithi087 ceph-mon[27999]: pgmap v286: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:41:04.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.814+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd581999e0 con 0x7fbd58107270 2024-04-22T13:41:04.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.815+0000 7fbd54ff9640 1 -- 172.21.15.87:0/3303160744 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fbd581999e0 con 0x7fbd58107270 2024-04-22T13:41:04.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:41:04.811 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:41:04.811 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 msgr2=0x7fbd440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 0x7fbd440a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009b40 tx=0x7fbd48005e50 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 msgr2=0x7fbd58198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c970 tx=0x7fbd4000ce30 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 shutdown_connections 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbd440a3d20 0x7fbd440a61e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd5819cf50 0x7fbd5819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbd58107270 0x7fbd58198c20 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 --2- 172.21.15.87:0/3303160744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd581021a0 0x7fbd581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 >> 172.21.15.87:0/3303160744 conn(0x7fbd580793a0 msgr2=0x7fbd580ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 shutdown_connections 2024-04-22T13:41:04.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.817+0000 7fbd5d7a4640 1 -- 172.21.15.87:0/3303160744 wait complete. 2024-04-22T13:41:04.824 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.928+0000 7f03d2888640 1 -- 172.21.15.87:0/772473222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 msgr2=0x7f03cc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.928+0000 7f03d2888640 1 --2- 172.21.15.87:0/772473222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc101470 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009900 tx=0x7f03bc02bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- 172.21.15.87:0/772473222 shutdown_connections 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- 172.21.15.87:0/772473222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc106160 0x7f03cc10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- 172.21.15.87:0/772473222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc101a40 0x7f03cc105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- 172.21.15.87:0/772473222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc101470 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- 172.21.15.87:0/772473222 >> 172.21.15.87:0/772473222 conn(0x7f03cc077f50 msgr2=0x7f03cc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- 172.21.15.87:0/772473222 shutdown_connections 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- 172.21.15.87:0/772473222 wait complete. 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 Processor -- start 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- start start 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc106160 0x7f03cc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 0x7f03cc1a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1110d0 con 0x7f03cc101090 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc110f50 con 0x7f03cc106160 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.929+0000 7f03d2888640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc111250 con 0x7f03cc19a0a0 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59046/0 (socket says 172.21.15.87:59046) 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 -- 172.21.15.87:0/612322293 learned_addr learned my addr 172.21.15.87:0/612322293 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03d0e25640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 0x7f03cc1a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc106160 0x7f03cc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 -- 172.21.15.87:0/612322293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 msgr2=0x7f03cc1a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 0x7f03cc1a4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 -- 172.21.15.87:0/612322293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc106160 msgr2=0x7f03cc199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc106160 0x7f03cc199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 -- 172.21.15.87:0/612322293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc1a4e50 con 0x7f03cc101090 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03d0e25640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 0x7f03cc1a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03cbfff640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005940 tx=0x7f03bc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03c97fa640 1 -- 172.21.15.87:0/612322293 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc043020 con 0x7f03cc101090 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03c97fa640 1 -- 172.21.15.87:0/612322293 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03bc02cbe0 con 0x7f03cc101090 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03cc1a50e0 con 0x7f03cc101090 2024-04-22T13:41:04.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03c97fa640 1 -- 172.21.15.87:0/612322293 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc003bf0 con 0x7f03cc101090 2024-04-22T13:41:04.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.930+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03cc1a5630 con 0x7f03cc101090 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.932+0000 7f03c97fa640 1 -- 172.21.15.87:0/612322293 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f03bc046020 con 0x7f03cc101090 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 Processor -- start 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 -- start start 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112977890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x5601129782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5601128d31f0 con 0x56011296e3b0 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5601128704b0 con 0x56011296ed60 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d547e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5601128cf1b0 con 0x560112977e60 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03c97fa640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59052/0 (socket says 172.21.15.87:59052) 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 -- 172.21.15.87:0/45828770 learned_addr learned my addr 172.21.15.87:0/45828770 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03cbfff640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112977890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x5601129782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03c97fa640 1 -- 172.21.15.87:0/612322293 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f03bc0f29c0 con 0x7f03cc101090 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 -- 172.21.15.87:0/45828770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 msgr2=0x5601129782c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x5601129782c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 -- 172.21.15.87:0/45828770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 msgr2=0x560112977890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112977890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03d0e25640 1 -- 172.21.15.87:0/45828770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601128621b0 con 0x56011296e3b0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.933+0000 7f03cbfff640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112977890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d0e25640 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f03c4015ef0 tx=0x7f03c4012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763761bb0306b28f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d0e25640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 0x7f03b40a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f03c40092f0 tx=0x7f03c401d000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03b27fc640 1 -- 172.21.15.87:0/45828770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4025070 con 0x56011296e3b0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03b27fc640 1 -- 172.21.15.87:0/45828770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03c400cbf0 con 0x56011296e3b0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03b27fc640 1 -- 172.21.15.87:0/45828770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c40054b0 con 0x56011296e3b0 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 -- 172.21.15.87:0/45828770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 msgr2=0x56011296e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f03c4015ef0 tx=0x7f03c4012760 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 -- 172.21.15.87:0/45828770 shutdown_connections 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x5601129782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112977890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 --2- 172.21.15.87:0/45828770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296e3b0 0x56011296e790 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 -- 172.21.15.87:0/45828770 >> 172.21.15.87:0/45828770 conn(0x560112968ba0 msgr2=0x560112968f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 -- 172.21.15.87:0/45828770 shutdown_connections 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.934+0000 7f03d547e740 1 -- 172.21.15.87:0/45828770 wait complete. 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 -- start start 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 Processor -- start 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x56011297e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56011297ed40 con 0x56011296ed60 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56011288c500 con 0x56011296e3b0 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56011297eec0 con 0x560112977e60 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03d547e740 10 client.0 _ll_register_callbacks cb 0x56011285ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59060/0 (socket says 172.21.15.87:59060) 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.935+0000 7f03cb7fe640 1 -- 172.21.15.87:0/1123266056 learned_addr learned my addr 172.21.15.87:0/1123266056 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cbfff640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x56011297e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03d0e25640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 -- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 msgr2=0x56011297bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 -- 172.21.15.87:0/1123266056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 msgr2=0x56011297e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x56011297e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 -- 172.21.15.87:0/1123266056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601125e53f0 con 0x56011296ed60 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03d0e25640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f03c000de60 tx=0x7f03c00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f9217334242e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0004220 con 0x56011296ed60 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f03c00043c0 con 0x56011296ed60 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03c400cbf0 con 0x56011296ed60 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f03c00035f0 con 0x56011296ed60 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03d547e740 10 client.18270 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:04.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5601126af7d0 con 0x56011296ed60 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03c400cbf0 con 0x56011296ed60 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.936+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f03c001b8c0 con 0x56011296ed60 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 10 client.18270 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 10 client.18270 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 20 client.18270 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56011298c120 0x56011298e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 4 client.18270 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560112872930 con 0x56011298c120 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03cbfff640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56011298c120 0x56011298e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03cbfff640 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56011298c120 0x56011298e550 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8dc371dc7dc51c32 server_cookie=5f22833c29234d2c in_seq=0 out_seq=0 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.937+0000 7f03ab7fe640 10 client.18270 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03ab7fe640 1 -- 172.21.15.87:0/1123266056 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560112872930 con 0x56011298c120 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03ab7fe640 10 client.18270 handle_command_reply: tid=0 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 client.18270 shutdown 2024-04-22T13:41:04.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 msgr2=0x56011297b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f03c000de60 tx=0x7f03c00117b0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 shutdown_connections 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56011298c120 0x56011298e550 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03ab7fe640 0 client.18270 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297b690 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 --2- 172.21.15.87:0/1123266056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x56011297e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 >> 172.21.15.87:0/1123266056 conn(0x560112968ba0 msgr2=0x56011296a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 shutdown_connections 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.938+0000 7f03d547e740 1 -- 172.21.15.87:0/1123266056 wait complete. 2024-04-22T13:41:04.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.939+0000 7f03d547e740 20 client.18270 trim_cache size 0 max 16384 2024-04-22T13:41:04.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.951+0000 7f03d547e740 1 Processor -- start 2024-04-22T13:41:04.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 -- start start 2024-04-22T13:41:04.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56011296e3b0 0x560112987c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112979250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1ae1e0 con 0x560112977e60 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1abab0 con 0x56011296ed60 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d547e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1a9920 con 0x56011296e3b0 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59066/0 (socket says 172.21.15.87:59066) 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 -- 172.21.15.87:0/3384755774 learned_addr learned my addr 172.21.15.87:0/3384755774 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112979250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 -- 172.21.15.87:0/3384755774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56011296e3b0 msgr2=0x560112987c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03cbfff640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56011296e3b0 0x560112987c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56011296e3b0 0x560112987c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 -- 172.21.15.87:0/3384755774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 msgr2=0x560112979250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112979250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 -- 172.21.15.87:0/3384755774 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56011296eb00 con 0x560112977e60 2024-04-22T13:41:04.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112979250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03d0e25640 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009e90 tx=0x7f03c4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d32d0e211cead9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03b1ffb640 1 -- 172.21.15.87:0/3384755774 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c402f070 con 0x560112977e60 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03b1ffb640 1 -- 172.21.15.87:0/3384755774 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03c4015a40 con 0x560112977e60 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.952+0000 7f03b1ffb640 1 -- 172.21.15.87:0/3384755774 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c4025040 con 0x560112977e60 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- 172.21.15.87:0/3384755774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 msgr2=0x56011297a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009e90 tx=0x7f03c4009ec0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- 172.21.15.87:0/3384755774 shutdown_connections 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560112977e60 0x56011297a670 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296ed60 0x560112979250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 --2- 172.21.15.87:0/3384755774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56011296e3b0 0x560112987c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- 172.21.15.87:0/3384755774 >> 172.21.15.87:0/3384755774 conn(0x560112968ba0 msgr2=0x560112979f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- 172.21.15.87:0/3384755774 shutdown_connections 2024-04-22T13:41:04.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- 172.21.15.87:0/3384755774 wait complete. 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.953+0000 7f03d547e740 1 -- start start 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 Processor -- start 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x560112984520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1a9f70 con 0x56011296ed60 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1ab3e0 con 0x56011296e3b0 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc1aa640 con 0x560112977e60 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 10 client.0 _ll_register_callbacks cb 0x7f03c000d9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59074/0 (socket says 172.21.15.87:59074) 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 -- 172.21.15.87:0/3371822637 learned_addr learned my addr 172.21.15.87:0/3371822637 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:04.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d0e25640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cbfff640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x560112984520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 -- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 msgr2=0x56011297d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 -- 172.21.15.87:0/3371822637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 msgr2=0x560112984520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x560112984520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 -- 172.21.15.87:0/3371822637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56011297dc40 con 0x56011296ed60 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cbfff640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x560112984520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d0e25640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03cb7fe640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006010 tx=0x7f03c000d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976a2b6a98ea7324 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0028070 con 0x56011296ed60 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f03c0020020 con 0x56011296ed60 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.954+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03c4015a40 con 0x56011296ed60 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f03c0005250 con 0x56011296ed60 2024-04-22T13:41:04.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 10 client.18276 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5601129f2620 con 0x56011296ed60 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03c4015a40 con 0x56011296ed60 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f03c0012880 con 0x56011296ed60 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 10 client.18276 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 10 client.18276 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 20 client.18276 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5601129f2ce0 0x5601129f5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 4 client.18276 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601128e0c10 con 0x5601129f2ce0 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.955+0000 7f03cbfff640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5601129f2ce0 0x5601129f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03cbfff640 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5601129f2ce0 0x5601129f5100 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d6c092429e85fd3 server_cookie=1dff24383b49c241 in_seq=0 out_seq=0 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03ab7fe640 10 client.18276 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:04.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03ab7fe640 1 -- 172.21.15.87:0/3371822637 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601128e0c10 con 0x5601129f2ce0 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03ab7fe640 10 client.18276 handle_command_reply: tid=0 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 client.18276 shutdown 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 msgr2=0x56011297cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006010 tx=0x7f03c000d550 comp rx=0 tx=0).stop 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 shutdown_connections 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5601129f2ce0 0x5601129f5100 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560112977e60 0x56011297d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56011296ed60 0x56011297cf20 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03ab7fe640 0 client.18276 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 --2- 172.21.15.87:0/3371822637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56011296e3b0 0x560112984520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 >> 172.21.15.87:0/3371822637 conn(0x560112968ba0 msgr2=0x560112967370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 shutdown_connections 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 1 -- 172.21.15.87:0/3371822637 wait complete. 2024-04-22T13:41:04.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.956+0000 7f03d547e740 20 client.18276 trim_cache size 0 max 16384 2024-04-22T13:41:04.960 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:04.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 msgr2=0x7f03b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 0x7f03b40a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f03c40092f0 tx=0x7f03c401d000 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 msgr2=0x7f03cc1995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005940 tx=0x7f03bc02c7e0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 shutdown_connections 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03cc19a0a0 0x7f03cc1a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03cc106160 0x7f03cc199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 --2- 172.21.15.87:0/612322293 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03cc101090 0x7f03cc1995d0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 >> 172.21.15.87:0/612322293 conn(0x7f03cc077f50 msgr2=0x7f03cc0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 shutdown_connections 2024-04-22T13:41:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:04.957+0000 7f03d2888640 1 -- 172.21.15.87:0/612322293 wait complete. 2024-04-22T13:41:04.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 -- 172.21.15.87:0/3873183499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 msgr2=0x7f1c44101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3873183499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c44101bb0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009900 tx=0x7f1c3802bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 -- 172.21.15.87:0/3873183499 shutdown_connections 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3873183499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 0x7f1c4410df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3873183499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c44102180 0x7f1c44105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3873183499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c44101bb0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 -- 172.21.15.87:0/3873183499 >> 172.21.15.87:0/3873183499 conn(0x7f1c440773e0 msgr2=0x7f1c440fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 -- 172.21.15.87:0/3873183499 shutdown_connections 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.068+0000 7f1c48e93640 1 -- 172.21.15.87:0/3873183499 wait complete. 2024-04-22T13:41:05.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 Processor -- start 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 -- start start 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 0x7f1c4419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c441987a0 0x7f1c44198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c44112400 con 0x7f1c441017d0 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c44112280 con 0x7f1c441987a0 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c44112580 con 0x7f1c44106380 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59092/0 (socket says 172.21.15.87:59092) 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 -- 172.21.15.87:0/3781977974 learned_addr learned my addr 172.21.15.87:0/3781977974 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 0x7f1c4419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 -- 172.21.15.87:0/3781977974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 msgr2=0x7f1c4419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 0x7f1c4419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 -- 172.21.15.87:0/3781977974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c441987a0 msgr2=0x7f1c44198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c441987a0 0x7f1c44198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c441987a0 0x7f1c44198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 -- 172.21.15.87:0/3781977974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c44199330 con 0x7f1c441017d0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005940 tx=0x7f1c380045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c337fe640 1 -- 172.21.15.87:0/3781977974 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38043020 con 0x7f1c441017d0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.069+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c441995c0 con 0x7f1c441017d0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.070+0000 7f1c337fe640 1 -- 172.21.15.87:0/3781977974 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c3802caf0 con 0x7f1c441017d0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.070+0000 7f1c337fe640 1 -- 172.21.15.87:0/3781977974 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38003af0 con 0x7f1c441017d0 2024-04-22T13:41:05.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.070+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c441a57f0 con 0x7f1c441017d0 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c337fe640 1 -- 172.21.15.87:0/3781977974 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1c38046020 con 0x7f1c441017d0 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 Processor -- start 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 -- start start 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68ed250 0x557fc68ed630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 0x557fc68fa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 0x557fc68fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557fc68521f0 con 0x557fc68edc00 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557fc67ef4b0 con 0x557fc68ed250 2024-04-22T13:41:05.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557fc684e1b0 con 0x557fc68fae10 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 0x557fc68fa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c337fe640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 0x7f1c2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 0x557fc68fa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59106/0 (socket says 172.21.15.87:59106) 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c4259c640 1 -- 172.21.15.87:0/213101187 learned_addr learned my addr 172.21.15.87:0/213101187 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.072+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 0x557fc68fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68ed250 0x557fc68ed630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 0x7f1c2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 -- 172.21.15.87:0/213101187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68ed250 msgr2=0x557fc68ed630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68ed250 0x557fc68ed630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c337fe640 1 -- 172.21.15.87:0/3781977974 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f1c380f28a0 con 0x7f1c441017d0 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 -- 172.21.15.87:0/213101187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 msgr2=0x557fc68fa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 0x557fc68fa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 -- 172.21.15.87:0/213101187 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fc67e11b0 con 0x557fc68fae10 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 0x7f1c2c0a61e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c005fd0 tx=0x7f1c3c009500 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 0x557fc68fd220 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1c44199af0 tx=0x7f1c3400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ec83fc8de86ac54 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c30ff9640 1 -- 172.21.15.87:0/213101187 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34015070 con 0x557fc68fae10 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c30ff9640 1 -- 172.21.15.87:0/213101187 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c340035f0 con 0x557fc68fae10 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.073+0000 7f1c30ff9640 1 -- 172.21.15.87:0/213101187 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c340116a0 con 0x557fc68fae10 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- 172.21.15.87:0/213101187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 msgr2=0x557fc68fd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 0x557fc68fd220 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1c44199af0 tx=0x7f1c3400f730 comp rx=0 tx=0).stop 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- 172.21.15.87:0/213101187 shutdown_connections 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68fae10 0x557fc68fd220 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68edc00 0x557fc68fa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/213101187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68ed250 0x557fc68ed630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- 172.21.15.87:0/213101187 >> 172.21.15.87:0/213101187 conn(0x557fc68e7ba0 msgr2=0x557fc68e7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- 172.21.15.87:0/213101187 shutdown_connections 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- 172.21.15.87:0/213101187 wait complete. 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 -- start start 2024-04-22T13:41:05.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.074+0000 7f1c4bb66740 1 Processor -- start 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68f7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x557fc68fdd40 con 0x557fc68ed250 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557fc680b500 con 0x557fc68fae10 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557fc68fdec0 con 0x557fc68edc00 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 10 client.0 _ll_register_callbacks cb 0x557fc67d9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59118/0 (socket says 172.21.15.87:59118) 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 -- 172.21.15.87:0/3284892128 learned_addr learned my addr 172.21.15.87:0/3284892128 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68f7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 -- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 msgr2=0x557fc68f7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68f7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 -- 172.21.15.87:0/3284892128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 msgr2=0x557fc68f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68f80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 -- 172.21.15.87:0/3284892128 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fc65643f0 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1c380badf0 tx=0x7f1c38009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1c64588590fb7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380be990 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f1c380053b0 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c340035f0 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1c380bf090 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 10 client.18288 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.075+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557fc662e7d0 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c340035f0 con 0x557fc68ed250 2024-04-22T13:41:05.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1c380a9110 con 0x557fc68ed250 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 10 client.18288 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 10 client.18288 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 20 client.18288 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc690afe0 0x557fc690d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 4 client.18288 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fc67f1930 con 0x557fc690afe0 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc690afe0 0x557fc690d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c29ffb640 10 client.18288 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.076+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc690afe0 0x557fc690d430 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6239e05d028c1ee2 server_cookie=d2914c0cb7ee0b3d in_seq=0 out_seq=0 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3284892128 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fc67f1930 con 0x557fc690afe0 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c29ffb640 10 client.18288 handle_command_reply: tid=0 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 client.18288 shutdown 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 msgr2=0x557fc68fa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1c380badf0 tx=0x7f1c38009880 comp rx=0 tx=0).stop 2024-04-22T13:41:05.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 shutdown_connections 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc690afe0 0x557fc690d430 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68f7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3284892128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc68fa6e0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 >> 172.21.15.87:0/3284892128 conn(0x557fc68e7ba0 msgr2=0x557fc68e98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c29ffb640 0 client.18288 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 shutdown_connections 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3284892128 wait complete. 2024-04-22T13:41:05.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.077+0000 7f1c4bb66740 20 client.18288 trim_cache size 0 max 16384 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.090+0000 7f1c4bb66740 1 Processor -- start 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.090+0000 7f1c4bb66740 1 -- start start 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.090+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68fa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441ae1c0 con 0x557fc68ed250 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441aba90 con 0x557fc68fae10 2024-04-22T13:41:05.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441a9900 con 0x557fc68edc00 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59132/0 (socket says 172.21.15.87:59132) 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 -- 172.21.15.87:0/423730975 learned_addr learned my addr 172.21.15.87:0/423730975 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68fa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 -- 172.21.15.87:0/423730975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 msgr2=0x557fc68fa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68fa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 -- 172.21.15.87:0/423730975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 msgr2=0x557fc68fa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 -- 172.21.15.87:0/423730975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fc68f6030 con 0x557fc68ed250 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68fa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c4259c640 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1c380f2040 tx=0x7f1c38045fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3751ce62da7c7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c2bfff640 1 -- 172.21.15.87:0/423730975 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380be990 con 0x557fc68ed250 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c2bfff640 1 -- 172.21.15.87:0/423730975 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c380c7600 con 0x557fc68ed250 2024-04-22T13:41:05.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.091+0000 7f1c2bfff640 1 -- 172.21.15.87:0/423730975 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380053b0 con 0x557fc68ed250 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- 172.21.15.87:0/423730975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 msgr2=0x557fc6906c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1c380f2040 tx=0x7f1c38045fb0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- 172.21.15.87:0/423730975 shutdown_connections 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68fae10 0x557fc68fa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68edc00 0x557fc68fa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/423730975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68ed250 0x557fc6906c40 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- 172.21.15.87:0/423730975 >> 172.21.15.87:0/423730975 conn(0x557fc68e7ba0 msgr2=0x557fc68e7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- 172.21.15.87:0/423730975 shutdown_connections 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- 172.21.15.87:0/423730975 wait complete. 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- start start 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 Processor -- start 2024-04-22T13:41:05.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68ed250 0x557fc69036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68edc00 0x557fc68fea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441a9f50 con 0x557fc68fae10 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441ab3c0 con 0x557fc68edc00 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c441aa620 con 0x557fc68ed250 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4bb66740 10 client.0 _ll_register_callbacks cb 0x7f1c38005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c42d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c42d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59134/0 (socket says 172.21.15.87:59134) 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c42d9d640 1 -- 172.21.15.87:0/3498342282 learned_addr learned my addr 172.21.15.87:0/3498342282 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.092+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68ed250 0x557fc69036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c41d9b640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68edc00 0x557fc68fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 -- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68ed250 msgr2=0x557fc69036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68ed250 0x557fc69036a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 -- 172.21.15.87:0/3498342282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68edc00 msgr2=0x557fc68fea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68edc00 0x557fc68fea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 -- 172.21.15.87:0/3498342282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fc6970520 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c42d9d640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c003c40 tx=0x7f1c3c0099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3d5309813bb117 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c024030 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f1c3c009040 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c380c7600 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1c3c030050 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 10 client.18294 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557fc6971630 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c380c7600 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1c3c00f510 con 0x557fc68fae10 2024-04-22T13:41:05.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 10 client.18294 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 10 client.18294 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 20 client.18294 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc6971de0 0x557fc6974200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 4 client.18294 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.093+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fc685fc10 con 0x557fc6971de0 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc6971de0 0x557fc6974200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c29ffb640 10 client.18294 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4259c640 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc6971de0 0x557fc6974200 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f71ef9344880ca2 server_cookie=52ca790c8a79bea2 in_seq=0 out_seq=0 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c29ffb640 1 -- 172.21.15.87:0/3498342282 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fc685fc10 con 0x557fc6971de0 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c29ffb640 10 client.18294 handle_command_reply: tid=0 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 client.18294 shutdown 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 msgr2=0x557fc68ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c003c40 tx=0x7f1c3c0099d0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 shutdown_connections 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x557fc6971de0 0x557fc6974200 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x557fc68fae10 0x557fc68ff150 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557fc68edc00 0x557fc68fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c29ffb640 0 client.18294 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 --2- 172.21.15.87:0/3498342282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557fc68ed250 0x557fc69036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 >> 172.21.15.87:0/3498342282 conn(0x557fc68e7ba0 msgr2=0x557fc68f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 shutdown_connections 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.094+0000 7f1c4bb66740 1 -- 172.21.15.87:0/3498342282 wait complete. 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c4bb66740 20 client.18294 trim_cache size 0 max 16384 2024-04-22T13:41:05.098 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:05.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 msgr2=0x7f1c2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 0x7f1c2c0a61e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c005fd0 tx=0x7f1c3c009500 comp rx=0 tx=0).stop 2024-04-22T13:41:05.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 msgr2=0x7f1c4419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005940 tx=0x7f1c380045e0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 shutdown_connections 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1c2c0a3d20 0x7f1c2c0a61e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c441987a0 0x7f1c44198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c44106380 0x7f1c4419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 --2- 172.21.15.87:0/3781977974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c441017d0 0x7f1c4419e630 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 >> 172.21.15.87:0/3781977974 conn(0x7f1c440773e0 msgr2=0x7f1c44104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 shutdown_connections 2024-04-22T13:41:05.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.095+0000 7f1c48e93640 1 -- 172.21.15.87:0/3781977974 wait complete. 2024-04-22T13:41:05.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:41:05.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/630984828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 msgr2=0x7f0ba01035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/630984828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01035c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0b90009900 tx=0x7f0b9002bd60 comp rx=0 tx=0).stop 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/630984828 shutdown_connections 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/630984828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba010c520 0x7f0ba010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/630984828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba0103b00 0x7f0ba010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/630984828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01035c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/630984828 >> 172.21.15.87:0/630984828 conn(0x7f0ba00771d0 msgr2=0x7f0ba00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/630984828 shutdown_connections 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.205+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/630984828 wait complete. 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 Processor -- start 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 -- start start 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba0103b00 0x7f0ba0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba010c520 0x7f0ba019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0112190 con 0x7f0ba01031e0 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0112010 con 0x7f0ba0103b00 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0ba4fa4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0112310 con 0x7f0ba010c520 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59150/0 (socket says 172.21.15.87:59150) 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4138282154 learned_addr learned my addr 172.21.15.87:0/4138282154 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba010c520 0x7f0ba019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.206+0000 7f0b97fff640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba0103b00 0x7f0ba0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4138282154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba010c520 msgr2=0x7f0ba019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba010c520 0x7f0ba019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4138282154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba0103b00 msgr2=0x7f0ba0198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba0103b00 0x7f0ba0198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4138282154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba019d6c0 con 0x7f0ba01031e0 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f0b90005940 tx=0x7f0b9002d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b977fe640 1 -- 172.21.15.87:0/4138282154 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b90043020 con 0x7f0ba01031e0 2024-04-22T13:41:05.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba019d950 con 0x7f0ba01031e0 2024-04-22T13:41:05.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b977fe640 1 -- 172.21.15.87:0/4138282154 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9002dd20 con 0x7f0ba01031e0 2024-04-22T13:41:05.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0b977fe640 1 -- 172.21.15.87:0/4138282154 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b90003b80 con 0x7f0ba01031e0 2024-04-22T13:41:05.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.207+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba01a5240 con 0x7f0ba01031e0 2024-04-22T13:41:05.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.209+0000 7f0b977fe640 1 -- 172.21.15.87:0/4138282154 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0b90046020 con 0x7f0ba01031e0 2024-04-22T13:41:05.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.209+0000 7f0ba7b97740 1 Processor -- start 2024-04-22T13:41:05.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 -- start start 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b977fe640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 0x7f0b8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e24d0 0x55d84b8e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d84b8471f0 con 0x55d84b8eba00 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d84b7e44b0 con 0x55d84b8e24d0 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d84b8431b0 con 0x55d84b8e2e80 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 0x7f0b8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b977fe640 1 -- 172.21.15.87:0/4138282154 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f0b900f2870 con 0x7f0ba01031e0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59160/0 (socket says 172.21.15.87:59160) 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 -- 172.21.15.87:0/782385752 learned_addr learned my addr 172.21.15.87:0/782385752 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e24d0 0x55d84b8e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b97fff640 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 -- 172.21.15.87:0/782385752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 msgr2=0x55d84b8eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 -- 172.21.15.87:0/782385752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e24d0 msgr2=0x55d84b8e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e24d0 0x55d84b8e28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.210+0000 7f0b9ed9d640 1 -- 172.21.15.87:0/782385752 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d84b7d61b0 con 0x55d84b8eba00 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 0x7f0b8c0a61e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0b980077f0 tx=0x7f0b98007e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0b98011c40 tx=0x7f0b9801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2beea755529dcf94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0b94ff9640 1 -- 172.21.15.87:0/782385752 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98022070 con 0x55d84b8eba00 2024-04-22T13:41:05.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0b94ff9640 1 -- 172.21.15.87:0/782385752 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9800fe70 con 0x55d84b8eba00 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0b94ff9640 1 -- 172.21.15.87:0/782385752 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b980054c0 con 0x55d84b8eba00 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0ba7b97740 1 -- 172.21.15.87:0/782385752 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 msgr2=0x55d84b8ede50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0b98011c40 tx=0x7f0b9801b000 comp rx=0 tx=0).stop 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0ba7b97740 1 -- 172.21.15.87:0/782385752 shutdown_connections 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8eba00 0x55d84b8ede50 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.211+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/782385752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e24d0 0x55d84b8e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- 172.21.15.87:0/782385752 >> 172.21.15.87:0/782385752 conn(0x55d84b8dcba0 msgr2=0x55d84b8dcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- 172.21.15.87:0/782385752 shutdown_connections 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- 172.21.15.87:0/782385752 wait complete. 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- start start 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 Processor -- start 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 0x55d84b8e8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8eba00 0x55d84b8e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d84b8ee910 con 0x55d84b8e24d0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d84b800500 con 0x55d84b8e2e80 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d84b8eea90 con 0x55d84b8eba00 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0ba7b97740 10 client.0 _ll_register_callbacks cb 0x55d84b7ceb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59168/0 (socket says 172.21.15.87:59168) 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4056118618 learned_addr learned my addr 172.21.15.87:0/4056118618 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8eba00 0x55d84b8e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.212+0000 7f0b97fff640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 0x55d84b8e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4056118618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8eba00 msgr2=0x55d84b8e8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8eba00 0x55d84b8e8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 msgr2=0x55d84b8e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 0x55d84b8e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4056118618 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d84b5593f0 con 0x55d84b8e24d0 2024-04-22T13:41:05.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b97fff640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 0x55d84b8e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f0b900be8f0 tx=0x7f0b900be380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb922b40848a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b900c2d10 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f0b900f0280 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b9800fe70 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0b900d71d0 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 10 client.18306 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d84b6237d0 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9800fe70 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0b900ddd80 con 0x55d84b8e24d0 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 10 client.18306 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 10 client.18306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 20 client.18306 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b8fbb40 0x55d84b8fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 4 client.18306 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:05.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.213+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d84b7e6930 con 0x55d84b8fbb40 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.214+0000 7f0b97fff640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b8fbb40 0x55d84b8fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.214+0000 7f0b85ffb640 10 client.18306 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.214+0000 7f0b97fff640 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b8fbb40 0x55d84b8fdf30 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44012450bdeafd1f server_cookie=2b13aa2c83c4a5d9 in_seq=0 out_seq=0 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0b85ffb640 1 -- 172.21.15.87:0/4056118618 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d84b7e6930 con 0x55d84b8fbb40 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0b85ffb640 10 client.18306 handle_command_reply: tid=0 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 client.18306 shutdown 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 msgr2=0x55d84b8eb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f0b900be8f0 tx=0x7f0b900be380 comp rx=0 tx=0).stop 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 shutdown_connections 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b8fbb40 0x55d84b8fdf30 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8eba00 0x55d84b8e8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8e2e80 0x55d84b8e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0b85ffb640 0 client.18306 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:05.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4056118618 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8eb2e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 >> 172.21.15.87:0/4056118618 conn(0x55d84b8dcba0 msgr2=0x55d84b8de8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 shutdown_connections 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.215+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4056118618 wait complete. 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.216+0000 7f0ba7b97740 20 client.18306 trim_cache size 0 max 16384 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.228+0000 7f0ba7b97740 1 Processor -- start 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 -- start start 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b8eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01b0280 con 0x55d84b8e24d0 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01adb50 con 0x55d84b8eba00 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01ab9c0 con 0x55d84b8e2e80 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59176/0 (socket says 172.21.15.87:59176) 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4087744479 learned_addr learned my addr 172.21.15.87:0/4087744479 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4087744479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 msgr2=0x55d84b8eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4087744479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 msgr2=0x55d84b8eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b8eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 -- 172.21.15.87:0/4087744479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d84b8fe2a0 con 0x55d84b8e24d0 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f0b900f01c0 tx=0x7f0b900bce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6865937a27d5c298 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b87fff640 1 -- 172.21.15.87:0/4087744479 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b900be4a0 con 0x55d84b8e24d0 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b87fff640 1 -- 172.21.15.87:0/4087744479 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b900d7020 con 0x55d84b8e24d0 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.229+0000 7f0b87fff640 1 -- 172.21.15.87:0/4087744479 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b900ddd40 con 0x55d84b8e24d0 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4087744479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 msgr2=0x55d84b8f7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f0b900f01c0 tx=0x7f0b900bce70 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4087744479 shutdown_connections 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b8eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e2e80 0x55d84b8eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/4087744479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e24d0 0x55d84b8f7850 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4087744479 >> 172.21.15.87:0/4087744479 conn(0x55d84b8dcba0 msgr2=0x55d84b8dbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4087744479 shutdown_connections 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- 172.21.15.87:0/4087744479 wait complete. 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- start start 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 Processor -- start 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e24d0 0x55d84b8eff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b9610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01ac010 con 0x55d84b8e2e80 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01ad480 con 0x55d84b8eba00 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01ac6e0 con 0x55d84b8e24d0 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0ba7b97740 10 client.0 _ll_register_callbacks cb 0x7f0b900be800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.230+0000 7f0b97fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59178/0 (socket says 172.21.15.87:59178) 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 -- 172.21.15.87:0/1801625552 learned_addr learned my addr 172.21.15.87:0/1801625552 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:05.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b9610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e24d0 0x55d84b8eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 -- 172.21.15.87:0/1801625552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e24d0 msgr2=0x55d84b8eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e24d0 0x55d84b8eff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 -- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 msgr2=0x55d84b9610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b9610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 -- 172.21.15.87:0/1801625552 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d84b9616a0 con 0x55d84b8e2e80 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b9ed9d640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b9610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b97fff640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f0b880058a0 tx=0x7f0b880039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a585abb341bdc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88021030 con 0x55d84b8e2e80 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f0b8801d420 con 0x55d84b8e2e80 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b900d7020 con 0x55d84b8e2e80 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0b8802b050 con 0x55d84b8e2e80 2024-04-22T13:41:05.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 10 client.18312 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d84b962a10 con 0x55d84b8e2e80 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b900d7020 con 0x55d84b8e2e80 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0b88011ce0 con 0x55d84b8e2e80 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 10 client.18312 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 20 client.18312 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.231+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b963210 0x55d84b965630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 4 client.18312 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d84b854c10 con 0x55d84b963210 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b963210 0x55d84b965630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b85ffb640 10 client.18312 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:05.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b9e59c640 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b963210 0x55d84b965630 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea13275176abd44c server_cookie=9114e2781fef3f0c in_seq=0 out_seq=0 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b85ffb640 1 -- 172.21.15.87:0/1801625552 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d84b854c10 con 0x55d84b963210 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b85ffb640 10 client.18312 handle_command_reply: tid=0 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 client.18312 shutdown 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 msgr2=0x55d84b8f04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f0b880058a0 tx=0x7f0b880039c0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 shutdown_connections 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55d84b963210 0x55d84b965630 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d84b8eba00 0x55d84b9610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d84b8e2e80 0x55d84b8f04a0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 --2- 172.21.15.87:0/1801625552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d84b8e24d0 0x55d84b8eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0b85ffb640 0 client.18312 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 >> 172.21.15.87:0/1801625552 conn(0x55d84b8dcba0 msgr2=0x55d84b8e32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 shutdown_connections 2024-04-22T13:41:05.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.232+0000 7f0ba7b97740 1 -- 172.21.15.87:0/1801625552 wait complete. 2024-04-22T13:41:05.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba7b97740 20 client.18312 trim_cache size 0 max 16384 2024-04-22T13:41:05.253 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 msgr2=0x7f0b8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 0x7f0b8c0a61e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0b980077f0 tx=0x7f0b98007e40 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 msgr2=0x7f0ba01986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f0b90005940 tx=0x7f0b9002d920 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 shutdown_connections 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0b8c0a3d20 0x7f0b8c0a61e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ba010c520 0x7f0ba019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ba0103b00 0x7f0ba0198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 --2- 172.21.15.87:0/4138282154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ba01031e0 0x7f0ba01986d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 >> 172.21.15.87:0/4138282154 conn(0x7f0ba00771d0 msgr2=0x7f0ba00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 shutdown_connections 2024-04-22T13:41:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:05.233+0000 7f0ba4fa4640 1 -- 172.21.15.87:0/4138282154 wait complete. 2024-04-22T13:41:05.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:05 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3303160744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:05.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:05 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3303160744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:05.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:05 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3303160744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:06.257 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:41:06.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:41:06.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:41:06.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 -- 172.21.15.87:0/2610533393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c102b40 msgr2=0x7fce6c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 --2- 172.21.15.87:0/2610533393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c102b40 0x7fce6c106b80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009900 tx=0x7fce5c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 -- 172.21.15.87:0/2610533393 shutdown_connections 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 --2- 172.21.15.87:0/2610533393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 0x7fce6c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 --2- 172.21.15.87:0/2610533393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c102b40 0x7fce6c106b80 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 --2- 172.21.15.87:0/2610533393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 0x7fce6c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.377+0000 7fce73406640 1 -- 172.21.15.87:0/2610533393 >> 172.21.15.87:0/2610533393 conn(0x7fce6c0793a0 msgr2=0x7fce6c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- 172.21.15.87:0/2610533393 shutdown_connections 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- 172.21.15.87:0/2610533393 wait complete. 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 Processor -- start 2024-04-22T13:41:06.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- start start 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 0x7fce6c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 0x7fce6c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c112180 con 0x7fce6c19cfe0 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c112000 con 0x7fce6c107260 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce73406640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c112300 con 0x7fce6c102190 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59196/0 (socket says 172.21.15.87:59196) 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce711a2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 0x7fce6c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.378+0000 7fce719a3640 1 -- 172.21.15.87:0/2163477372 learned_addr learned my addr 172.21.15.87:0/2163477372 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce709a1640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 0x7fce6c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 -- 172.21.15.87:0/2163477372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 msgr2=0x7fce6c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 0x7fce6c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 -- 172.21.15.87:0/2163477372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 msgr2=0x7fce6c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 0x7fce6c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 -- 172.21.15.87:0/2163477372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c19dad0 con 0x7fce6c19cfe0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce719a3640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c970 tx=0x7fce6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce627fc640 1 -- 172.21.15.87:0/2163477372 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64007bb0 con 0x7fce6c19cfe0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6c19ddc0 con 0x7fce6c19cfe0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce627fc640 1 -- 172.21.15.87:0/2163477372 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce64007d50 con 0x7fce6c19cfe0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce627fc640 1 -- 172.21.15.87:0/2163477372 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64004e60 con 0x7fce6c19cfe0 2024-04-22T13:41:06.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.379+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce6c1a5580 con 0x7fce6c19cfe0 2024-04-22T13:41:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.381+0000 7fce627fc640 1 -- 172.21.15.87:0/2163477372 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fce64005020 con 0x7fce6c19cfe0 2024-04-22T13:41:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 Processor -- start 2024-04-22T13:41:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 -- start start 2024-04-22T13:41:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce627fc640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 0x7fce580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b30cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b39840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5565d1a94f00 con 0x5565d1b39e10 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565d1a32590 con 0x5565d1b31290 2024-04-22T13:41:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce75ff6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5565d1a90ec0 con 0x5565d1b308e0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce709a1640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 0x7fce580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce627fc640 1 -- 172.21.15.87:0/2163477372 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fce640d0340 con 0x7fce6c19cfe0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59202/0 (socket says 172.21.15.87:59202) 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce719a3640 1 -- 172.21.15.87:0/3069070220 learned_addr learned my addr 172.21.15.87:0/3069070220 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce709a1640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b39840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.382+0000 7fce711a2640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b30cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 -- 172.21.15.87:0/3069070220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 msgr2=0x5565d1b30cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b30cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 -- 172.21.15.87:0/3069070220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 msgr2=0x5565d1b39840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b39840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 -- 172.21.15.87:0/3069070220 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d186f7e0 con 0x5565d1b39e10 2024-04-22T13:41:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce709a1640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b39840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce709a1640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 0x7fce580a61e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fce5c005ec0 tx=0x7fce5c004530 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce719a3640 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fce64098830 tx=0x7fce6409bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992c5a60f9142bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce577fe640 1 -- 172.21.15.87:0/3069070220 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640a06d0 con 0x5565d1b39e10 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce577fe640 1 -- 172.21.15.87:0/3069070220 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce640a4f90 con 0x5565d1b39e10 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.383+0000 7fce577fe640 1 -- 172.21.15.87:0/3069070220 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64005310 con 0x5565d1b39e10 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- 172.21.15.87:0/3069070220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 msgr2=0x5565d1b3c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fce64098830 tx=0x7fce6409bd30 comp rx=0 tx=0).stop 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- 172.21.15.87:0/3069070220 shutdown_connections 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b3c260 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b39840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3069070220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b30cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- 172.21.15.87:0/3069070220 >> 172.21.15.87:0/3069070220 conn(0x5565d1b2afb0 msgr2=0x5565d1b2b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- 172.21.15.87:0/3069070220 shutdown_connections 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- 172.21.15.87:0/3069070220 wait complete. 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 -- start start 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 Processor -- start 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.384+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b36e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b31290 0x5565d1b373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5565d1b3cd20 con 0x5565d1b39e10 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565d1a4e580 con 0x5565d1b308e0 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5565d1b3cea0 con 0x5565d1b31290 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 10 client.0 _ll_register_callbacks cb 0x5565d1a54650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59212/0 (socket says 172.21.15.87:59212) 2024-04-22T13:41:06.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 -- 172.21.15.87:0/2433194021 learned_addr learned my addr 172.21.15.87:0/2433194021 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce709a1640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b31290 0x5565d1b373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce711a2640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b36e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 -- 172.21.15.87:0/2433194021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b31290 msgr2=0x5565d1b373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b31290 0x5565d1b373c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 -- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 msgr2=0x5565d1b36e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b36e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 -- 172.21.15.87:0/2433194021 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d18cff70 con 0x5565d1b39e10 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce719a3640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fce6409c0d0 tx=0x7fce6409be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d851b2a456210b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640ba060 con 0x5565d1b39e10 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fce64005310 con 0x5565d1b39e10 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce640a4f90 con 0x5565d1b39e10 2024-04-22T13:41:06.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fce640bb250 con 0x5565d1b39e10 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 10 client.18324 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.385+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5565d1767170 con 0x5565d1b39e10 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce640a4f90 con 0x5565d1b39e10 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fce64098120 con 0x5565d1b39e10 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 10 client.18324 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 10 client.18324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 20 client.18324 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b4a220 0x5565d1b4c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 4 client.18324 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565d1aa2920 con 0x5565d1b4a220 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce711a2640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b4a220 0x5565d1b4c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce711a2640 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b4a220 0x5565d1b4c610 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57fbf94f5f6e75d server_cookie=e6c74e4bb2b1032e in_seq=0 out_seq=0 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.386+0000 7fce54ff9640 10 client.18324 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce54ff9640 1 -- 172.21.15.87:0/2433194021 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565d1aa2920 con 0x5565d1b4a220 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce54ff9640 10 client.18324 handle_command_reply: tid=0 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 client.18324 shutdown 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 msgr2=0x5565d1b37900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fce6409c0d0 tx=0x7fce6409be60 comp rx=0 tx=0).stop 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 shutdown_connections 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b4a220 0x5565d1b4c610 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b39e10 0x5565d1b37900 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b31290 0x5565d1b373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce54ff9640 0 client.18324 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 --2- 172.21.15.87:0/2433194021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b36e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.387+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 >> 172.21.15.87:0/2433194021 conn(0x5565d1b2afb0 msgr2=0x5565d1b2cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.388+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 shutdown_connections 2024-04-22T13:41:06.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.388+0000 7fce75ff6740 1 -- 172.21.15.87:0/2433194021 wait complete. 2024-04-22T13:41:06.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.388+0000 7fce75ff6740 20 client.18324 trim_cache size 0 max 16384 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 Processor -- start 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 -- start start 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b2afb0 0x5565d1b395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b4f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1b0420 con 0x5565d1b31290 2024-04-22T13:41:06.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1adcf0 con 0x5565d1b308e0 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce75ff6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1abb60 con 0x5565d1b2afb0 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59228/0 (socket says 172.21.15.87:59228) 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 -- 172.21.15.87:0/1336290895 learned_addr learned my addr 172.21.15.87:0/1336290895 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 -- 172.21.15.87:0/1336290895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b2afb0 msgr2=0x5565d1b395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b2afb0 0x5565d1b395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 -- 172.21.15.87:0/1336290895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 msgr2=0x5565d1b4f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b4f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.405+0000 7fce719a3640 1 -- 172.21.15.87:0/1336290895 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d1b2d180 con 0x5565d1b31290 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce719a3640 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fce640ce0d0 tx=0x7fce6409bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2271b24a231c9e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce56ffd640 1 -- 172.21.15.87:0/1336290895 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640b4070 con 0x5565d1b31290 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce56ffd640 1 -- 172.21.15.87:0/1336290895 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce640a06d0 con 0x5565d1b31290 2024-04-22T13:41:06.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce56ffd640 1 -- 172.21.15.87:0/1336290895 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6409b5f0 con 0x5565d1b31290 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 -- 172.21.15.87:0/1336290895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 msgr2=0x5565d1b4fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fce640ce0d0 tx=0x7fce6409bd30 comp rx=0 tx=0).stop 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 -- 172.21.15.87:0/1336290895 shutdown_connections 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b31290 0x5565d1b4fb70 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b308e0 0x5565d1b4f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 --2- 172.21.15.87:0/1336290895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b2afb0 0x5565d1b395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 -- 172.21.15.87:0/1336290895 >> 172.21.15.87:0/1336290895 conn(0x5565d1b319e0 msgr2=0x5565d1b45980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 -- 172.21.15.87:0/1336290895 shutdown_connections 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.406+0000 7fce75ff6740 1 -- 172.21.15.87:0/1336290895 wait complete. 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 -- start start 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 Processor -- start 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b4dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b4e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1ac1b0 con 0x5565d1b2afb0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1ad620 con 0x5565d1b31290 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c1ac880 con 0x5565d1b308e0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce75ff6740 10 client.0 _ll_register_callbacks cb 0x7fce68007910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59244/0 (socket says 172.21.15.87:59244) 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 -- 172.21.15.87:0/3538506481 learned_addr learned my addr 172.21.15.87:0/3538506481 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce719a3640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b4e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 -- 172.21.15.87:0/3538506481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 msgr2=0x5565d1b4dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce709a1640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b4dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b4dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.407+0000 7fce711a2640 1 -- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 msgr2=0x5565d1b4e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce711a2640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b4e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce711a2640 1 -- 172.21.15.87:0/3538506481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d1b4e610 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce709a1640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b4dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce719a3640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b4e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce711a2640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fce68010570 tx=0x7fce68010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da83916c7d7b622 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6800fe20 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce640a06d0 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7fce6801f160 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fce6802f050 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 10 client.18330 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5565d1bb12b0 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce640a06d0 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fce6801c970 con 0x5565d1b2afb0 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 10 client.18330 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:06.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 10 client.18330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 20 client.18330 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b39e10 0x5565d1bb3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 4 client.18330 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.408+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5565d1a9c790 con 0x5565d1b39e10 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.409+0000 7fce709a1640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b39e10 0x5565d1bb3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.409+0000 7fce54ff9640 10 client.18330 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.409+0000 7fce709a1640 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b39e10 0x5565d1bb3a70 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9de1fac446dedbe server_cookie=51d771f9679af92b in_seq=0 out_seq=0 2024-04-22T13:41:06.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce54ff9640 1 -- 172.21.15.87:0/3538506481 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5565d1a9c790 con 0x5565d1b39e10 2024-04-22T13:41:06.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce54ff9640 10 client.18330 handle_command_reply: tid=0 2024-04-22T13:41:06.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 client.18330 shutdown 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 msgr2=0x5565d1b4f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fce68010570 tx=0x7fce68010d40 comp rx=0 tx=0).stop 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 shutdown_connections 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5565d1b39e10 0x5565d1bb3a70 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565d1b31290 0x5565d1b4e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5565d1b308e0 0x5565d1b4dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.410+0000 7fce75ff6740 1 --2- 172.21.15.87:0/3538506481 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5565d1b2afb0 0x5565d1b4f740 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 >> 172.21.15.87:0/3538506481 conn(0x5565d1b319e0 msgr2=0x5565d1b2c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce54ff9640 0 client.18330 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 shutdown_connections 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce75ff6740 1 -- 172.21.15.87:0/3538506481 wait complete. 2024-04-22T13:41:06.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce75ff6740 20 client.18330 trim_cache size 0 max 16384 2024-04-22T13:41:06.413 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:06.413 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:41:06.413 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "f057da25-038e-4a21-9b92-c5e2d93c6282", 2024-04-22T13:41:06.413 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:41:06.413 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 msgr2=0x7fce580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 0x7fce580a61e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fce5c005ec0 tx=0x7fce5c004530 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 msgr2=0x7fce6c19d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c970 tx=0x7fce6400ce30 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 shutdown_connections 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fce580a3d20 0x7fce580a61e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce6c19cfe0 0x7fce6c19d490 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce6c107260 0x7fce6c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 --2- 172.21.15.87:0/2163477372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce6c102190 0x7fce6c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 >> 172.21.15.87:0/2163477372 conn(0x7fce6c0793a0 msgr2=0x7fce6c0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 shutdown_connections 2024-04-22T13:41:06.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.411+0000 7fce73406640 1 -- 172.21.15.87:0/2163477372 wait complete. 2024-04-22T13:41:06.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:41:06.418 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 -- 172.21.15.87:0/3411909636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14101a30 msgr2=0x7f5b14105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 --2- 172.21.15.87:0/3411909636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14101a30 0x7f5b14105a70 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009900 tx=0x7f5b0802bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 -- 172.21.15.87:0/3411909636 shutdown_connections 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 --2- 172.21.15.87:0/3411909636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14106150 0x7f5b1410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 --2- 172.21.15.87:0/3411909636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14101a30 0x7f5b14105a70 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 --2- 172.21.15.87:0/3411909636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b14101080 0x7f5b14101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 -- 172.21.15.87:0/3411909636 >> 172.21.15.87:0/3411909636 conn(0x7f5b14078000 msgr2=0x7f5b14078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.523+0000 7f5b194ba640 1 -- 172.21.15.87:0/3411909636 shutdown_connections 2024-04-22T13:41:06.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 -- 172.21.15.87:0/3411909636 wait complete. 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 Processor -- start 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 -- start start 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14101080 0x7f5b14198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b1419cff0 0x7f5b1419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141121d0 con 0x7f5b14106150 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14112050 con 0x7f5b14101080 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.524+0000 7f5b194ba640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14112350 con 0x7f5b1419cff0 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59258/0 (socket says 172.21.15.87:59258) 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 -- 172.21.15.87:0/1166733397 learned_addr learned my addr 172.21.15.87:0/1166733397 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14101080 0x7f5b14198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b137fe640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b1419cff0 0x7f5b1419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 -- 172.21.15.87:0/1166733397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b1419cff0 msgr2=0x7f5b1419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b1419cff0 0x7f5b1419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 -- 172.21.15.87:0/1166733397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14101080 msgr2=0x7f5b14198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14101080 0x7f5b14198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 -- 172.21.15.87:0/1166733397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1419dae0 con 0x7f5b14106150 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b127fc640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f5b08036040 tx=0x7f5b0802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5afbfff640 1 -- 172.21.15.87:0/1166733397 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08043020 con 0x7f5b14106150 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1419dd70 con 0x7f5b14106150 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5afbfff640 1 -- 172.21.15.87:0/1166733397 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0802cbe0 con 0x7f5b14106150 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5afbfff640 1 -- 172.21.15.87:0/1166733397 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08003bf0 con 0x7f5b14106150 2024-04-22T13:41:06.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.525+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b141a55c0 con 0x7f5b14106150 2024-04-22T13:41:06.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.527+0000 7f5afbfff640 1 -- 172.21.15.87:0/1166733397 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b08046020 con 0x7f5b14106150 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 Processor -- start 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 -- start start 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5afbfff640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 0x7f5afc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712686e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126879e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5571267d31e0 con 0x55712686eea0 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557126770390 con 0x55712686e580 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b1c183740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5571267cf1a0 con 0x557126877a80 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b137fe640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 0x7f5afc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5afbfff640 1 -- 172.21.15.87:0/1166733397 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f5b080f28d0 con 0x7f5b14106150 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59270/0 (socket says 172.21.15.87:59270) 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 -- 172.21.15.87:0/374965736 learned_addr learned my addr 172.21.15.87:0/374965736 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b137fe640 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126879e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712686e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 -- 172.21.15.87:0/374965736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 msgr2=0x557126879e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126879e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 -- 172.21.15.87:0/374965736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 msgr2=0x55712686e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712686e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b127fc640 1 -- 172.21.15.87:0/374965736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571265af7d0 con 0x55712686eea0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.528+0000 7f5b137fe640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 0x7f5afc0a61e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009fd0 tx=0x7f5b0c009da0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b127fc640 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5b080bb390 tx=0x7f5b080be5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2978dfa8d79b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5af97fa640 1 -- 172.21.15.87:0/374965736 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080be9c0 con 0x55712686eea0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5af97fa640 1 -- 172.21.15.87:0/374965736 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b080c2d70 con 0x55712686eea0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5af97fa640 1 -- 172.21.15.87:0/374965736 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080f02e0 con 0x55712686eea0 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 -- 172.21.15.87:0/374965736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 msgr2=0x557126877540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5b080bb390 tx=0x7f5b080be5c0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 -- 172.21.15.87:0/374965736 shutdown_connections 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126879e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x557126877540 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 --2- 172.21.15.87:0/374965736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712686e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 -- 172.21.15.87:0/374965736 >> 172.21.15.87:0/374965736 conn(0x557126868cb0 msgr2=0x557126869090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 -- 172.21.15.87:0/374965736 shutdown_connections 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.529+0000 7f5b1c183740 1 -- 172.21.15.87:0/374965736 wait complete. 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 -- start start 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 Processor -- start 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55712686eea0 0x557126874f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 0x5571268754d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55712687a630 con 0x55712686e580 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55712678c2f0 con 0x557126877a80 2024-04-22T13:41:06.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55712687a7b0 con 0x55712686eea0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b1c183740 10 client.0 _ll_register_callbacks cb 0x557126799580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59280/0 (socket says 172.21.15.87:59280) 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 -- 172.21.15.87:0/2542412894 learned_addr learned my addr 172.21.15.87:0/2542412894 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b127fc640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55712686eea0 0x557126874f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b137fe640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 0x5571268754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 -- 172.21.15.87:0/2542412894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55712686eea0 msgr2=0x557126874f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55712686eea0 0x557126874f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 -- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 msgr2=0x5571268754d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 0x5571268754d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 -- 172.21.15.87:0/2542412894 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571264e53f0 con 0x55712686e580 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b137fe640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 0x5571268754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400bab0 tx=0x7f5b0400bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2ece5643200338 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.530+0000 7f5af6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04011440 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8455+0+0 (secure 0 0 0) 0x7f5b040115e0 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b080c2d70 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5b040117e0 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 10 client.18342 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5571264a50f0 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b080c2d70 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5b04010a80 con 0x55712686e580 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 10 client.18342 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 10 client.18342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 20 client.18342 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5571268877f0 0x557126889c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 4 client.18342 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.531+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571267e0c00 con 0x5571268877f0 2024-04-22T13:41:06.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.532+0000 7f5b127fc640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5571268877f0 0x557126889c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.532+0000 7f5af6ff5640 10 client.18342 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.532+0000 7f5b127fc640 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5571268877f0 0x557126889c90 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a2ba58a86546817 server_cookie=e8d41e2ed8c645e6 in_seq=0 out_seq=0 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2542412894 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571267e0c00 con 0x5571268877f0 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5af6ff5640 10 client.18342 handle_command_reply: tid=0 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 client.18342 shutdown 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 msgr2=0x557126874a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400bab0 tx=0x7f5b0400bf70 comp rx=0 tx=0).stop 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 shutdown_connections 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5571268877f0 0x557126889c90 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557126877a80 0x5571268754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55712686eea0 0x557126874f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5af6ff5640 0 client.18342 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2542412894 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126874a50 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 >> 172.21.15.87:0/2542412894 conn(0x557126868cb0 msgr2=0x55712686aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 shutdown_connections 2024-04-22T13:41:06.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 1 -- 172.21.15.87:0/2542412894 wait complete. 2024-04-22T13:41:06.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.533+0000 7f5b1c183740 20 client.18342 trim_cache size 0 max 16384 2024-04-22T13:41:06.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:06 smithi087 ceph-mon[27999]: pgmap v287: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:41:06.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.551+0000 7f5b1c183740 1 Processor -- start 2024-04-22T13:41:06.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.551+0000 7f5b1c183740 1 -- start start 2024-04-22T13:41:06.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.551+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126877180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686eea0 0x5571268833a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b1c183740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141b03d0 con 0x55712686e580 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b1c183740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141adca0 con 0x55712686eea0 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b1c183740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141abb10 con 0x557126877a80 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:36292/0 (socket says 172.21.15.87:36292) 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 -- 172.21.15.87:0/837615107 learned_addr learned my addr 172.21.15.87:0/837615107 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b127fc640 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686eea0 0x5571268833a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 -- 172.21.15.87:0/837615107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686eea0 msgr2=0x5571268833a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686eea0 0x5571268833a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 -- 172.21.15.87:0/837615107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 msgr2=0x557126877180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126877180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.552+0000 7f5b137fe640 1 -- 172.21.15.87:0/837615107 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557126872fa0 con 0x557126877a80 2024-04-22T13:41:06.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.553+0000 7f5b137fe640 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0142f0 tx=0x7f5b0c01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a918270a3fb7cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.561+0000 7f5af8ff9640 1 -- 172.21.15.87:0/837615107 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c025070 con 0x557126877a80 2024-04-22T13:41:06.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.561+0000 7f5af8ff9640 1 -- 172.21.15.87:0/837615107 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c014320 con 0x557126877a80 2024-04-22T13:41:06.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.561+0000 7f5af8ff9640 1 -- 172.21.15.87:0/837615107 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c021ca0 con 0x557126877a80 2024-04-22T13:41:06.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 -- 172.21.15.87:0/837615107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 msgr2=0x557126867bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0142f0 tx=0x7f5b0c01e000 comp rx=0 tx=0).stop 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 -- 172.21.15.87:0/837615107 shutdown_connections 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x557126867bf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686eea0 0x5571268833a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 --2- 172.21.15.87:0/837615107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686e580 0x557126877180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.562+0000 7f5b1c183740 1 -- 172.21.15.87:0/837615107 >> 172.21.15.87:0/837615107 conn(0x557126868cb0 msgr2=0x5571268674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.563+0000 7f5b1c183740 1 -- 172.21.15.87:0/837615107 shutdown_connections 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.563+0000 7f5b1c183740 1 -- 172.21.15.87:0/837615107 wait complete. 2024-04-22T13:41:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.563+0000 7f5b1c183740 1 -- start start 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 Processor -- start 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712687b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x55712687bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141ac160 con 0x55712686eea0 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141ad5d0 con 0x55712686e580 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141ac830 con 0x557126877a80 2024-04-22T13:41:06.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.564+0000 7f5b1c183740 10 client.0 _ll_register_callbacks cb 0x7f5b04004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.566+0000 7f5b12ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712687b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.566+0000 7f5b137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:36294/0 (socket says 172.21.15.87:36294) 2024-04-22T13:41:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 -- 172.21.15.87:0/2719853111 learned_addr learned my addr 172.21.15.87:0/2719853111 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b127fc640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x55712687bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 -- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 msgr2=0x55712687b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712687b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 -- 172.21.15.87:0/2719853111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 msgr2=0x55712687bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:06.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x55712687bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.567+0000 7f5b137fe640 1 -- 172.21.15.87:0/2719853111 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557126906b30 con 0x557126877a80 2024-04-22T13:41:06.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.568+0000 7f5b137fe640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5778374bcd0d404a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:06.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.568+0000 7f5af6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-04-22T13:41:06.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.572+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c025040 con 0x557126877a80 2024-04-22T13:41:06.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.572+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8499+0+0 (secure 0 0 0) 0x7f5b0c012d00 con 0x557126877a80 2024-04-22T13:41:06.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.572+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b0c014320 con 0x557126877a80 2024-04-22T13:41:06.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.572+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5b0c004e10 con 0x557126877a80 2024-04-22T13:41:06.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.573+0000 7f5b1c183740 10 client.27163 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.573+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557126907ca0 con 0x557126877a80 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.573+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c014320 con 0x557126877a80 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.573+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5b0c00f050 con 0x557126877a80 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 10 client.27163 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 10 client.27163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 20 client.27163 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:06.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557126908340 0x55712690a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:06.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 4 client.27163 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:41:06.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.574+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5571267daa70 con 0x557126908340 2024-04-22T13:41:06.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.576+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557126908340 0x55712690a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:06.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5b12ffd640 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557126908340 0x55712690a760 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8219e3765ae3731a server_cookie=a731ba6246c2bf04 in_seq=0 out_seq=0 2024-04-22T13:41:06.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5af6ff5640 10 client.27163 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5af6ff5640 1 -- 172.21.15.87:0/2719853111 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5571267daa70 con 0x557126908340 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5af6ff5640 10 client.27163 handle_command_reply: tid=0 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5b1c183740 1 client.27163 shutdown 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 msgr2=0x5571268f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.577+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c004c70 comp rx=0 tx=0).stop 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 shutdown_connections 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x557126908340 0x55712690a760 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557126877a80 0x5571268f55e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55712686eea0 0x55712687bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 --2- 172.21.15.87:0/2719853111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712686e580 0x55712687b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 >> 172.21.15.87:0/2719853111 conn(0x557126868cb0 msgr2=0x557126867d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.578+0000 7f5af6ff5640 0 client.27163 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:06.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.579+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 shutdown_connections 2024-04-22T13:41:06.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.579+0000 7f5b1c183740 1 -- 172.21.15.87:0/2719853111 wait complete. 2024-04-22T13:41:06.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.579+0000 7f5b1c183740 20 client.27163 trim_cache size 0 max 16384 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "f057da25-038e-4a21-9b92-c5e2d93c6282": { 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "f057da25-038e-4a21-9b92-c5e2d93c6282", 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:06.576 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 msgr2=0x7f5afc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 0x7f5afc0a61e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009fd0 tx=0x7f5b0c009da0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 msgr2=0x7f5b14198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f5b08036040 tx=0x7f5b0802c7e0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 shutdown_connections 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f5afc0a3d20 0x7f5afc0a61e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b1419cff0 0x7f5b1419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b14106150 0x7f5b14198c70 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 --2- 172.21.15.87:0/1166733397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b14101080 0x7f5b14198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.580+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 >> 172.21.15.87:0/1166733397 conn(0x7f5b14078000 msgr2=0x7f5b140fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.581+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 shutdown_connections 2024-04-22T13:41:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:06.581+0000 7f5b194ba640 1 -- 172.21.15.87:0/1166733397 wait complete. 2024-04-22T13:41:06.594 INFO:tasks.cephfs.filesystem:scrub status for tag:f057da25-038e-4a21-9b92-c5e2d93c6282 - {'path': '/', 'tag': 'f057da25-038e-4a21-9b92-c5e2d93c6282', 'options': 'recursive,force'} 2024-04-22T13:41:06.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:06 smithi120 ceph-mon[31101]: pgmap v287: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:41:06.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:06 smithi132 ceph-mon[31531]: pgmap v287: 97 pgs: 97 active+clean; 37 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:41:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:07 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:41:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:07 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:07 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:41:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:07 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:07 smithi120 ceph-mon[31101]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T13:41:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:07 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:41:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:07 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:07 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:41:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:07 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:07 smithi132 ceph-mon[31531]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T13:41:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:07 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:41:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:07 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:07 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:41:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:07 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:07 smithi087 ceph-mon[27999]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T13:41:08.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:08 smithi120 ceph-mon[31101]: pgmap v289: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 31 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:08.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:08 smithi132 ceph-mon[31531]: pgmap v289: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 31 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:08.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:08 smithi087 ceph-mon[27999]: pgmap v289: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 31 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:10.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:10 smithi120 ceph-mon[31101]: pgmap v290: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 29 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:10.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:10 smithi132 ceph-mon[31531]: pgmap v290: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 29 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:10.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:10 smithi087 ceph-mon[27999]: pgmap v290: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 29 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 -- 172.21.15.87:0/1514180100 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 msgr2=0x7eff7c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 --2- 172.21.15.87:0/1514180100 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c10dd40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7eff74009ca0 tx=0x7eff7402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 -- 172.21.15.87:0/1514180100 shutdown_connections 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 --2- 172.21.15.87:0/1514180100 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c10dd40 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 --2- 172.21.15.87:0/1514180100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c101aa0 0x7eff7c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 --2- 172.21.15.87:0/1514180100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c1010f0 0x7eff7c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.498+0000 7eff810a5640 1 -- 172.21.15.87:0/1514180100 >> 172.21.15.87:0/1514180100 conn(0x7eff7c077fd0 msgr2=0x7eff7c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.499+0000 7eff810a5640 1 -- 172.21.15.87:0/1514180100 shutdown_connections 2024-04-22T13:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.499+0000 7eff810a5640 1 -- 172.21.15.87:0/1514180100 wait complete. 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.499+0000 7eff810a5640 1 Processor -- start 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.499+0000 7eff810a5640 1 -- start start 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1010f0 0x7eff7c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c19cee0 0x7eff7c19d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1110e0 con 0x7eff7c106170 2024-04-22T13:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c110f60 con 0x7eff7c1010f0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c111260 con 0x7eff7c19cee0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52414/0 (socket says 172.21.15.87:52414) 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 -- 172.21.15.87:0/2486025746 learned_addr learned my addr 172.21.15.87:0/2486025746 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c19cee0 0x7eff7c19d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1010f0 0x7eff7c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 -- 172.21.15.87:0/2486025746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c19cee0 msgr2=0x7eff7c19d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c19cee0 0x7eff7c19d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 -- 172.21.15.87:0/2486025746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1010f0 msgr2=0x7eff7c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1010f0 0x7eff7c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 -- 172.21.15.87:0/2486025746 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c19d9d0 con 0x7eff7c106170 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00e9c0 tx=0x7eff6c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff63fff640 1 -- 172.21.15.87:0/2486025746 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00cd80 con 0x7eff7c106170 2024-04-22T13:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff63fff640 1 -- 172.21.15.87:0/2486025746 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff6c0054d0 con 0x7eff7c106170 2024-04-22T13:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c19dcc0 con 0x7eff7c106170 2024-04-22T13:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff63fff640 1 -- 172.21.15.87:0/2486025746 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c010430 con 0x7eff7c106170 2024-04-22T13:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.500+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c1a54b0 con 0x7eff7c106170 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.502+0000 7eff63fff640 1 -- 172.21.15.87:0/2486025746 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7eff6c004fd0 con 0x7eff7c106170 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 Processor -- start 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 -- start start 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff63fff640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 0x7eff640a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 0x55bfc3bdf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3b448d0 con 0x55bfc3bdfb60 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3ae1e80 con 0x55bfc3bed4f0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff83d71740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3b40890 con 0x55bfc3be08d0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 0x7eff640a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52422/0 (socket says 172.21.15.87:52422) 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7a59c640 1 -- 172.21.15.87:0/2073351072 learned_addr learned my addr 172.21.15.87:0/2073351072 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff63fff640 1 -- 172.21.15.87:0/2486025746 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8499+0+0 (secure 0 0 0) 0x7eff6c0d0c10 con 0x7eff7c106170 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.503+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 0x55bfc3bdf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 -- 172.21.15.87:0/2073351072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 msgr2=0x55bfc3bdf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 0x55bfc3bdf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 -- 172.21.15.87:0/2073351072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 msgr2=0x55bfc3bef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 -- 172.21.15.87:0/2073351072 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc3b03320 con 0x55bfc3bdfb60 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 0x55bfc3bdf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 0x7eff640a6280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff74004af0 tx=0x7eff7402cf90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7eff6c099020 tx=0x7eff6c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90592b575b2a7a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff617fa640 1 -- 172.21.15.87:0/2073351072 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c09c990 con 0x55bfc3bdfb60 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff617fa640 1 -- 172.21.15.87:0/2073351072 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff6c0046c0 con 0x55bfc3bdfb60 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.504+0000 7eff617fa640 1 -- 172.21.15.87:0/2073351072 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0071c0 con 0x55bfc3bdfb60 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- 172.21.15.87:0/2073351072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 msgr2=0x55bfc3bdffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7eff6c099020 tx=0x7eff6c007850 comp rx=0 tx=0).stop 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- 172.21.15.87:0/2073351072 shutdown_connections 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bdffe0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- 172.21.15.87:0/2073351072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3be08d0 0x55bfc3bdf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- 172.21.15.87:0/2073351072 >> 172.21.15.87:0/2073351072 conn(0x55bfc3ba6c90 msgr2=0x55bfc3bda850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- 172.21.15.87:0/2073351072 shutdown_connections 2024-04-22T13:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- 172.21.15.87:0/2073351072 wait complete. 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- start start 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 Processor -- start 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bece20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3bf0220 con 0x55bfc3be08d0 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3acceb0 con 0x55bfc3bed4f0 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bfc3bf03a0 con 0x55bfc3bdfb60 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff83d71740 10 client.0 _ll_register_callbacks cb 0x55bfc3af9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52426/0 (socket says 172.21.15.87:52426) 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff7a59c640 1 -- 172.21.15.87:0/437019425 learned_addr learned my addr 172.21.15.87:0/437019425 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.505+0000 7eff7b59e640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 -- 172.21.15.87:0/437019425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 msgr2=0x55bfc3bece20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bece20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 -- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 msgr2=0x55bfc3bea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bea6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 -- 172.21.15.87:0/437019425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc3857210 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff7a59c640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000c00 tx=0x7eff6c09ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db06ea33d9fc0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0ce020 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bfc3b58880 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8499+0+0 (secure 0 0 0) 0x7eff6c09aec0 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7eff6c0c7940 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 10 client.18357 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bfc3923620 con 0x55bfc3be08d0 2024-04-22T13:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bfc3b58880 con 0x55bfc3be08d0 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7eff6c0c7bc0 con 0x55bfc3be08d0 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 10 client.18357 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 10 client.18357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 20 client.18357 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3bff720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 4 client.18357 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.506+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfc3ae43d0 con 0x55bfc3bfd330 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.507+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3bff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.507+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3bff720 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cbc1cafd10f1c79 server_cookie=dfa2bf1b6cd9c655 in_seq=0 out_seq=0 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.507+0000 7eff5eff5640 10 client.18357 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff5eff5640 1 -- 172.21.15.87:0/437019425 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfc3ae43d0 con 0x55bfc3bfd330 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff5eff5640 10 client.18357 handle_command_reply: tid=0 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 client.18357 shutdown 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 msgr2=0x55bfc3bea1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000c00 tx=0x7eff6c09ca00 comp rx=0 tx=0).stop 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 shutdown_connections 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3bff720 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bea1b0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff5eff5640 0 client.18357 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 --2- 172.21.15.87:0/437019425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 >> 172.21.15.87:0/437019425 conn(0x55bfc3ba6c90 msgr2=0x55bfc3bdc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 shutdown_connections 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 1 -- 172.21.15.87:0/437019425 wait complete. 2024-04-22T13:41:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.508+0000 7eff83d71740 20 client.18357 trim_cache size 0 max 16384 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.522+0000 7eff83d71740 1 Processor -- start 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.522+0000 7eff83d71740 1 -- start start 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bd9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bece20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1ae1e0 con 0x55bfc3be08d0 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1abab0 con 0x55bfc3bdfb60 2024-04-22T13:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff83d71740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1a9920 con 0x55bfc3bed4f0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52430/0 (socket says 172.21.15.87:52430) 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 -- 172.21.15.87:0/1357564563 learned_addr learned my addr 172.21.15.87:0/1357564563 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7b59e640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bd9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 -- 172.21.15.87:0/1357564563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 msgr2=0x55bfc3bece20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bece20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 -- 172.21.15.87:0/1357564563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 msgr2=0x55bfc3bd9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bd9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 -- 172.21.15.87:0/1357564563 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc3bdee30 con 0x55bfc3be08d0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7b59e640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bd9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff7a59c640 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0988a0 tx=0x7eff6c09ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfd69ba8e5ba072 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff60ff9640 1 -- 172.21.15.87:0/1357564563 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0ce050 con 0x55bfc3be08d0 2024-04-22T13:41:11.524 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff60ff9640 1 -- 172.21.15.87:0/1357564563 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff6c09d2c0 con 0x55bfc3be08d0 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.523+0000 7eff60ff9640 1 -- 172.21.15.87:0/1357564563 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0c7f30 con 0x55bfc3be08d0 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- 172.21.15.87:0/1357564563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 msgr2=0x55bfc3bec8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0988a0 tx=0x7eff6c09ca70 comp rx=0 tx=0).stop 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- 172.21.15.87:0/1357564563 shutdown_connections 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bed4f0 0x55bfc3bece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3be08d0 0x55bfc3bec8e0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- 172.21.15.87:0/1357564563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3bdfb60 0x55bfc3bd9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- 172.21.15.87:0/1357564563 >> 172.21.15.87:0/1357564563 conn(0x55bfc3ba6c90 msgr2=0x55bfc3bd9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- 172.21.15.87:0/1357564563 shutdown_connections 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- 172.21.15.87:0/1357564563 wait complete. 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- start start 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 Processor -- start 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3be3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3be08d0 0x55bfc3be4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1a9f70 con 0x55bfc3bed4f0 2024-04-22T13:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1ab3e0 con 0x55bfc3be08d0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c1aa640 con 0x55bfc3bdfb60 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.524+0000 7eff83d71740 10 client.0 _ll_register_callbacks cb 0x7eff6c0a5830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52444/0 (socket says 172.21.15.87:52444) 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 -- 172.21.15.87:0/2381125792 learned_addr learned my addr 172.21.15.87:0/2381125792 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3be3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7a59c640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3be08d0 0x55bfc3be4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 -- 172.21.15.87:0/2381125792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 msgr2=0x55bfc3be3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3be3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 -- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3be08d0 msgr2=0x55bfc3be4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3be08d0 0x55bfc3be4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 -- 172.21.15.87:0/2381125792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc3c00900 con 0x55bfc3bed4f0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff7b59e640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7eff74006010 tx=0x7eff7402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e89b0e20952d304 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74052060 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8499+0+0 (secure 0 0 0) 0x7eff74004450 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6c09d2c0 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7eff74039040 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff83d71740 10 client.18363 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bfc3c67f30 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.525+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff6c09d2c0 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7eff7403f070 con 0x55bfc3bed4f0 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 10 client.18363 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 10 client.18363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 20 client.18363 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3c6a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 4 client.18363 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfc3b522f0 con 0x55bfc3bfd330 2024-04-22T13:41:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3c6a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff5eff5640 10 client.18363 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.526+0000 7eff7ad9d640 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3c6a6a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d78b27ba8782f3 server_cookie=6294508e6419a068 in_seq=0 out_seq=0 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff5eff5640 1 -- 172.21.15.87:0/2381125792 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfc3b522f0 con 0x55bfc3bfd330 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff5eff5640 10 client.18363 handle_command_reply: tid=0 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 client.18363 shutdown 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 msgr2=0x55bfc3be4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7eff74006010 tx=0x7eff7402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 shutdown_connections 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bfc3bfd330 0x55bfc3c6a6a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bfc3bed4f0 0x55bfc3be4980 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc3be08d0 0x55bfc3be4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff5eff5640 0 client.18363 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 --2- 172.21.15.87:0/2381125792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bfc3bdfb60 0x55bfc3be3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 >> 172.21.15.87:0/2381125792 conn(0x55bfc3ba6c90 msgr2=0x55bfc3bf8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 shutdown_connections 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 1 -- 172.21.15.87:0/2381125792 wait complete. 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff83d71740 20 client.18363 trim_cache size 0 max 16384 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:41:11.530 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:41:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 msgr2=0x7eff640a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 0x7eff640a6280 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7eff74004af0 tx=0x7eff7402cf90 comp rx=0 tx=0).stop 2024-04-22T13:41:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 msgr2=0x7eff7c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.533+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00e9c0 tx=0x7eff6c00ee80 comp rx=0 tx=0).stop 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 shutdown_connections 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7eff640a3dc0 0x7eff640a6280 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff7c19cee0 0x7eff7c19d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff7c106170 0x7eff7c198c00 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 --2- 172.21.15.87:0/2486025746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff7c1010f0 0x7eff7c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 >> 172.21.15.87:0/2486025746 conn(0x7eff7c077fd0 msgr2=0x7eff7c0698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 shutdown_connections 2024-04-22T13:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:11.534+0000 7eff810a5640 1 -- 172.21.15.87:0/2486025746 wait complete. 2024-04-22T13:41:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:11 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:41:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:11 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:41:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:11 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:41:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:11 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:41:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:11 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:41:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:11 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:41:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:11 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:41:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:11 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:41:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:11 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:41:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:11 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:41:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:11 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:41:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:11 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:41:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:11 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:41:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:11 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:41:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:11 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:41:12.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:12 smithi120 ceph-mon[31101]: pgmap v291: 97 pgs: 97 active+clean; 27 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:12.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:12 smithi132 ceph-mon[31531]: pgmap v291: 97 pgs: 97 active+clean; 27 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:12.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:12 smithi087 ceph-mon[27999]: pgmap v291: 97 pgs: 97 active+clean; 27 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T13:41:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:14 smithi120 ceph-mon[31101]: pgmap v292: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-04-22T13:41:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:14 smithi132 ceph-mon[31531]: pgmap v292: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-04-22T13:41:14.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:14 smithi087 ceph-mon[27999]: pgmap v292: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-04-22T13:41:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:15 smithi120 ceph-mon[31101]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T13:41:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:15 smithi132 ceph-mon[31531]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T13:41:15.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:15 smithi087 ceph-mon[27999]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T13:41:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:16 smithi087 ceph-mon[27999]: pgmap v294: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.0 KiB/s wr, 10 op/s 2024-04-22T13:41:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:16 smithi120 ceph-mon[31101]: pgmap v294: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.0 KiB/s wr, 10 op/s 2024-04-22T13:41:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:16 smithi132 ceph-mon[31531]: pgmap v294: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.0 KiB/s wr, 10 op/s 2024-04-22T13:41:18.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:18 smithi120 ceph-mon[31101]: pgmap v295: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:18.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:18 smithi132 ceph-mon[31531]: pgmap v295: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:18.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:18 smithi087 ceph-mon[27999]: pgmap v295: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:20.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:20 smithi120 ceph-mon[31101]: pgmap v296: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:20.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:20 smithi132 ceph-mon[31531]: pgmap v296: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:20.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:20 smithi087 ceph-mon[27999]: pgmap v296: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-04-22T13:41:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:22 smithi120 ceph-mon[31101]: pgmap v297: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:41:22.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:22 smithi132 ceph-mon[31531]: pgmap v297: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:41:22.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:22 smithi087 ceph-mon[27999]: pgmap v297: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- 172.21.15.87:0/9469528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc001005e0 msgr2=0x7fbc00100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/9469528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc001005e0 0x7fbc00100a60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400aa40 tx=0x7fbbf402cf50 comp rx=0 tx=0).stop 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- 172.21.15.87:0/9469528 shutdown_connections 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/9469528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00100fa0 0x7fbc0010d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/9469528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc001005e0 0x7fbc00100a60 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/9469528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00106680 0x7fbc00106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- 172.21.15.87:0/9469528 >> 172.21.15.87:0/9469528 conn(0x7fbc00077390 msgr2=0x7fbc000fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- 172.21.15.87:0/9469528 shutdown_connections 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- 172.21.15.87:0/9469528 wait complete. 2024-04-22T13:41:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 Processor -- start 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.629+0000 7fbc05eb3640 1 -- start start 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 0x7fbc00196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00106680 0x7fbc0019acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00110030 con 0x7fbc00100fa0 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0010feb0 con 0x7fbc001005e0 2024-04-22T13:41:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001101b0 con 0x7fbc00106680 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54158/0 (socket says 172.21.15.87:54158) 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 -- 172.21.15.87:0/162250473 learned_addr learned my addr 172.21.15.87:0/162250473 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 0x7fbc00196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfffff640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00106680 0x7fbc0019acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 -- 172.21.15.87:0/162250473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00106680 msgr2=0x7fbc0019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00106680 0x7fbc0019acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 -- 172.21.15.87:0/162250473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 msgr2=0x7fbc00196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 0x7fbc00196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 -- 172.21.15.87:0/162250473 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0019b420 con 0x7fbc00100fa0 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 0x7fbc00196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400a340 tx=0x7fbbf40049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfcff9640 1 -- 172.21.15.87:0/162250473 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4044020 con 0x7fbc00100fa0 2024-04-22T13:41:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0019b6b0 con 0x7fbc00100fa0 2024-04-22T13:41:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.630+0000 7fbbfcff9640 1 -- 172.21.15.87:0/162250473 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf4035d00 con 0x7fbc00100fa0 2024-04-22T13:41:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.631+0000 7fbbfcff9640 1 -- 172.21.15.87:0/162250473 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4005250 con 0x7fbc00100fa0 2024-04-22T13:41:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.631+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc001a2fa0 con 0x7fbc00100fa0 2024-04-22T13:41:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.633+0000 7fbbfcff9640 1 -- 172.21.15.87:0/162250473 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbbf4047020 con 0x7fbc00100fa0 2024-04-22T13:41:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 Processor -- start 2024-04-22T13:41:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 -- start start 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cd5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55eb93c3a8d0 con 0x55eb93cd5b60 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eb93bd7e80 con 0x55eb93cd68d0 2024-04-22T13:41:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eb93c36890 con 0x55eb93ce12f0 2024-04-22T13:41:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbfcff9640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 0x7fbbe80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54172/0 (socket says 172.21.15.87:54172) 2024-04-22T13:41:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 -- 172.21.15.87:0/146724308 learned_addr learned my addr 172.21.15.87:0/146724308 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbfffff640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 0x7fbbe80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbfffff640 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cd5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbfcff9640 1 -- 172.21.15.87:0/162250473 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbbf40f3800 con 0x7fbc00100fa0 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 -- 172.21.15.87:0/146724308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 msgr2=0x55eb93ce36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 -- 172.21.15.87:0/146724308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 msgr2=0x55eb93cd5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cd5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.634+0000 7fbbff7fe640 1 -- 172.21.15.87:0/146724308 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb93bf9320 con 0x55eb93cd5b60 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0010370 tx=0x7fbbf0010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ba31f70f6534ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbbfffff640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 0x7fbbe80a6280 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8010570 tx=0x7fbbf8009570 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbbe5ffb640 1 -- 172.21.15.87:0/146724308 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0016070 con 0x55eb93cd5b60 2024-04-22T13:41:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbbe5ffb640 1 -- 172.21.15.87:0/146724308 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf0003e20 con 0x55eb93cd5b60 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbbe5ffb640 1 -- 172.21.15.87:0/146724308 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00047d0 con 0x55eb93cd5b60 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbc08b7c740 1 -- 172.21.15.87:0/146724308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 msgr2=0x55eb93cd5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.635+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0010370 tx=0x7fbbf0010830 comp rx=0 tx=0).stop 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- 172.21.15.87:0/146724308 shutdown_connections 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cd5fe0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/146724308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cd5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- 172.21.15.87:0/146724308 >> 172.21.15.87:0/146724308 conn(0x55eb93c9cc90 msgr2=0x55eb93cd0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- 172.21.15.87:0/146724308 shutdown_connections 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- 172.21.15.87:0/146724308 wait complete. 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- start start 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 Processor -- start 2024-04-22T13:41:24.650 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cddfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93cde530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55eb93ce4020 con 0x55eb93cd5b60 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eb93bc2eb0 con 0x55eb93cd68d0 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eb93ce41a0 con 0x55eb93ce12f0 2024-04-22T13:41:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbc08b7c740 10 client.0 _ll_register_callbacks cb 0x55eb93befb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.636+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54174/0 (socket says 172.21.15.87:54174) 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 -- 172.21.15.87:0/996711737 learned_addr learned my addr 172.21.15.87:0/996711737 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cddfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 -- 172.21.15.87:0/996711737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 msgr2=0x55eb93cde530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbfffff640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93cde530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93cde530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 -- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 msgr2=0x55eb93cddfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cddfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 -- 172.21.15.87:0/996711737 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb9394d210 con 0x55eb93cd5b60 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cddfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0010d30 tx=0x7fbbf0004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a066b7d6eddbe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0029060 con 0x55eb93cd5b60 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbbf0003970 con 0x55eb93cd5b60 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf0003e20 con 0x55eb93cd5b60 2024-04-22T13:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbbf0016070 con 0x55eb93cd5b60 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbc08b7c740 10 client.18375 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.637+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eb93a19620 con 0x55eb93cd5b60 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf0003e20 con 0x55eb93cd5b60 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbbf000f040 con 0x55eb93cd5b60 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 10 client.18375 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 10 client.18375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 20 client.18375 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93cf1260 0x55eb93cf3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 4 client.18375 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb93c4e880 con 0x55eb93cf1260 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.638+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93cf1260 0x55eb93cf3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93cf1260 0x55eb93cf3650 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f54f826e98d083f server_cookie=47a192106273fa58 in_seq=0 out_seq=0 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbbdeffd640 10 client.18375 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbbdeffd640 1 -- 172.21.15.87:0/996711737 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb93c4e880 con 0x55eb93cf1260 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbbdeffd640 10 client.18375 handle_command_reply: tid=0 2024-04-22T13:41:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbc08b7c740 1 client.18375 shutdown 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 msgr2=0x55eb93ce0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.639+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0010d30 tx=0x7fbbf0004230 comp rx=0 tx=0).stop 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 shutdown_connections 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93cf1260 0x55eb93cf3650 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93cde530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93cddfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/996711737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93ce0c40 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbbdeffd640 0 client.18375 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 >> 172.21.15.87:0/996711737 conn(0x55eb93c9cc90 msgr2=0x55eb93cd25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 shutdown_connections 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 1 -- 172.21.15.87:0/996711737 wait complete. 2024-04-22T13:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.640+0000 7fbc08b7c740 20 client.18375 trim_cache size 0 max 16384 2024-04-22T13:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.654+0000 7fbc08b7c740 1 Processor -- start 2024-04-22T13:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 -- start start 2024-04-22T13:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93ce0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001ae000 con 0x55eb93cd5b60 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001ab8d0 con 0x55eb93cd68d0 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001a9740 con 0x55eb93ce12f0 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54188/0 (socket says 172.21.15.87:54188) 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 -- 172.21.15.87:0/1942700163 learned_addr learned my addr 172.21.15.87:0/1942700163 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93ce0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 -- 172.21.15.87:0/1942700163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 msgr2=0x55eb93ce0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbfffff640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 -- 172.21.15.87:0/1942700163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 msgr2=0x55eb93ce0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93ce0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.663 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 -- 172.21.15.87:0/1942700163 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb93cd4e30 con 0x55eb93cd5b60 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93ce0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbfffff640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0012e90 tx=0x7fbbf0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dac2a38cd6011a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.655+0000 7fbbe57fa640 1 -- 172.21.15.87:0/1942700163 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0029070 con 0x55eb93cd5b60 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbbe57fa640 1 -- 172.21.15.87:0/1942700163 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf00042a0 con 0x55eb93cd5b60 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbbe57fa640 1 -- 172.21.15.87:0/1942700163 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0009c30 con 0x55eb93cd5b60 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- 172.21.15.87:0/1942700163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 msgr2=0x55eb93cecf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0012e90 tx=0x7fbbf0005d30 comp rx=0 tx=0).stop 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- 172.21.15.87:0/1942700163 shutdown_connections 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93ce12f0 0x55eb93ce0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93cd68d0 0x55eb93ce0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/1942700163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cecf20 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- 172.21.15.87:0/1942700163 >> 172.21.15.87:0/1942700163 conn(0x55eb93c9cc90 msgr2=0x55eb93cd0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.664 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- 172.21.15.87:0/1942700163 shutdown_connections 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- 172.21.15.87:0/1942700163 wait complete. 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.656+0000 7fbc08b7c740 1 -- start start 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 Processor -- start 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 0x55eb93cdb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 0x55eb93cdb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001a9d90 con 0x55eb93cd5b60 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001ab200 con 0x55eb93ce12f0 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001aa460 con 0x55eb93cd68d0 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbc08b7c740 10 client.0 _ll_register_callbacks cb 0x7fbbf4035eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54196/0 (socket says 172.21.15.87:54196) 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 -- 172.21.15.87:0/601939719 learned_addr learned my addr 172.21.15.87:0/601939719 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbfffff640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 0x55eb93cdb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.665 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 0x55eb93cdb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 -- 172.21.15.87:0/601939719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 msgr2=0x55eb93cdb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 0x55eb93cdb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 -- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 msgr2=0x55eb93cdb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 0x55eb93cdb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 -- 172.21.15.87:0/601939719 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb93d56620 con 0x55eb93cd5b60 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbfffff640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 0x55eb93cdb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 0x55eb93cdb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbff7fe640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0000f80 tx=0x7fbbf001a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d27cac4ca5c282 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.657+0000 7fbbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0029070 con 0x55eb93cd5b60 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbbf0003b80 con 0x55eb93cd5b60 2024-04-22T13:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf00042a0 con 0x55eb93cd5b60 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbbf0003d80 con 0x55eb93cd5b60 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 10 client.18381 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eb93d57820 con 0x55eb93cd5b60 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf00042a0 con 0x55eb93cd5b60 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbbf0012a10 con 0x55eb93cd5b60 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 10 client.18381 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 10 client.18381 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 20 client.18381 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93d57dd0 0x55eb93d5a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 4 client.18381 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb93bda3d0 con 0x55eb93d57dd0 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.658+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93d57dd0 0x55eb93d5a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.659+0000 7fbbdeffd640 10 client.18381 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.659+0000 7fbbfeffd640 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93d57dd0 0x55eb93d5a1f0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94a847e695722028 server_cookie=5772b6f37ac8dd2f in_seq=0 out_seq=0 2024-04-22T13:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbbdeffd640 1 -- 172.21.15.87:0/601939719 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eb93bda3d0 con 0x55eb93d57dd0 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbbdeffd640 10 client.18381 handle_command_reply: tid=0 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 client.18381 shutdown 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 msgr2=0x55eb93cdae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0000f80 tx=0x7fbbf001a6c0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 shutdown_connections 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eb93d57dd0 0x55eb93d5a1f0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb93ce12f0 0x55eb93cdb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb93cd68d0 0x55eb93cdb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 --2- 172.21.15.87:0/601939719 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eb93cd5b60 0x55eb93cdae10 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbbdeffd640 0 client.18381 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.664+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 >> 172.21.15.87:0/601939719 conn(0x55eb93c9cc90 msgr2=0x55eb93cd1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 shutdown_connections 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc08b7c740 1 -- 172.21.15.87:0/601939719 wait complete. 2024-04-22T13:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc08b7c740 20 client.18381 trim_cache size 0 max 16384 2024-04-22T13:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 msgr2=0x7fbbe80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 0x7fbbe80a6280 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8010570 tx=0x7fbbf8009570 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 msgr2=0x7fbc00196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400a340 tx=0x7fbbf40049e0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 shutdown_connections 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbbe80a3dc0 0x7fbbe80a6280 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc00106680 0x7fbc0019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc00100fa0 0x7fbc00196970 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 --2- 172.21.15.87:0/162250473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc001005e0 0x7fbc00196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 >> 172.21.15.87:0/162250473 conn(0x7fbc00077390 msgr2=0x7fbc000fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 shutdown_connections 2024-04-22T13:41:24.671 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:24.665+0000 7fbc05eb3640 1 -- 172.21.15.87:0/162250473 wait complete. 2024-04-22T13:41:24.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:24 smithi120 ceph-mon[31101]: pgmap v298: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:24.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:24 smithi132 ceph-mon[31531]: pgmap v298: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:24.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:24 smithi087 ceph-mon[27999]: pgmap v298: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:25.727 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-22T13:41:25.727 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:41:25.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- 172.21.15.87:0/1807769750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7734101b60 msgr2=0x7f773410dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:25.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 --2- 172.21.15.87:0/1807769750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7734101b60 0x7f773410dc90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f772c009cd0 tx=0x7f772c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- 172.21.15.87:0/1807769750 shutdown_connections 2024-04-22T13:41:25.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 --2- 172.21.15.87:0/1807769750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7734101b60 0x7f773410dc90 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 --2- 172.21.15.87:0/1807769750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77341011a0 0x7f7734101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 --2- 172.21.15.87:0/1807769750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f7734106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- 172.21.15.87:0/1807769750 >> 172.21.15.87:0/1807769750 conn(0x7f7734077e30 msgr2=0x7f7734078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- 172.21.15.87:0/1807769750 shutdown_connections 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- 172.21.15.87:0/1807769750 wait complete. 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 Processor -- start 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.843+0000 7f773a406640 1 -- start start 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 0x7f7734198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:25.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f773419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f77341110c0 con 0x7f77341011a0 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7734110f40 con 0x7f7734101b60 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f773a406640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7734111240 con 0x7f7734105e30 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54214/0 (socket says 172.21.15.87:54214) 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 -- 172.21.15.87:0/1836994320 learned_addr learned my addr 172.21.15.87:0/1836994320 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f77389a3640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f773419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 -- 172.21.15.87:0/1836994320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 msgr2=0x7f773419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f77337fe640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 0x7f7734198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f773419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 -- 172.21.15.87:0/1836994320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 msgr2=0x7f7734198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:25.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 0x7f7734198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 -- 172.21.15.87:0/1836994320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773419d650 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f77389a3640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f773419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f77337fe640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 0x7f7734198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f7733fff640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f772800d950 tx=0x7f772800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.844+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728017030 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.845+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7728004020 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.845+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773419d940 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.845+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280047a0 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.845+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77341a51d0 con 0x7f77341011a0 2024-04-22T13:41:25.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.845+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7734068de0 con 0x7f77341011a0 2024-04-22T13:41:25.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.847+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7728009a30 con 0x7f77341011a0 2024-04-22T13:41:25.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.847+0000 7f77317fa640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 0x7f771c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:25.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.847+0000 7f77337fe640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 0x7f771c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:25.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.847+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f77280d0930 con 0x7f77341011a0 2024-04-22T13:41:25.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.848+0000 7f77337fe640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 0x7f771c0a61e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7734104ee0 tx=0x7f7724005e50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:25.851+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f772809d010 con 0x7f77341011a0 2024-04-22T13:41:26.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.066+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f773410a3c0 con 0x7f77341011a0 2024-04-22T13:41:26.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.067+0000 7f77317fa640 1 -- 172.21.15.87:0/1836994320 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f772809c910 con 0x7f77341011a0 2024-04-22T13:41:26.063 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:41:26.063 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:41:26.064 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:41:26.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 msgr2=0x7f771c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 0x7f771c0a61e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7734104ee0 tx=0x7f7724005e50 comp rx=0 tx=0).stop 2024-04-22T13:41:26.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 msgr2=0x7f7734198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f772800d950 tx=0x7f772800de10 comp rx=0 tx=0).stop 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 shutdown_connections 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f771c0a3d20 0x7f771c0a61e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7734105e30 0x7f773419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7734101b60 0x7f7734198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 --2- 172.21.15.87:0/1836994320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77341011a0 0x7f7734198660 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 >> 172.21.15.87:0/1836994320 conn(0x7f7734077e30 msgr2=0x7f7734069360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 shutdown_connections 2024-04-22T13:41:26.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.069+0000 7f773a406640 1 -- 172.21.15.87:0/1836994320 wait complete. 2024-04-22T13:41:26.076 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 70 --await' 2024-04-22T13:41:26.076 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 70 --await 2024-04-22T13:41:26.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3987634330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 msgr2=0x7fe648103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3987634330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe648103590 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fe638009900 tx=0x7fe63802bcd0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3987634330 shutdown_connections 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3987634330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64810c580 0x7fe64810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3987634330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe648103ad0 0x7fe64810bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3987634330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe648103590 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3987634330 >> 172.21.15.87:0/3987634330 conn(0x7fe6480771d0 msgr2=0x7fe6480775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3987634330 shutdown_connections 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.189+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3987634330 wait complete. 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 Processor -- start 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 -- start start 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 0x7fe64819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 0x7fe6481985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481121d0 con 0x7fe6481031b0 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe648112050 con 0x7fe648103ad0 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe64e7e2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe648112350 con 0x7fe64810c580 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54230/0 (socket says 172.21.15.87:54230) 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe647fff640 1 -- 172.21.15.87:0/3491680215 learned_addr learned my addr 172.21.15.87:0/3491680215 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.190+0000 7fe6477fe640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 0x7fe64819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 -- 172.21.15.87:0/3491680215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 msgr2=0x7fe6481985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 0x7fe6481985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 0x7fe6481985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 -- 172.21.15.87:0/3491680215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 msgr2=0x7fe64819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 0x7fe64819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 -- 172.21.15.87:0/3491680215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe648198d60 con 0x7fe6481031b0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe6477fe640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 0x7fe64819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 0x7fe6481985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe647fff640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fe638005940 tx=0x7fe6380045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe6457fa640 1 -- 172.21.15.87:0/3491680215 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638043020 con 0x7fe6481031b0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe648198ff0 con 0x7fe6481031b0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe6457fa640 1 -- 172.21.15.87:0/3491680215 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63802dba0 con 0x7fe6481031b0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe6457fa640 1 -- 172.21.15.87:0/3491680215 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638003930 con 0x7fe6481031b0 2024-04-22T13:41:26.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.191+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6481a52f0 con 0x7fe6481031b0 2024-04-22T13:41:26.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6457fa640 1 -- 172.21.15.87:0/3491680215 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe638046020 con 0x7fe6481031b0 2024-04-22T13:41:26.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 Processor -- start 2024-04-22T13:41:26.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 -- start start 2024-04-22T13:41:26.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b597da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55746b4f80b0 con 0x55746b5a0ec0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55746b4a9680 con 0x55746b5982e0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.193+0000 7fe6513cb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55746b4b5c60 con 0x55746b5979c0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54240/0 (socket says 172.21.15.87:54240) 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 -- 172.21.15.87:0/3825336061 learned_addr learned my addr 172.21.15.87:0/3825336061 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6457fa640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 0x7fe6300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe647fff640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b597da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 -- 172.21.15.87:0/3825336061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 msgr2=0x55746b597da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b597da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 -- 172.21.15.87:0/3825336061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 msgr2=0x55746b5a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 -- 172.21.15.87:0/3825336061 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55746b2dac90 con 0x55746b5a0ec0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b597da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 0x7fe6300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6457fa640 1 -- 172.21.15.87:0/3491680215 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe6380f2810 con 0x7fe6481031b0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe6477fe640 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009cb0 tx=0x7fe63c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03fc18412773e22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe62e7fc640 1 -- 172.21.15.87:0/3825336061 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c012450 con 0x55746b5a0ec0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe62e7fc640 1 -- 172.21.15.87:0/3825336061 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63c012aa0 con 0x55746b5a0ec0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.194+0000 7fe62e7fc640 1 -- 172.21.15.87:0/3825336061 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00fc00 con 0x55746b5a0ec0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- 172.21.15.87:0/3825336061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 msgr2=0x55746b5a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009cb0 tx=0x7fe63c0117b0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 0x7fe6300a61e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe640005c40 tx=0x7fe640004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- 172.21.15.87:0/3825336061 shutdown_connections 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b5a32b0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 --2- 172.21.15.87:0/3825336061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b597da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- 172.21.15.87:0/3825336061 >> 172.21.15.87:0/3825336061 conn(0x55746b5920f0 msgr2=0x55746b5924d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- 172.21.15.87:0/3825336061 shutdown_connections 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- 172.21.15.87:0/3825336061 wait complete. 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 -- start start 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.195+0000 7fe6513cb740 1 Processor -- start 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b59ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55746b4fb400 con 0x55746b5a0ec0 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55746b4b7840 con 0x55746b5982e0 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55746b500720 con 0x55746b5979c0 2024-04-22T13:41:26.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 10 client.0 _ll_register_callbacks cb 0x55746b508ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54248/0 (socket says 172.21.15.87:54248) 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 -- 172.21.15.87:0/1212309798 learned_addr learned my addr 172.21.15.87:0/1212309798 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe647fff640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b59ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6477fe640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 -- 172.21.15.87:0/1212309798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 msgr2=0x55746b59ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b59ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 -- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 msgr2=0x55746b59e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 -- 172.21.15.87:0/1212309798 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55746b33b420 con 0x55746b5a0ec0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6477fe640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe6400126f0 tx=0x7fe640004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eed79d1d5fd0443 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6277fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640022070 con 0x55746b5a0ec0 2024-04-22T13:41:26.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe640011a30 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.196+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe63c012aa0 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe64002e050 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 10 client.18399 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55746b1ce580 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe63c012aa0 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe64001a070 con 0x55746b5a0ec0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 10 client.18399 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 10 client.18399 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 20 client.18399 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b5b0bb0 0x55746b5b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 4 client.18399 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55746b509b10 con 0x55746b5b0bb0 2024-04-22T13:41:26.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.197+0000 7fe647fff640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b5b0bb0 0x55746b5b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.198+0000 7fe647fff640 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b5b0bb0 0x55746b5b3010 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfa6bec659e8d6c server_cookie=8ebc0d657d392dcb in_seq=0 out_seq=0 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.198+0000 7fe6277fe640 10 client.18399 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.198+0000 7fe6277fe640 1 -- 172.21.15.87:0/1212309798 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55746b509b10 con 0x55746b5b0bb0 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.198+0000 7fe6277fe640 10 client.18399 handle_command_reply: tid=0 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.198+0000 7fe6513cb740 1 client.18399 shutdown 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 msgr2=0x55746b59e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe6400126f0 tx=0x7fe640004ab0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 shutdown_connections 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b5b0bb0 0x55746b5b3010 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59e950 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6277fe640 0 client.18399 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1212309798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b59ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 >> 172.21.15.87:0/1212309798 conn(0x55746b5920f0 msgr2=0x55746b593e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 shutdown_connections 2024-04-22T13:41:26.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 1 -- 172.21.15.87:0/1212309798 wait complete. 2024-04-22T13:41:26.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.199+0000 7fe6513cb740 20 client.18399 trim_cache size 0 max 16384 2024-04-22T13:41:26.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.215+0000 7fe6513cb740 1 Processor -- start 2024-04-22T13:41:26.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 -- start start 2024-04-22T13:41:26.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b5a0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481adc80 con 0x55746b5a0ec0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481abaf0 con 0x55746b5982e0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6513cb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481ad5b0 con 0x55746b5979c0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe6477fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54256/0 (socket says 172.21.15.87:54256) 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 -- 172.21.15.87:0/2017584807 learned_addr learned my addr 172.21.15.87:0/2017584807 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe647fff640 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b5a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 -- 172.21.15.87:0/2017584807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 msgr2=0x55746b5a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b5a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 -- 172.21.15.87:0/2017584807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 msgr2=0x55746b5ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 -- 172.21.15.87:0/2017584807 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55746b596ce0 con 0x55746b5a0ec0 2024-04-22T13:41:26.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe64001ee40 tx=0x7fe640009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1201d31623c2bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe62dffb640 1 -- 172.21.15.87:0/2017584807 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640036070 con 0x55746b5a0ec0 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.216+0000 7fe62dffb640 1 -- 172.21.15.87:0/2017584807 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe640011390 con 0x55746b5a0ec0 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe62dffb640 1 -- 172.21.15.87:0/2017584807 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640022070 con 0x55746b5a0ec0 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- 172.21.15.87:0/2017584807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 msgr2=0x55746b59d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe64001ee40 tx=0x7fe640009ce0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- 172.21.15.87:0/2017584807 shutdown_connections 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5a0ec0 0x55746b59d040 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b5ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 --2- 172.21.15.87:0/2017584807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5979c0 0x55746b5a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- 172.21.15.87:0/2017584807 >> 172.21.15.87:0/2017584807 conn(0x55746b5920f0 msgr2=0x55746b5911d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- 172.21.15.87:0/2017584807 shutdown_connections 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- 172.21.15.87:0/2017584807 wait complete. 2024-04-22T13:41:26.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.217+0000 7fe6513cb740 1 -- start start 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 Processor -- start 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 0x55746b59b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481ac810 con 0x55746b5979c0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481ac140 con 0x55746b5982e0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481acee0 con 0x55746b5a0ec0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6513cb740 10 client.0 _ll_register_callbacks cb 0x7fe640009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54262/0 (socket says 172.21.15.87:54262) 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 -- 172.21.15.87:0/1894559402 learned_addr learned my addr 172.21.15.87:0/1894559402 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 0x55746b59b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 -- 172.21.15.87:0/1894559402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 msgr2=0x55746b59b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 0x55746b59b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 -- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 msgr2=0x55746b59b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 -- 172.21.15.87:0/1894559402 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55746b616250 con 0x55746b5979c0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe64cd7f640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 0x55746b59b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe647fff640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe638006010 tx=0x7fe6380df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ff523f5ae498f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6277fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.218+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6380c7570 con 0x55746b5979c0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe6380bcc00 con 0x55746b5979c0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe640011390 con 0x55746b5979c0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe63802dd50 con 0x55746b5979c0 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 10 client.18405 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55746b617570 con 0x55746b5979c0 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe640011390 con 0x55746b5979c0 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6380efc00 con 0x55746b5979c0 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 10 client.18405 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 10 client.18405 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 20 client.18405 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b617d10 0x55746b61a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 4 client.18405 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await": true}] 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await": true}) -- 0x55746b503980 con 0x55746b617d10 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.219+0000 7fe6477fe640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b617d10 0x55746b61a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.220+0000 7fe6277fe640 10 client.18405 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.220+0000 7fe6477fe640 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b617d10 0x55746b61a130 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=635e02c7d49a9256 server_cookie=7852c5a6a3135af1 in_seq=0 out_seq=0 2024-04-22T13:41:26.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6277fe640 1 -- 172.21.15.87:0/1894559402 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55746b503980 con 0x55746b617d10 2024-04-22T13:41:26.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6277fe640 10 client.18405 handle_command_reply: tid=0 2024-04-22T13:41:26.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 client.18405 shutdown 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 msgr2=0x55746b59ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe638006010 tx=0x7fe6380df040 comp rx=0 tx=0).stop 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 shutdown_connections 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55746b617d10 0x55746b61a130 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55746b5a0ec0 0x55746b59b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6277fe640 0 client.18405 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55746b5982e0 0x55746b59b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 --2- 172.21.15.87:0/1894559402 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55746b5979c0 0x55746b59ac90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 >> 172.21.15.87:0/1894559402 conn(0x55746b5920f0 msgr2=0x55746b59de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 shutdown_connections 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 1 -- 172.21.15.87:0/1894559402 wait complete. 2024-04-22T13:41:26.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.231+0000 7fe6513cb740 20 client.18405 trim_cache size 0 max 16384 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 34, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "9dcd7ccc": { 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 34, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:41:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 70.0, 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:26.230 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 msgr2=0x7fe6300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 0x7fe6300a61e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe640005c40 tx=0x7fe640004a40 comp rx=0 tx=0).stop 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 msgr2=0x7fe64819e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fe638005940 tx=0x7fe6380045e0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 shutdown_connections 2024-04-22T13:41:26.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6300a3d20 0x7fe6300a61e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe64810c580 0x7fe6481985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe648103ad0 0x7fe64819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 --2- 172.21.15.87:0/3491680215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6481031b0 0x7fe64819e430 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 >> 172.21.15.87:0/3491680215 conn(0x7fe6480771d0 msgr2=0x7fe6480fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 shutdown_connections 2024-04-22T13:41:26.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:26.232+0000 7fe64e7e2640 1 -- 172.21.15.87:0/3491680215 wait complete. 2024-04-22T13:41:26.239 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 9dcd7ccc, quiesce took 0.5 seconds. Will release after: 10 2024-04-22T13:41:26.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:26 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1836994320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:26.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:26 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1836994320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:26.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:26 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1836994320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:27.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:27 smithi120 ceph-mon[31101]: pgmap v299: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:27.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:27 smithi132 ceph-mon[31531]: pgmap v299: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:27.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:27 smithi087 ceph-mon[27999]: pgmap v299: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:41:28.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:28 smithi120 ceph-mon[31101]: pgmap v300: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:28 smithi132 ceph-mon[31531]: pgmap v300: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:28 smithi087 ceph-mon[27999]: pgmap v300: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:30.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:30 smithi120 ceph-mon[31101]: pgmap v301: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:30.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:30 smithi132 ceph-mon[31531]: pgmap v301: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:30.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:30 smithi087 ceph-mon[27999]: pgmap v301: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:32 smithi120 ceph-mon[31101]: pgmap v302: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:32.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:32 smithi132 ceph-mon[31531]: pgmap v302: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:32 smithi087 ceph-mon[27999]: pgmap v302: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:34 smithi120 ceph-mon[31101]: pgmap v303: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:34 smithi132 ceph-mon[31531]: pgmap v303: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:34.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:34 smithi087 ceph-mon[27999]: pgmap v303: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:36.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:41:36.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/3031940170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102b40 msgr2=0x7fbf68106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/3031940170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102b40 0x7fbf68106b80 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009900 tx=0x7fbf5002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/3031940170 shutdown_connections 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/3031940170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 0x7fbf6810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/3031940170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102b40 0x7fbf68106b80 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/3031940170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf68102190 0x7fbf68102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/3031940170 >> 172.21.15.87:0/3031940170 conn(0x7fbf680793e0 msgr2=0x7fbf680797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/3031940170 shutdown_connections 2024-04-22T13:41:36.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.355+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/3031940170 wait complete. 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 Processor -- start 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 -- start start 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 0x7fbf68198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 0x7fbf6819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbf681121a0 con 0x7fbf68102190 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf68112020 con 0x7fbf6819d020 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.356+0000 7fbf6cd0e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf68112320 con 0x7fbf68107260 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57050/0 (socket says 172.21.15.87:57050) 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 -- 172.21.15.87:0/4164461853 learned_addr learned my addr 172.21.15.87:0/4164461853 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf5ffff640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 0x7fbf68198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf66d9d640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 0x7fbf6819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 -- 172.21.15.87:0/4164461853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 msgr2=0x7fbf68198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 0x7fbf68198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 -- 172.21.15.87:0/4164461853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 msgr2=0x7fbf6819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 0x7fbf6819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 -- 172.21.15.87:0/4164461853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6819db10 con 0x7fbf68102190 2024-04-22T13:41:36.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf66d9d640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 0x7fbf6819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6659c640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800d950 tx=0x7fbf5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf58017030 con 0x7fbf68102190 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf58004020 con 0x7fbf68102190 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6819de00 con 0x7fbf68102190 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf580047a0 con 0x7fbf68102190 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.357+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf681a55c0 con 0x7fbf68102190 2024-04-22T13:41:36.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.358+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf68069230 con 0x7fbf68102190 2024-04-22T13:41:36.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.359+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf58009a30 con 0x7fbf68102190 2024-04-22T13:41:36.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.360+0000 7fbf5f7fe640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 0x7fbf540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.360+0000 7fbf5ffff640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 0x7fbf540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.360+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbf580d0240 con 0x7fbf68102190 2024-04-22T13:41:36.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.360+0000 7fbf5ffff640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 0x7fbf540a61e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009230 tx=0x7fbf50005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.364+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf5809c920 con 0x7fbf68102190 2024-04-22T13:41:36.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.579+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf6810a490 con 0x7fbf68102190 2024-04-22T13:41:36.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.580+0000 7fbf5f7fe640 1 -- 172.21.15.87:0/4164461853 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fbf5809c220 con 0x7fbf68102190 2024-04-22T13:41:36.576 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:41:36.576 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:41:36.576 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 msgr2=0x7fbf540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 0x7fbf540a61e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009230 tx=0x7fbf50005fd0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 msgr2=0x7fbf68198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800d950 tx=0x7fbf5800de10 comp rx=0 tx=0).stop 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 shutdown_connections 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf540a3d20 0x7fbf540a61e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6819d020 0x7fbf6819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf68107260 0x7fbf68198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 --2- 172.21.15.87:0/4164461853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf68102190 0x7fbf68198760 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 >> 172.21.15.87:0/4164461853 conn(0x7fbf680793e0 msgr2=0x7fbf681000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 shutdown_connections 2024-04-22T13:41:36.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.582+0000 7fbf6cd0e640 1 -- 172.21.15.87:0/4164461853 wait complete. 2024-04-22T13:41:36.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:36 smithi087 ceph-mon[27999]: pgmap v304: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:36.589 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id 9dcd7ccc --release --await' 2024-04-22T13:41:36.589 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id 9dcd7ccc --release --await 2024-04-22T13:41:36.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:41:36.595 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:41:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:36 smithi120 ceph-mon[31101]: pgmap v304: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2235974523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894101090 msgr2=0x7fb894101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2235974523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894101090 0x7fb894101470 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb884009980 tx=0x7fb88402de60 comp rx=0 tx=0).stop 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2235974523 shutdown_connections 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2235974523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb894106160 0x7fb89410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2235974523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101a40 0x7fb894105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2235974523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894101090 0x7fb894101470 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2235974523 >> 172.21.15.87:0/2235974523 conn(0x7fb894077f50 msgr2=0x7fb894078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2235974523 shutdown_connections 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.700+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2235974523 wait complete. 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 Processor -- start 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 -- start start 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 0x7fb8941a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89419a770 0x7fb89419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb89410ffc0 con 0x7fb894106160 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb89410fe40 con 0x7fb894101090 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb89a4b2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb894110140 con 0x7fb89419a770 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb8937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb8937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57060/0 (socket says 172.21.15.87:57060) 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.701+0000 7fb8937fe640 1 -- 172.21.15.87:0/2609258036 learned_addr learned my addr 172.21.15.87:0/2609258036 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb893fff640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 0x7fb8941a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb898a4f640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89419a770 0x7fb89419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 -- 172.21.15.87:0/2609258036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89419a770 msgr2=0x7fb89419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89419a770 0x7fb89419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 -- 172.21.15.87:0/2609258036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 msgr2=0x7fb8941a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 0x7fb8941a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 -- 172.21.15.87:0/2609258036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89419b370 con 0x7fb894106160 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb893fff640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 0x7fb8941a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8937fe640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb88800d770 tx=0x7fb88800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8917fa640 1 -- 172.21.15.87:0/2609258036 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888017070 con 0x7fb894106160 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8941a7450 con 0x7fb894106160 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8917fa640 1 -- 172.21.15.87:0/2609258036 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb888009c30 con 0x7fb894106160 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8941a78b0 con 0x7fb894106160 2024-04-22T13:41:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.702+0000 7fb8917fa640 1 -- 172.21.15.87:0/2609258036 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88800f620 con 0x7fb894106160 2024-04-22T13:41:36.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.704+0000 7fb89d177740 1 Processor -- start 2024-04-22T13:41:36.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.704+0000 7fb8917fa640 1 -- 172.21.15.87:0/2609258036 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb88801a070 con 0x7fb894106160 2024-04-22T13:41:36.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.704+0000 7fb89d177740 1 -- start start 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92310 0x55a709c926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c9dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a709bf7500 con 0x55a709c92c30 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a709b9d6a0 con 0x55a709c9b810 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb89d177740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a709bf34c0 con 0x55a709c92310 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57062/0 (socket says 172.21.15.87:57062) 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 -- 172.21.15.87:0/2886813694 learned_addr learned my addr 172.21.15.87:0/2886813694 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb898a4f640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92310 0x55a709c926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb8917fa640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 0x7fb87c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb8937fe640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c9dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 -- 172.21.15.87:0/2886813694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92310 msgr2=0x55a709c926f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92310 0x55a709c926f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 -- 172.21.15.87:0/2886813694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 msgr2=0x55a709c9dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c9dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 -- 172.21.15.87:0/2886813694 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a709bbc5e0 con 0x55a709c92c30 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb8937fe640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c9dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb898a4f640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 0x7fb87c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb8917fa640 1 -- 172.21.15.87:0/2609258036 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb8880d03c0 con 0x7fb894106160 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.705+0000 7fb893fff640 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb884038040 tx=0x7fb884005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfe79074f803e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb87a7fc640 1 -- 172.21.15.87:0/2886813694 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884004030 con 0x55a709c92c30 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb87a7fc640 1 -- 172.21.15.87:0/2886813694 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88402e800 con 0x55a709c92c30 2024-04-22T13:41:36.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb87a7fc640 1 -- 172.21.15.87:0/2886813694 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884003ab0 con 0x55a709c92c30 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 -- 172.21.15.87:0/2886813694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 msgr2=0x55a709c9b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb898a4f640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 0x7fb87c0a61e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb88c005fd0 tx=0x7fb88c004a40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb884038040 tx=0x7fb884005e00 comp rx=0 tx=0).stop 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 -- 172.21.15.87:0/2886813694 shutdown_connections 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c9dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709c9b2d0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 --2- 172.21.15.87:0/2886813694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92310 0x55a709c926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 -- 172.21.15.87:0/2886813694 >> 172.21.15.87:0/2886813694 conn(0x55a709c8ca40 msgr2=0x55a709c8ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 -- 172.21.15.87:0/2886813694 shutdown_connections 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.706+0000 7fb89d177740 1 -- 172.21.15.87:0/2886813694 wait complete. 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 -- start start 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 Processor -- start 2024-04-22T13:41:36.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709c985e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c98d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a709c9e5a0 con 0x55a709c92310 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a709bb0a60 con 0x55a709c9b810 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a709c9e720 con 0x55a709c92c30 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb89d177740 10 client.0 _ll_register_callbacks cb 0x55a709909110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57072/0 (socket says 172.21.15.87:57072) 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb893fff640 1 -- 172.21.15.87:0/1029494650 learned_addr learned my addr 172.21.15.87:0/1029494650 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb898a4f640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c98d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.707+0000 7fb8937fe640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709c985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 -- 172.21.15.87:0/1029494650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 msgr2=0x55a709c985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709c985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 -- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 msgr2=0x55a709c98d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c98d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 -- 172.21.15.87:0/1029494650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a709909920 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb893fff640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb884009400 tx=0x7fb88402ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7a96a92c8bbb1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88404e070 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb884037ac0 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88402e800 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb884037cc0 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb89d177740 10 client.18423 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a7099d5d30 con 0x55a709c92310 2024-04-22T13:41:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88402e800 con 0x55a709c92310 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.708+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb88404a4d0 con 0x55a709c92310 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 10 client.18423 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 10 client.18423 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 20 client.18423 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709cab6c0 0x55a709cadab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 4 client.18423 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a709bcaa10 con 0x55a709cab6c0 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb8937fe640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709cab6c0 0x55a709cadab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb8937fe640 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709cab6c0 0x55a709cadab0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51ed05d5f8b7ea0 server_cookie=d20cabb8b38c74c7 in_seq=0 out_seq=0 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.709+0000 7fb8737fe640 10 client.18423 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb8737fe640 1 -- 172.21.15.87:0/1029494650 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a709bcaa10 con 0x55a709cab6c0 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb8737fe640 10 client.18423 handle_command_reply: tid=0 2024-04-22T13:41:36.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 client.18423 shutdown 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 msgr2=0x55a709c9b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb884009400 tx=0x7fb88402ee20 comp rx=0 tx=0).stop 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 shutdown_connections 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709cab6c0 0x55a709cadab0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c98d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709c985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 --2- 172.21.15.87:0/1029494650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c9b250 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb8737fe640 0 client.18423 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 >> 172.21.15.87:0/1029494650 conn(0x55a709c8ca40 msgr2=0x55a709c8e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 shutdown_connections 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.710+0000 7fb89d177740 1 -- 172.21.15.87:0/1029494650 wait complete. 2024-04-22T13:41:36.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.711+0000 7fb89d177740 20 client.18423 trim_cache size 0 max 16384 2024-04-22T13:41:36.717 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:36 smithi132 ceph-mon[31531]: pgmap v304: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.727+0000 7fb89d177740 1 Processor -- start 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.727+0000 7fb89d177740 1 -- start start 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709ca7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c95860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941b2640 con 0x55a709c92310 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941aff10 con 0x55a709c9b810 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb89d177740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941add80 con 0x55a709c92c30 2024-04-22T13:41:36.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57086/0 (socket says 172.21.15.87:57086) 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 -- 172.21.15.87:0/4199850187 learned_addr learned my addr 172.21.15.87:0/4199850187 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb8937fe640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709ca7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb898a4f640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c95860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 -- 172.21.15.87:0/4199850187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 msgr2=0x55a709ca7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709ca7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 -- 172.21.15.87:0/4199850187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 msgr2=0x55a709c95860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c95860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 -- 172.21.15.87:0/4199850187 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a709c97590 con 0x55a709c92310 2024-04-22T13:41:36.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb898a4f640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb8937fe640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709ca7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb893fff640 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb88404a8a0 tx=0x7fb884001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11053b5498dfb709 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb879ffb640 1 -- 172.21.15.87:0/4199850187 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88404e040 con 0x55a709c92310 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb879ffb640 1 -- 172.21.15.87:0/4199850187 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88402e800 con 0x55a709c92310 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.728+0000 7fb879ffb640 1 -- 172.21.15.87:0/4199850187 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88404a440 con 0x55a709c92310 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 -- 172.21.15.87:0/4199850187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 msgr2=0x55a709c8bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb88404a8a0 tx=0x7fb884001e70 comp rx=0 tx=0).stop 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 -- 172.21.15.87:0/4199850187 shutdown_connections 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c9b810 0x55a709c95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c92c30 0x55a709ca7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 --2- 172.21.15.87:0/4199850187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92310 0x55a709c8bb00 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 -- 172.21.15.87:0/4199850187 >> 172.21.15.87:0/4199850187 conn(0x55a709c8ca40 msgr2=0x55a709ca6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 -- 172.21.15.87:0/4199850187 shutdown_connections 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.729+0000 7fb89d177740 1 -- 172.21.15.87:0/4199850187 wait complete. 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 -- start start 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 Processor -- start 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c92310 0x55a709cb0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c9b810 0x55a709cad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941ae3d0 con 0x55a709c92c30 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941af840 con 0x55a709c92310 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8941aeaa0 con 0x55a709c9b810 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb89d177740 10 client.0 _ll_register_callbacks cb 0x7fb88c009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57098/0 (socket says 172.21.15.87:57098) 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 -- 172.21.15.87:0/3595237620 learned_addr learned my addr 172.21.15.87:0/3595237620 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb893fff640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c92310 0x55a709cb0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb898a4f640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c9b810 0x55a709cad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 -- 172.21.15.87:0/3595237620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c9b810 msgr2=0x55a709cad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c9b810 0x55a709cad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 -- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c92310 msgr2=0x55a709cb0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.730+0000 7fb8937fe640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c92310 0x55a709cb0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8937fe640 1 -- 172.21.15.87:0/3595237620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a709c92a60 con 0x55a709c92c30 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8937fe640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb8880cdbb0 tx=0x7fb88800de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b0c5ff626c5cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888021040 con 0x55a709c92c30 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb888098a10 con 0x55a709c92c30 2024-04-22T13:41:36.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88402e800 con 0x55a709c92c30 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb88809c3a0 con 0x55a709c92c30 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 10 client.18429 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a709d12700 con 0x55a709c92c30 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88402e800 con 0x55a709c92c30 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb88809b260 con 0x55a709c92c30 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 10 client.18429 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 10 client.18429 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 20 client.18429 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709d12de0 0x55a709d15200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 4 client.18429 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9dcd7ccc", "release": true, "await": true}] 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.731+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9dcd7ccc", "release": true, "await": true}) -- 0x55a709c04f20 con 0x55a709d12de0 2024-04-22T13:41:36.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.732+0000 7fb893fff640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709d12de0 0x55a709d15200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.732+0000 7fb8737fe640 10 client.18429 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.732+0000 7fb893fff640 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709d12de0 0x55a709d15200 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b454cab3843f388d server_cookie=834e502f07ad8cd1 in_seq=0 out_seq=0 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.734+0000 7fb8737fe640 1 -- 172.21.15.87:0/3595237620 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a709c04f20 con 0x55a709d12de0 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.734+0000 7fb8737fe640 10 client.18429 handle_command_reply: tid=0 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.734+0000 7fb89d177740 1 client.18429 shutdown 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 msgr2=0x55a709cad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb8880cdbb0 tx=0x7fb88800de80 comp rx=0 tx=0).stop 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 shutdown_connections 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a709d12de0 0x55a709d15200 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a709c9b810 0x55a709cad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a709c92c30 0x55a709cad280 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb8737fe640 0 client.18429 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 --2- 172.21.15.87:0/3595237620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a709c92310 0x55a709cb0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 >> 172.21.15.87:0/3595237620 conn(0x55a709c8ca40 msgr2=0x55a709ca7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 shutdown_connections 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 1 -- 172.21.15.87:0/3595237620 wait complete. 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89d177740 20 client.18429 trim_cache size 0 max 16384 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 36, 2024-04-22T13:41:36.737 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "9dcd7ccc": { 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 36, 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 70.0, 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:41:36.738 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:36.739 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:36.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 msgr2=0x7fb87c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 0x7fb87c0a61e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb88c005fd0 tx=0x7fb88c004a40 comp rx=0 tx=0).stop 2024-04-22T13:41:36.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 msgr2=0x7fb8941a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb88800d770 tx=0x7fb88800dc30 comp rx=0 tx=0).stop 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 shutdown_connections 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb87c0a3d20 0x7fb87c0a61e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89419a770 0x7fb89419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb894106160 0x7fb8941a0bf0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 --2- 172.21.15.87:0/2609258036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb894101090 0x7fb8941a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.735+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 >> 172.21.15.87:0/2609258036 conn(0x7fb894077f50 msgr2=0x7fb8940ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.736+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 shutdown_connections 2024-04-22T13:41:36.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.736+0000 7fb89a4b2640 1 -- 172.21.15.87:0/2609258036 wait complete. 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- 172.21.15.87:0/202675077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 msgr2=0x7fc50c10dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 --2- 172.21.15.87:0/202675077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c10dc20 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc504009cd0 tx=0x7fc50402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- 172.21.15.87:0/202675077 shutdown_connections 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 --2- 172.21.15.87:0/202675077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c10dc20 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 --2- 172.21.15.87:0/202675077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c1010a0 0x7fc50c101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 --2- 172.21.15.87:0/202675077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c106fe0 0x7fc50c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- 172.21.15.87:0/202675077 >> 172.21.15.87:0/202675077 conn(0x7fc50c077c10 msgr2=0x7fc50c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- 172.21.15.87:0/202675077 shutdown_connections 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- 172.21.15.87:0/202675077 wait complete. 2024-04-22T13:41:36.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 Processor -- start 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 -- start start 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.742+0000 7fc513255640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 0x7fc50c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c106fe0 0x7fc50c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1110f0 con 0x7fc50c101a60 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c110f70 con 0x7fc50c1010a0 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c111270 con 0x7fc50c106fe0 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57112/0 (socket says 172.21.15.87:57112) 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 -- 172.21.15.87:0/743625329 learned_addr learned my addr 172.21.15.87:0/743625329 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc510ff1640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 0x7fc50c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc5117f2640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c106fe0 0x7fc50c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 -- 172.21.15.87:0/743625329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c106fe0 msgr2=0x7fc50c19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c106fe0 0x7fc50c19bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 -- 172.21.15.87:0/743625329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 msgr2=0x7fc50c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 0x7fc50c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 -- 172.21.15.87:0/743625329 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c19c660 con 0x7fc50c101a60 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc510ff1640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 0x7fc50c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc50bfff640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9f0 tx=0x7fc4f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc509ffb640 1 -- 172.21.15.87:0/743625329 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800cd80 con 0x7fc50c101a60 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc509ffb640 1 -- 172.21.15.87:0/743625329 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f80054d0 con 0x7fc50c101a60 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c19c950 con 0x7fc50c101a60 2024-04-22T13:41:36.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.743+0000 7fc509ffb640 1 -- 172.21.15.87:0/743625329 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8010690 con 0x7fc50c101a60 2024-04-22T13:41:36.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.744+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c1a41e0 con 0x7fc50c101a60 2024-04-22T13:41:36.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.746+0000 7fc509ffb640 1 -- 172.21.15.87:0/743625329 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4f8004e90 con 0x7fc50c101a60 2024-04-22T13:41:36.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 Processor -- start 2024-04-22T13:41:36.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 -- start start 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c35270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c42b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561684b9a1e0 con 0x561684c43080 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561684b37390 con 0x561684c357b0 2024-04-22T13:41:36.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc515e46740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561684b961a0 con 0x561684c34e90 2024-04-22T13:41:36.755 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 9dcd7ccc, seconds elapsed: 11.0 2024-04-22T13:41:36.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57124/0 (socket says 172.21.15.87:57124) 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 -- 172.21.15.87:0/4146105718 learned_addr learned my addr 172.21.15.87:0/4146105718 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc5117f2640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c35270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc509ffb640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 0x7fc4fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc510ff1640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c42b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc5117f2640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 0x7fc4fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc509ffb640 1 -- 172.21.15.87:0/743625329 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fc4f80d0bb0 con 0x7fc50c101a60 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 -- 172.21.15.87:0/4146105718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 msgr2=0x561684c35270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c35270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 -- 172.21.15.87:0/4146105718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 msgr2=0x561684c42b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c42b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc50bfff640 1 -- 172.21.15.87:0/4146105718 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616849767d0 con 0x561684c43080 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc5117f2640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c35270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.747+0000 7fc510ff1640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c42b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.748+0000 7fc50bfff640 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc4f809c890 tx=0x7fc4f8004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b172b06c7b714812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.748+0000 7fc5117f2640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 0x7fc4fc0a61e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc504005cc0 tx=0x7fc50402c0f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.748+0000 7fc4f6ffd640 1 -- 172.21.15.87:0/4146105718 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f809c4f0 con 0x561684c43080 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.748+0000 7fc4f6ffd640 1 -- 172.21.15.87:0/4146105718 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f809d290 con 0x561684c43080 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.748+0000 7fc4f6ffd640 1 -- 172.21.15.87:0/4146105718 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8005180 con 0x561684c43080 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- 172.21.15.87:0/4146105718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 msgr2=0x561684c45470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc4f809c890 tx=0x7fc4f8004a90 comp rx=0 tx=0).stop 2024-04-22T13:41:36.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- 172.21.15.87:0/4146105718 shutdown_connections 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c45470 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c42b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- 172.21.15.87:0/4146105718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c35270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- 172.21.15.87:0/4146105718 >> 172.21.15.87:0/4146105718 conn(0x561684c2fcb0 msgr2=0x561684c30090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- 172.21.15.87:0/4146105718 shutdown_connections 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- 172.21.15.87:0/4146105718 wait complete. 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- start start 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 Processor -- start 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c40050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 0x561684c40ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561684c45c50 con 0x561684c357b0 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561684b532f0 con 0x561684c43080 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561684c45dd0 con 0x561684c34e90 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc515e46740 10 client.0 _ll_register_callbacks cb 0x561684b60580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.749+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57132/0 (socket says 172.21.15.87:57132) 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 -- 172.21.15.87:0/1320320882 learned_addr learned my addr 172.21.15.87:0/1320320882 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc510ff1640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c40050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc5117f2640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 0x561684c40ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 -- 172.21.15.87:0/1320320882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 msgr2=0x561684c40050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c40050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 -- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 msgr2=0x561684c40ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 0x561684c40ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 -- 172.21.15.87:0/1320320882 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616848ac3f0 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc5117f2640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 0x561684c40ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc50bfff640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8099560 tx=0x7fc4f809c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3573c24db00c4c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc4ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8005180 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fc4f80ce5c0 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4f809d290 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc4f809c4f0 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc515e46740 10 client.18441 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56168486c0f0 con 0x561684c357b0 2024-04-22T13:41:36.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.750+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f809d290 con 0x561684c357b0 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc4f80a5370 con 0x561684c357b0 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 10 client.18441 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 10 client.18441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 20 client.18441 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684c52cc0 0x561684c550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 4 client.18441 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561684ba7c00 con 0x561684c52cc0 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc510ff1640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684c52cc0 0x561684c550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc4ebfff640 10 client.18441 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.751+0000 7fc510ff1640 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684c52cc0 0x561684c550b0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbeeda212cad5236 server_cookie=92949036e4c036cc in_seq=0 out_seq=0 2024-04-22T13:41:36.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc4ebfff640 1 -- 172.21.15.87:0/1320320882 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561684ba7c00 con 0x561684c52cc0 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc4ebfff640 10 client.18441 handle_command_reply: tid=0 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 client.18441 shutdown 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 msgr2=0x561684c40590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8099560 tx=0x7fc4f809c7c0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 shutdown_connections 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684c52cc0 0x561684c550b0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c43080 0x561684c40ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c357b0 0x561684c40590 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 --2- 172.21.15.87:0/1320320882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc4ebfff640 0 client.18441 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 >> 172.21.15.87:0/1320320882 conn(0x561684c2fcb0 msgr2=0x561684c31a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 shutdown_connections 2024-04-22T13:41:36.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 1 -- 172.21.15.87:0/1320320882 wait complete. 2024-04-22T13:41:36.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.752+0000 7fc515e46740 20 client.18441 trim_cache size 0 max 16384 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.769+0000 7fc515e46740 1 Processor -- start 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.769+0000 7fc515e46740 1 -- start start 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c42780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c3db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ae160 con 0x561684c43080 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1aba30 con 0x561684c357b0 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc515e46740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1a98a0 con 0x561684c34e90 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57148/0 (socket says 172.21.15.87:57148) 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 -- 172.21.15.87:0/3444854204 learned_addr learned my addr 172.21.15.87:0/3444854204 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc510ff1640 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c42780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc50bfff640 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c3db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 -- 172.21.15.87:0/3444854204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 msgr2=0x561684c42780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c42780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 -- 172.21.15.87:0/3444854204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 msgr2=0x561684c3db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c3db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 -- 172.21.15.87:0/3444854204 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561684c30ee0 con 0x561684c43080 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc504032340 tx=0x7fc5040323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f678fbb627caf536 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc4f67fc640 1 -- 172.21.15.87:0/3444854204 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504047070 con 0x561684c43080 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc4f67fc640 1 -- 172.21.15.87:0/3444854204 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc504053070 con 0x561684c43080 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.770+0000 7fc4f67fc640 1 -- 172.21.15.87:0/3444854204 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504039040 con 0x561684c43080 2024-04-22T13:41:36.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- 172.21.15.87:0/3444854204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 msgr2=0x561684c3f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc504032340 tx=0x7fc5040323c0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- 172.21.15.87:0/3444854204 shutdown_connections 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684c3f1b0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c357b0 0x561684c3db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 --2- 172.21.15.87:0/3444854204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c34e90 0x561684c42780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- 172.21.15.87:0/3444854204 >> 172.21.15.87:0/3444854204 conn(0x561684c2fcb0 msgr2=0x561684c310d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- 172.21.15.87:0/3444854204 shutdown_connections 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- 172.21.15.87:0/3444854204 wait complete. 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 -- start start 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.771+0000 7fc515e46740 1 Processor -- start 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c34e90 0x561684c37110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c357b0 0x561684c37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1a9ef0 con 0x561684c43080 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ab360 con 0x561684c34e90 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1aa5c0 con 0x561684c357b0 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 10 client.0 _ll_register_callbacks cb 0x7fc500009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57154/0 (socket says 172.21.15.87:57154) 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 -- 172.21.15.87:0/3567122503 learned_addr learned my addr 172.21.15.87:0/3567122503 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc50bfff640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c357b0 0x561684c37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc510ff1640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c34e90 0x561684c37110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 -- 172.21.15.87:0/3567122503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c357b0 msgr2=0x561684c37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c357b0 0x561684c37650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 -- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c34e90 msgr2=0x561684c37110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c34e90 0x561684c37110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 -- 172.21.15.87:0/3567122503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561684cb83d0 con 0x561684c43080 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc5117f2640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc504032390 tx=0x7fc504004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1426ce45aec68d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc4ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50405b070 con 0x561684c43080 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fc504003b70 con 0x561684c43080 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc504053070 con 0x561684c43080 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.772+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc504067050 con 0x561684c43080 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 10 client.18447 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:36.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561684cb9520 con 0x561684c43080 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc504053070 con 0x561684c43080 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc50403f070 con 0x561684c43080 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 10 client.18447 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 10 client.18447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 20 client.18447 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684cb9cd0 0x561684cbc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 4 client.18447 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561684ba1a70 con 0x561684cb9cd0 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc510ff1640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684cb9cd0 0x561684cbc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc4ebfff640 10 client.18447 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.773+0000 7fc510ff1640 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684cb9cd0 0x561684cbc0f0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e34a38ab2e779db5 server_cookie=c968d212dbcfd194 in_seq=0 out_seq=0 2024-04-22T13:41:36.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc4ebfff640 1 -- 172.21.15.87:0/3567122503 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561684ba1a70 con 0x561684cb9cd0 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc4ebfff640 10 client.18447 handle_command_reply: tid=0 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 client.18447 shutdown 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 msgr2=0x561684cb7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc504032390 tx=0x7fc504004380 comp rx=0 tx=0).stop 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 shutdown_connections 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561684cb9cd0 0x561684cbc0f0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561684c43080 0x561684cb7e90 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc4ebfff640 0 client.18447 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561684c357b0 0x561684c37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 --2- 172.21.15.87:0/3567122503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561684c34e90 0x561684c37110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 >> 172.21.15.87:0/3567122503 conn(0x561684c2fcb0 msgr2=0x561684c2ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 shutdown_connections 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 1 -- 172.21.15.87:0/3567122503 wait complete. 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc515e46740 20 client.18447 trim_cache size 0 max 16384 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:41:36.779 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:36.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 msgr2=0x7fc4fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 0x7fc4fc0a61e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc504005cc0 tx=0x7fc50402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 msgr2=0x7fc50c197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.774+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9f0 tx=0x7fc4f800edf0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 shutdown_connections 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc4fc0a3d20 0x7fc4fc0a61e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc50c106fe0 0x7fc50c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc50c101a60 0x7fc50c197bb0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 --2- 172.21.15.87:0/743625329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc50c1010a0 0x7fc50c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 >> 172.21.15.87:0/743625329 conn(0x7fc50c077c10 msgr2=0x7fc50c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 shutdown_connections 2024-04-22T13:41:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.775+0000 7fc513255640 1 -- 172.21.15.87:0/743625329 wait complete. 2024-04-22T13:41:36.783 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:41:36.783 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:41:36.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 -- 172.21.15.87:0/3314909006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 msgr2=0x7f0c64103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 --2- 172.21.15.87:0/3314909006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c64103200 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009900 tx=0x7f0c5402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 -- 172.21.15.87:0/3314909006 shutdown_connections 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 --2- 172.21.15.87:0/3314909006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c64110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 --2- 172.21.15.87:0/3314909006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c64100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 --2- 172.21.15.87:0/3314909006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c64103200 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 -- 172.21.15.87:0/3314909006 >> 172.21.15.87:0/3314909006 conn(0x7f0c64077450 msgr2=0x7f0c640fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 -- 172.21.15.87:0/3314909006 shutdown_connections 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.923+0000 7f0c6a796640 1 -- 172.21.15.87:0/3314909006 wait complete. 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 Processor -- start 2024-04-22T13:41:36.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 -- start start 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c641a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c641a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0c641143f0 con 0x7f0c64102e20 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64114270 con 0x7f0c641004c0 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c6a796640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64114570 con 0x7f0c64101050 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57178/0 (socket says 172.21.15.87:57178) 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 -- 172.21.15.87:0/4230100645 learned_addr learned my addr 172.21.15.87:0/4230100645 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c63fff640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c641a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c637fe640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c641a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 -- 172.21.15.87:0/4230100645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 msgr2=0x7f0c641a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c641a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 -- 172.21.15.87:0/4230100645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 msgr2=0x7f0c641a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:36.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c641a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 -- 172.21.15.87:0/4230100645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6419afd0 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c637fe640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c641a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c63fff640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c641a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.924+0000 7f0c68d33640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009fd0 tx=0x7f0c5c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.925+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c004d20 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.925+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c5c005020 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.925+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6419b2c0 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.925+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0116b0 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.925+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c641a7580 con 0x7f0c64102e20 2024-04-22T13:41:36.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.926+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c641014d0 con 0x7f0c64102e20 2024-04-22T13:41:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.927+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c5c0051d0 con 0x7f0c64102e20 2024-04-22T13:41:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.927+0000 7f0c617fa640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 0x7f0c4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.927+0000 7f0c63fff640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 0x7f0c4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.927+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f0c5c0d0d70 con 0x7f0c64102e20 2024-04-22T13:41:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.928+0000 7f0c63fff640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 0x7f0c4c0a61e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0c54005ab0 tx=0x7f0c54005a20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:36.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:36.932+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c5c016030 con 0x7f0c64102e20 2024-04-22T13:41:37.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.146+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c6410c320 con 0x7f0c64102e20 2024-04-22T13:41:37.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.147+0000 7f0c617fa640 1 -- 172.21.15.87:0/4230100645 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f0c5c09d450 con 0x7f0c64102e20 2024-04-22T13:41:37.143 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:41:37.144 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:41:37.144 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:41:37.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 msgr2=0x7f0c4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 0x7f0c4c0a61e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0c54005ab0 tx=0x7f0c54005a20 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 msgr2=0x7f0c6419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009fd0 tx=0x7f0c5c00ee60 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 shutdown_connections 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c4c0a3d20 0x7f0c4c0a61e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c64102e20 0x7f0c6419a710 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c64101050 0x7f0c641a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 --2- 172.21.15.87:0/4230100645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c641004c0 0x7f0c641a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 >> 172.21.15.87:0/4230100645 conn(0x7f0c64077450 msgr2=0x7f0c640fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 shutdown_connections 2024-04-22T13:41:37.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.149+0000 7f0c6a796640 1 -- 172.21.15.87:0/4230100645 wait complete. 2024-04-22T13:41:37.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:41:37.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 -- 172.21.15.87:0/3680437860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f916406a880 msgr2=0x7f916406ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 --2- 172.21.15.87:0/3680437860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f916406a880 0x7f916406ac60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9154009900 tx=0x7f915402bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:37.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 -- 172.21.15.87:0/3680437860 shutdown_connections 2024-04-22T13:41:37.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 --2- 172.21.15.87:0/3680437860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9164107270 0x7f916410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 --2- 172.21.15.87:0/3680437860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406b230 0x7f9164106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 --2- 172.21.15.87:0/3680437860 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f916406a880 0x7f916406ac60 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 -- 172.21.15.87:0/3680437860 >> 172.21.15.87:0/3680437860 conn(0x7f9164077eb0 msgr2=0x7f91640782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 -- 172.21.15.87:0/3680437860 shutdown_connections 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.262+0000 7f9169985640 1 -- 172.21.15.87:0/3680437860 wait complete. 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 Processor -- start 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 -- start start 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406a880 0x7f9164198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f916419cff0 0x7f916419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f91641121b0 con 0x7f9164107270 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9164112030 con 0x7f916406a880 2024-04-22T13:41:37.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9169985640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9164112330 con 0x7f916419cff0 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57194/0 (socket says 172.21.15.87:57194) 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 -- 172.21.15.87:0/3677437012 learned_addr learned my addr 172.21.15.87:0/3677437012 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9162ffd640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406a880 0x7f9164198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f91637fe640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f916419cff0 0x7f916419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 -- 172.21.15.87:0/3677437012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f916419cff0 msgr2=0x7f916419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f916419cff0 0x7f916419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 -- 172.21.15.87:0/3677437012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406a880 msgr2=0x7f9164198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406a880 0x7f9164198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 -- 172.21.15.87:0/3677437012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916419dae0 con 0x7f9164107270 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f915bfff640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e9c0 tx=0x7f914c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.263+0000 7f9168983640 1 -- 172.21.15.87:0/3677437012 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c00cd80 con 0x7f9164107270 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.264+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916419ddd0 con 0x7f9164107270 2024-04-22T13:41:37.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.264+0000 7f9168983640 1 -- 172.21.15.87:0/3677437012 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914c0054d0 con 0x7f9164107270 2024-04-22T13:41:37.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.264+0000 7f9168983640 1 -- 172.21.15.87:0/3677437012 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c010430 con 0x7f9164107270 2024-04-22T13:41:37.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.264+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91641a5610 con 0x7f9164107270 2024-04-22T13:41:37.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9168983640 1 -- 172.21.15.87:0/3677437012 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f914c004fd0 con 0x7f9164107270 2024-04-22T13:41:37.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 Processor -- start 2024-04-22T13:41:37.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 -- start start 2024-04-22T13:41:37.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 0x55b1c22948b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b1c21f91f0 con 0x55b1c2294e80 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b1c21964b0 con 0x55b1c22944d0 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f916c574740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b1c21f51b0 con 0x55b1c229da00 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9168983640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f91637fe640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57198/0 (socket says 172.21.15.87:57198) 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9162ffd640 1 -- 172.21.15.87:0/1101086518 learned_addr learned my addr 172.21.15.87:0/1101086518 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f91637fe640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 0x55b1c22948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f9168983640 1 -- 172.21.15.87:0/3677437012 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f914c0d0bc0 con 0x7f9164107270 2024-04-22T13:41:37.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.266+0000 7f915bfff640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 -- 172.21.15.87:0/1101086518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 msgr2=0x55b1c229fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 -- 172.21.15.87:0/1101086518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 msgr2=0x55b1c22948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 0x55b1c22948b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 -- 172.21.15.87:0/1101086518 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c21881b0 con 0x55b1c2294e80 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f91637fe640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 0x55b1c22948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9162ffd640 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9154009900 tx=0x7f915402c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebfc088d2f3e63f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f91637fe640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 0x7f91500a61e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f915c009fd0 tx=0x7f915c004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9159ffb640 1 -- 172.21.15.87:0/1101086518 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154045020 con 0x55b1c2294e80 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9159ffb640 1 -- 172.21.15.87:0/1101086518 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f915402c7e0 con 0x55b1c2294e80 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.267+0000 7f9159ffb640 1 -- 172.21.15.87:0/1101086518 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91540046c0 con 0x55b1c2294e80 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- 172.21.15.87:0/1101086518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 msgr2=0x55b1c229d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9154009900 tx=0x7f915402c610 comp rx=0 tx=0).stop 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- 172.21.15.87:0/1101086518 shutdown_connections 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d430 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- 172.21.15.87:0/1101086518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c22944d0 0x55b1c22948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- 172.21.15.87:0/1101086518 >> 172.21.15.87:0/1101086518 conn(0x55b1c228eba0 msgr2=0x55b1c228ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- 172.21.15.87:0/1101086518 shutdown_connections 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- 172.21.15.87:0/1101086518 wait complete. 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- start start 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 Processor -- start 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 0x55b1c229a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b1c22a0910 con 0x55b1c22944d0 2024-04-22T13:41:37.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b1c21b2500 con 0x55b1c2294e80 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.268+0000 7f916c574740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b1c22a0a90 con 0x55b1c229da00 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f916c574740 10 client.0 _ll_register_callbacks cb 0x55b1c2180b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57210/0 (socket says 172.21.15.87:57210) 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 -- 172.21.15.87:0/3526047971 learned_addr learned my addr 172.21.15.87:0/3526047971 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f915bfff640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 0x55b1c229a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f91637fe640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 -- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 msgr2=0x55b1c229abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 -- 172.21.15.87:0/3526047971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 msgr2=0x55b1c229a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 0x55b1c229a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 -- 172.21.15.87:0/3526047971 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c1f0b3f0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f915bfff640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 0x55b1c229a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9162ffd640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9154036040 tx=0x7f9154000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edd90e24d302a49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915404e070 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f91540035f0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f915402c7e0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f91540037f0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f916c574740 10 client.18465 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b1c1fd57d0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.269+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f915402c7e0 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f9154004c70 con 0x55b1c22944d0 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 10 client.18465 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 10 client.18465 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 20 client.18465 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c22afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 4 client.18465 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1c2198930 con 0x55b1c22ada00 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f915bfff640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c22afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f915bfff640 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c22afdf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5269ee7f07ab48d8 server_cookie=f50566bc1725b3cf in_seq=0 out_seq=0 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.270+0000 7f9142ffd640 10 client.18465 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f9142ffd640 1 -- 172.21.15.87:0/3526047971 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1c2198930 con 0x55b1c22ada00 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f9142ffd640 10 client.18465 handle_command_reply: tid=0 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 client.18465 shutdown 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 msgr2=0x55b1c229d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9154036040 tx=0x7f9154000bc0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 shutdown_connections 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c22afdf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c229da00 0x55b1c229abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c2294e80 0x55b1c229a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f9142ffd640 0 client.18465 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 --2- 172.21.15.87:0/3526047971 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c22944d0 0x55b1c229d2e0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 >> 172.21.15.87:0/3526047971 conn(0x55b1c228eba0 msgr2=0x55b1c22908f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 shutdown_connections 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 1 -- 172.21.15.87:0/3526047971 wait complete. 2024-04-22T13:41:37.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.271+0000 7f916c574740 20 client.18465 trim_cache size 0 max 16384 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 Processor -- start 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 -- start start 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c229cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f91641b0300 con 0x55b1c2294e80 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f91641adbd0 con 0x55b1c229da00 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.284+0000 7f916c574740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f91641aba40 con 0x55b1c22944d0 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57222/0 (socket says 172.21.15.87:57222) 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 -- 172.21.15.87:0/1865128942 learned_addr learned my addr 172.21.15.87:0/1865128942 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f9162ffd640 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c229cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 -- 172.21.15.87:0/1865128942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 msgr2=0x55b1c229cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c229cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 -- 172.21.15.87:0/1865128942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 msgr2=0x55b1c22978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 -- 172.21.15.87:0/1865128942 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c2294c20 con 0x55b1c2294e80 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f915bfff640 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ce3b0 tx=0x7f914c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcedc4c2086d992f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f91597fa640 1 -- 172.21.15.87:0/1865128942 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0b5070 con 0x55b1c2294e80 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f91597fa640 1 -- 172.21.15.87:0/1865128942 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914c0046c0 con 0x55b1c2294e80 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.285+0000 7f91597fa640 1 -- 172.21.15.87:0/1865128942 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0bd0b0 con 0x55b1c2294e80 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- 172.21.15.87:0/1865128942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 msgr2=0x55b1c229d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ce3b0 tx=0x7f914c0bd040 comp rx=0 tx=0).stop 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- 172.21.15.87:0/1865128942 shutdown_connections 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c229d2e0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- 172.21.15.87:0/1865128942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c229cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- 172.21.15.87:0/1865128942 >> 172.21.15.87:0/1865128942 conn(0x55b1c228eba0 msgr2=0x55b1c229a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- 172.21.15.87:0/1865128942 shutdown_connections 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- 172.21.15.87:0/1865128942 wait complete. 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- start start 2024-04-22T13:41:37.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 Processor -- start 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c22b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f91641ac090 con 0x55b1c2294e80 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f91641ad500 con 0x55b1c229da00 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f91641ac760 con 0x55b1c22944d0 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.286+0000 7f916c574740 10 client.0 _ll_register_callbacks cb 0x7f91540095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57228/0 (socket says 172.21.15.87:57228) 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 -- 172.21.15.87:0/858104657 learned_addr learned my addr 172.21.15.87:0/858104657 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9162ffd640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c22b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f91637fe640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 -- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 msgr2=0x55b1c22b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c22b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 -- 172.21.15.87:0/858104657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 msgr2=0x55b1c22b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 -- 172.21.15.87:0/858104657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c22b1270 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f915bfff640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f914c09c8a0 tx=0x7f914c0a0f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45d09421ea428dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0b5070 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f914c0bd0b0 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1c2206c10 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f914c0a5810 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f916c574740 10 client.18471 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b1c2313bb0 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.287+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b1c2206c10 con 0x55b1c2294e80 2024-04-22T13:41:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f914c0a53f0 con 0x55b1c2294e80 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 10 client.18471 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 10 client.18471 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 20 client.18471 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c2316280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 4 client.18471 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1c2200a80 con 0x55b1c22ada00 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9162ffd640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c2316280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9142ffd640 10 client.18471 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9162ffd640 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c2316280 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca8e9e27882b4e2e server_cookie=101441af0377dbbb in_seq=0 out_seq=0 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9142ffd640 1 -- 172.21.15.87:0/858104657 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1c2200a80 con 0x55b1c22ada00 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f9142ffd640 10 client.18471 handle_command_reply: tid=0 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 1 client.18471 shutdown 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 msgr2=0x55b1c22b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.288+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f914c09c8a0 tx=0x7f914c0a0f50 comp rx=0 tx=0).stop 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 shutdown_connections 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b1c22ada00 0x55b1c2316280 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b1c229da00 0x55b1c22b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b1c2294e80 0x55b1c22b0530 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9142ffd640 0 client.18471 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 --2- 172.21.15.87:0/858104657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1c22944d0 0x55b1c22b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 >> 172.21.15.87:0/858104657 conn(0x55b1c228eba0 msgr2=0x55b1c22a8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 shutdown_connections 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 1 -- 172.21.15.87:0/858104657 wait complete. 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f916c574740 20 client.18471 trim_cache size 0 max 16384 2024-04-22T13:41:37.292 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:37.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 msgr2=0x7f91500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 0x7f91500a61e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f915c009fd0 tx=0x7f915c004ab0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 msgr2=0x7f9164198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e9c0 tx=0x7f914c00ee80 comp rx=0 tx=0).stop 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 shutdown_connections 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f916419cff0 0x7f916419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9164107270 0x7f9164198c70 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 --2- 172.21.15.87:0/3677437012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f916406a880 0x7f9164198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 >> 172.21.15.87:0/3677437012 conn(0x7f9164077eb0 msgr2=0x7f9164108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 shutdown_connections 2024-04-22T13:41:37.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.289+0000 7f9169985640 1 -- 172.21.15.87:0/3677437012 wait complete. 2024-04-22T13:41:37.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 -- 172.21.15.87:0/515708582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60102b40 msgr2=0x7fbb60106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 --2- 172.21.15.87:0/515708582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60102b40 0x7fbb60106b80 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009900 tx=0x7fbb5002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 -- 172.21.15.87:0/515708582 shutdown_connections 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 --2- 172.21.15.87:0/515708582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60107260 0x7fbb6010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 --2- 172.21.15.87:0/515708582 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60102b40 0x7fbb60106b80 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 --2- 172.21.15.87:0/515708582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60102190 0x7fbb60102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 -- 172.21.15.87:0/515708582 >> 172.21.15.87:0/515708582 conn(0x7fbb600793a0 msgr2=0x7fbb600797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 -- 172.21.15.87:0/515708582 shutdown_connections 2024-04-22T13:41:37.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.400+0000 7fbb64855640 1 -- 172.21.15.87:0/515708582 wait complete. 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 Processor -- start 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 -- start start 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 0x7fbb6019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60198600 0x7fbb60198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60112180 con 0x7fbb60107260 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60112000 con 0x7fbb60102190 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb64855640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60112300 con 0x7fbb60198600 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57236/0 (socket says 172.21.15.87:57236) 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 -- 172.21.15.87:0/1465398193 learned_addr learned my addr 172.21.15.87:0/1465398193 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 0x7fbb6019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 -- 172.21.15.87:0/1465398193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60198600 msgr2=0x7fbb60198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60198600 0x7fbb60198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60198600 0x7fbb60198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 -- 172.21.15.87:0/1465398193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 msgr2=0x7fbb6019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 0x7fbb6019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 -- 172.21.15.87:0/1465398193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb60199120 con 0x7fbb60107260 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 0x7fbb6019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fbb50036040 tx=0x7fbb5002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.401+0000 7fbb4f7fe640 1 -- 172.21.15.87:0/1465398193 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50043020 con 0x7fbb60107260 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.402+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb601993b0 con 0x7fbb60107260 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.402+0000 7fbb4f7fe640 1 -- 172.21.15.87:0/1465398193 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb5002cbe0 con 0x7fbb60107260 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.402+0000 7fbb4f7fe640 1 -- 172.21.15.87:0/1465398193 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50003bf0 con 0x7fbb60107260 2024-04-22T13:41:37.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.402+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb601a55b0 con 0x7fbb60107260 2024-04-22T13:41:37.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb4f7fe640 1 -- 172.21.15.87:0/1465398193 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb50046020 con 0x7fbb60107260 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 Processor -- start 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 -- start start 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cd48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 0x55fe31cdfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fe31c391f0 con 0x55fe31cd4e80 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe31bd64b0 con 0x55fe31cdda00 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.404+0000 7fbb67449740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe31c351b0 con 0x55fe31cd44d0 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb4f7fe640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 0x7fbb480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57248/0 (socket says 172.21.15.87:57248) 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 -- 172.21.15.87:0/2715030413 learned_addr learned my addr 172.21.15.87:0/2715030413 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cd48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 0x7fbb480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb4f7fe640 1 -- 172.21.15.87:0/1465398193 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbb500f2a10 con 0x7fbb60107260 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 0x55fe31cdfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 -- 172.21.15.87:0/2715030413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 msgr2=0x55fe31cd48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cd48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 -- 172.21.15.87:0/2715030413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 msgr2=0x55fe31cdfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 0x55fe31cdfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 -- 172.21.15.87:0/2715030413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe31bc81b0 con 0x55fe31cd4e80 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cd48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 0x55fe31cdfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb5400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dc01623c4e356f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 0x7fbb480a61e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800afa0 tx=0x7fbb58005c80 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb4cff9640 1 -- 172.21.15.87:0/2715030413 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54016070 con 0x55fe31cd4e80 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb4cff9640 1 -- 172.21.15.87:0/2715030413 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb540054d0 con 0x55fe31cd4e80 2024-04-22T13:41:37.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.405+0000 7fbb4cff9640 1 -- 172.21.15.87:0/2715030413 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54011470 con 0x55fe31cd4e80 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- 172.21.15.87:0/2715030413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 msgr2=0x55fe31cdd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb5400e6c0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- 172.21.15.87:0/2715030413 shutdown_connections 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cdda00 0x55fe31cdfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd4e80 0x55fe31cdd430 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 --2- 172.21.15.87:0/2715030413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cd48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- 172.21.15.87:0/2715030413 >> 172.21.15.87:0/2715030413 conn(0x55fe31cceba0 msgr2=0x55fe31ccef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- 172.21.15.87:0/2715030413 shutdown_connections 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- 172.21.15.87:0/2715030413 wait complete. 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.406+0000 7fbb67449740 1 -- start start 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 Processor -- start 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cdd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cda660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fe31ce0970 con 0x55fe31cdda00 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe31bf2500 con 0x55fe31cd4e80 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe31ce0af0 con 0x55fe31cd44d0 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb67449740 10 client.0 _ll_register_callbacks cb 0x55fe31bc0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57258/0 (socket says 172.21.15.87:57258) 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/1492376312 learned_addr learned my addr 172.21.15.87:0/1492376312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cdd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cda660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 msgr2=0x55fe31cdd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cdd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/1492376312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 msgr2=0x55fe31cda660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cda660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/1492376312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3194b3f0 con 0x55fe31cdda00 2024-04-22T13:41:37.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cda660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cdd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800fbe0 tx=0x7fbb58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28047af9dc39e4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.407+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58004b20 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbb58004cc0 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb540054d0 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb58005020 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 10 client.18483 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe31a157d0 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb540054d0 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb58019070 con 0x55fe31cdda00 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 10 client.18483 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 10 client.18483 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 20 client.18483 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31cedb40 0x55fe31ceffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 4 client.18483 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe31bd8930 con 0x55fe31cedb40 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.408+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31cedb40 0x55fe31ceffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.409+0000 7fbb45ffb640 10 client.18483 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.409+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31cedb40 0x55fe31ceffc0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c90a963f07f509eb server_cookie=793d0f5d4a850e22 in_seq=0 out_seq=0 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.409+0000 7fbb45ffb640 1 -- 172.21.15.87:0/1492376312 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe31bd8930 con 0x55fe31cedb40 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.409+0000 7fbb45ffb640 10 client.18483 handle_command_reply: tid=0 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.409+0000 7fbb67449740 1 client.18483 shutdown 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 msgr2=0x55fe31cdabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800fbe0 tx=0x7fbb58004a40 comp rx=0 tx=0).stop 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 shutdown_connections 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31cedb40 0x55fe31ceffc0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdabc0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cda660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb45ffb640 0 client.18483 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:37.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 --2- 172.21.15.87:0/1492376312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31cdd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 >> 172.21.15.87:0/1492376312 conn(0x55fe31cceba0 msgr2=0x55fe31cd08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 shutdown_connections 2024-04-22T13:41:37.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 1 -- 172.21.15.87:0/1492376312 wait complete. 2024-04-22T13:41:37.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.410+0000 7fbb67449740 20 client.18483 trim_cache size 0 max 16384 2024-04-22T13:41:37.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 Processor -- start 2024-04-22T13:41:37.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 -- start start 2024-04-22T13:41:37.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31ce9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601b0430 con 0x55fe31cdda00 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601add00 con 0x55fe31cd4e80 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb67449740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601abb70 con 0x55fe31cd44d0 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57260/0 (socket says 172.21.15.87:57260) 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/3336727437 learned_addr learned my addr 172.21.15.87:0/3336727437 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/3336727437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 msgr2=0x55fe31ce9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31ce9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/3336727437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 msgr2=0x55fe31cd0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5ed9d640 1 -- 172.21.15.87:0/3336727437 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe31cdaf30 con 0x55fe31cdda00 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.423+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbb58012cc0 tx=0x7fbb5800ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4db842f7c1c411e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb47fff640 1 -- 172.21.15.87:0/3336727437 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58034070 con 0x55fe31cdda00 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb47fff640 1 -- 172.21.15.87:0/3336727437 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb58029020 con 0x55fe31cdda00 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb47fff640 1 -- 172.21.15.87:0/3336727437 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5802ed00 con 0x55fe31cdda00 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- 172.21.15.87:0/3336727437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 msgr2=0x55fe31cdcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbb58012cc0 tx=0x7fbb5800ae40 comp rx=0 tx=0).stop 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- 172.21.15.87:0/3336727437 shutdown_connections 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cdda00 0x55fe31cdcd90 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 --2- 172.21.15.87:0/3336727437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cd44d0 0x55fe31ce9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- 172.21.15.87:0/3336727437 >> 172.21.15.87:0/3336727437 conn(0x55fe31cceba0 msgr2=0x55fe31ccff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- 172.21.15.87:0/3336727437 shutdown_connections 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- 172.21.15.87:0/3336727437 wait complete. 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.424+0000 7fbb67449740 1 -- start start 2024-04-22T13:41:37.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 Processor -- start 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 0x55fe31cd7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601ac1c0 con 0x55fe31cd44d0 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601ad630 con 0x55fe31cd4e80 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601ac890 con 0x55fe31cdda00 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 10 client.0 _ll_register_callbacks cb 0x7fbb5800f860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57266/0 (socket says 172.21.15.87:57266) 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 -- 172.21.15.87:0/3291916175 learned_addr learned my addr 172.21.15.87:0/3291916175 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 0x55fe31cd7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 -- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 msgr2=0x55fe31cd7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 0x55fe31cd7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 -- 172.21.15.87:0/3291916175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 msgr2=0x55fe31cd7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 -- 172.21.15.87:0/3291916175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe31cd7da0 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5ed9d640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 0x55fe31cd7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb5e59c640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb54009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d86216cac1edf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54016040 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fbb54004220 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.425+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb58029020 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb540098a0 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 10 client.18489 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe31d54090 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb58029020 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb5400f040 con 0x55fe31cd44d0 2024-04-22T13:41:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 10 client.18489 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 10 client.18489 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 20 client.18489 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31d546a0 0x55fe31d56ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 4 client.18489 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe31c46c10 con 0x55fe31d546a0 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.426+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31d546a0 0x55fe31d56ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb5dd9b640 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31d546a0 0x55fe31d56ac0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=129f2b31d51d6735 server_cookie=a4524ce71d272584 in_seq=0 out_seq=0 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb45ffb640 10 client.18489 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb45ffb640 1 -- 172.21.15.87:0/3291916175 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe31c46c10 con 0x55fe31d546a0 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb45ffb640 10 client.18489 handle_command_reply: tid=0 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 client.18489 shutdown 2024-04-22T13:41:37.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 msgr2=0x55fe31ce6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fbb54005fd0 tx=0x7fbb54009560 comp rx=0 tx=0).stop 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 shutdown_connections 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fe31d546a0 0x55fe31d56ac0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe31cdda00 0x55fe31cd7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe31cd4e80 0x55fe31cd7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 --2- 172.21.15.87:0/3291916175 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fe31cd44d0 0x55fe31ce6110 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb45ffb640 0 client.18489 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 >> 172.21.15.87:0/3291916175 conn(0x55fe31cceba0 msgr2=0x55fe31ccdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 shutdown_connections 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 1 -- 172.21.15.87:0/3291916175 wait complete. 2024-04-22T13:41:37.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb67449740 20 client.18489 trim_cache size 0 max 16384 2024-04-22T13:41:37.431 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:37.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 msgr2=0x7fbb480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 0x7fbb480a61e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800afa0 tx=0x7fbb58005c80 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 msgr2=0x7fbb6019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.427+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fbb50036040 tx=0x7fbb5002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 shutdown_connections 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb480a3d20 0x7fbb480a61e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb60198600 0x7fbb60198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb60107260 0x7fbb6019e940 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 --2- 172.21.15.87:0/1465398193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb60102190 0x7fbb6019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 >> 172.21.15.87:0/1465398193 conn(0x7fbb600793a0 msgr2=0x7fbb60100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 shutdown_connections 2024-04-22T13:41:37.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.428+0000 7fbb64855640 1 -- 172.21.15.87:0/1465398193 wait complete. 2024-04-22T13:41:37.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:37 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4164461853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:37 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4230100645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- 172.21.15.87:0/386153244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c105e20 msgr2=0x7f335c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 --2- 172.21.15.87:0/386153244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c105e20 0x7f335c106200 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f3350009930 tx=0x7f335002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- 172.21.15.87:0/386153244 shutdown_connections 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 --2- 172.21.15.87:0/386153244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 0x7f335c10dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 --2- 172.21.15.87:0/386153244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c101990 0x7f335c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 --2- 172.21.15.87:0/386153244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c105e20 0x7f335c106200 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- 172.21.15.87:0/386153244 >> 172.21.15.87:0/386153244 conn(0x7f335c0760e0 msgr2=0x7f335c0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- 172.21.15.87:0/386153244 shutdown_connections 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- 172.21.15.87:0/386153244 wait complete. 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 Processor -- start 2024-04-22T13:41:37.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.537+0000 7f33619a4640 1 -- start start 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 0x7f335c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c105e20 0x7f335c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1110a0 con 0x7f335c101990 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f335c110f20 con 0x7f335c102350 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f335c111220 con 0x7f335c105e20 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57282/0 (socket says 172.21.15.87:57282) 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 -- 172.21.15.87:0/1054113984 learned_addr learned my addr 172.21.15.87:0/1054113984 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335a7fc640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 0x7f335c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c105e20 0x7f335c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 -- 172.21.15.87:0/1054113984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c105e20 msgr2=0x7f335c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c105e20 0x7f335c19cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 -- 172.21.15.87:0/1054113984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 msgr2=0x7f335c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 0x7f335c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 -- 172.21.15.87:0/1054113984 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c19d690 con 0x7f335c101990 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f335affd640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f3350000c00 tx=0x7f3350004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33609a2640 1 -- 172.21.15.87:0/1054113984 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350043020 con 0x7f335c101990 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33609a2640 1 -- 172.21.15.87:0/1054113984 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33500052b0 con 0x7f335c101990 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c19d920 con 0x7f335c101990 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33609a2640 1 -- 172.21.15.87:0/1054113984 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350003c10 con 0x7f335c101990 2024-04-22T13:41:37.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.538+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c1a5210 con 0x7f335c101990 2024-04-22T13:41:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.540+0000 7f33609a2640 1 -- 172.21.15.87:0/1054113984 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3350046020 con 0x7f335c101990 2024-04-22T13:41:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 Processor -- start 2024-04-22T13:41:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 -- start start 2024-04-22T13:41:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7d48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 0x564c2f7dfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564c2f7391f0 con 0x564c2f7d4e80 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c2f6d64b0 con 0x564c2f7dda00 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f3364594740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564c2f7351b0 con 0x564c2f7d44d0 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57284/0 (socket says 172.21.15.87:57284) 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 -- 172.21.15.87:0/1544304832 learned_addr learned my addr 172.21.15.87:0/1544304832 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f33609a2640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 0x7f33440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7d48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335a7fc640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 0x564c2f7dfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 0x7f33440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f33609a2640 1 -- 172.21.15.87:0/1054113984 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f33500f2310 con 0x7f335c101990 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 -- 172.21.15.87:0/1544304832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 msgr2=0x564c2f7d48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7d48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 -- 172.21.15.87:0/1544304832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 msgr2=0x564c2f7dfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 0x564c2f7dfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335affd640 1 -- 172.21.15.87:0/1544304832 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c2f6c81b0 con 0x564c2f7d4e80 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.541+0000 7f335a7fc640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 0x564c2f7dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 0x7f33440a61e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f335400afd0 tx=0x7f3354005e90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f335affd640 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f33500be100 tx=0x7f335002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d01e12778d7815 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3341ffb640 1 -- 172.21.15.87:0/1544304832 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500be1d0 con 0x564c2f7d4e80 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3341ffb640 1 -- 172.21.15.87:0/1544304832 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33500035f0 con 0x564c2f7d4e80 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3341ffb640 1 -- 172.21.15.87:0/1544304832 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33500bdd60 con 0x564c2f7d4e80 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 -- 172.21.15.87:0/1544304832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 msgr2=0x564c2f7dd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f33500be100 tx=0x7f335002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 -- 172.21.15.87:0/1544304832 shutdown_connections 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7dda00 0x564c2f7dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dd430 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 --2- 172.21.15.87:0/1544304832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7d48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 -- 172.21.15.87:0/1544304832 >> 172.21.15.87:0/1544304832 conn(0x564c2f7ceba0 msgr2=0x564c2f7cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.542+0000 7f3364594740 1 -- 172.21.15.87:0/1544304832 shutdown_connections 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 -- 172.21.15.87:0/1544304832 wait complete. 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 -- start start 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 Processor -- start 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7dd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d4e80 0x564c2f7da6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564c2f7391f0 con 0x564c2f7dda00 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c2f6f2500 con 0x564c2f7d4e80 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564c2f7e0730 con 0x564c2f7d44d0 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f3364594740 10 client.0 _ll_register_callbacks cb 0x564c2f6c0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f335b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f335b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57292/0 (socket says 172.21.15.87:57292) 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f335b7fe640 1 -- 172.21.15.87:0/1336004762 learned_addr learned my addr 172.21.15.87:0/1336004762 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f335a7fc640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d4e80 0x564c2f7da6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.543+0000 7f335affd640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7dd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 -- 172.21.15.87:0/1336004762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 msgr2=0x564c2f7dd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7dd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 -- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d4e80 msgr2=0x564c2f7da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d4e80 0x564c2f7da6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 -- 172.21.15.87:0/1336004762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2f44b3f0 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f335b7fe640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f335400fa80 tx=0x7f3354004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c72dcd3a067c8a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354004d20 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f335401ca40 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564c2f6d8930 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f335401cc40 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 10 client.18501 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c2f5157d0 con 0x564c2f7dda00 2024-04-22T13:41:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564c2f6d8930 con 0x564c2f7dda00 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f335400d3e0 con 0x564c2f7dda00 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 10 client.18501 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 10 client.18501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.544+0000 7f3364594740 20 client.18501 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f7ed870 0x564c2f7efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f3364594740 4 client.18501 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c2f746c10 con 0x564c2f7ed870 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f335affd640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f7ed870 0x564c2f7efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f333affd640 10 client.18501 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.545+0000 7f335affd640 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f7ed870 0x564c2f7efcb0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e31466051431033 server_cookie=6e59d7be5f609201 in_seq=0 out_seq=0 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f333affd640 1 -- 172.21.15.87:0/1336004762 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c2f746c10 con 0x564c2f7ed870 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f333affd640 10 client.18501 handle_command_reply: tid=0 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 client.18501 shutdown 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 msgr2=0x564c2f7dac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f335400fa80 tx=0x7f3354004a40 comp rx=0 tx=0).stop 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 shutdown_connections 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f7ed870 0x564c2f7efcb0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7dac50 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f333affd640 0 client.18501 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d4e80 0x564c2f7da6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 --2- 172.21.15.87:0/1336004762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d44d0 0x564c2f7dd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 >> 172.21.15.87:0/1336004762 conn(0x564c2f7ceba0 msgr2=0x564c2f7d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 shutdown_connections 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 1 -- 172.21.15.87:0/1336004762 wait complete. 2024-04-22T13:41:37.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.546+0000 7f3364594740 20 client.18501 trim_cache size 0 max 16384 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.559+0000 7f3364594740 1 Processor -- start 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 -- start start 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7dda00 0x564c2f7dd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1b03a0 con 0x564c2f7d4e80 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1adc70 con 0x564c2f7d44d0 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f3364594740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1abae0 con 0x564c2f7dda00 2024-04-22T13:41:37.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57294/0 (socket says 172.21.15.87:57294) 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 -- 172.21.15.87:0/764228084 learned_addr learned my addr 172.21.15.87:0/764228084 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335affd640 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 -- 172.21.15.87:0/764228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7dda00 msgr2=0x564c2f7dd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7dda00 0x564c2f7dd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 -- 172.21.15.87:0/764228084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 msgr2=0x564c2f7e93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7e93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 -- 172.21.15.87:0/764228084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c2f7d4c20 con 0x564c2f7d4e80 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335affd640 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f335a7fc640 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f334c00bd20 tx=0x7f334c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12d7187183e3caa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f33417fa640 1 -- 172.21.15.87:0/764228084 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c00be10 con 0x564c2f7d4e80 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f33417fa640 1 -- 172.21.15.87:0/764228084 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c0054d0 con 0x564c2f7d4e80 2024-04-22T13:41:37.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.560+0000 7f33417fa640 1 -- 172.21.15.87:0/764228084 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c01c6d0 con 0x564c2f7d4e80 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- 172.21.15.87:0/764228084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 msgr2=0x564c2f7dcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f334c00bd20 tx=0x7f334c012820 comp rx=0 tx=0).stop 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- 172.21.15.87:0/764228084 shutdown_connections 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7dda00 0x564c2f7dd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7d4e80 0x564c2f7dcde0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 --2- 172.21.15.87:0/764228084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- 172.21.15.87:0/764228084 >> 172.21.15.87:0/764228084 conn(0x564c2f7ceba0 msgr2=0x564c2f7cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- 172.21.15.87:0/764228084 shutdown_connections 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- 172.21.15.87:0/764228084 wait complete. 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 -- start start 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.561+0000 7f3364594740 1 Processor -- start 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 0x564c2f7d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ac130 con 0x564c2f7dda00 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ad5a0 con 0x564c2f7d44d0 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ac800 con 0x564c2f7d4e80 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 10 client.0 _ll_register_callbacks cb 0x7f3354012780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57302/0 (socket says 172.21.15.87:57302) 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 -- 172.21.15.87:0/183093811 learned_addr learned my addr 172.21.15.87:0/183093811 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335a7fc640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 0x564c2f7d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335affd640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 -- 172.21.15.87:0/183093811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 msgr2=0x564c2f7d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 0x564c2f7d7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 -- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 msgr2=0x564c2f7d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7d7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 -- 172.21.15.87:0/183093811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2f852ff0 con 0x564c2f7dda00 2024-04-22T13:41:37.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335a7fc640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 0x564c2f7d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335affd640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f335b7fe640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f335400d9f0 tx=0x7f335400d690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5032abf7200edc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f333affd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335400dde0 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f3354004bd0 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f334c0054d0 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3354004dd0 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 10 client.18507 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.562+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c2f854220 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c0054d0 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3354037030 con 0x564c2f7dda00 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 10 client.18507 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 10 client.18507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:37.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 20 client.18507 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f854920 0x564c2f856d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 4 client.18507 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c2f740a80 con 0x564c2f854920 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f335affd640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f854920 0x564c2f856d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f333affd640 10 client.18507 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.563+0000 7f335affd640 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f854920 0x564c2f856d40 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e8d804f3d912d11 server_cookie=d961e39bedc78b5 in_seq=0 out_seq=0 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f333affd640 1 -- 172.21.15.87:0/183093811 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c2f740a80 con 0x564c2f854920 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f333affd640 10 client.18507 handle_command_reply: tid=0 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 client.18507 shutdown 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 msgr2=0x564c2f7d7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f335400d9f0 tx=0x7f335400d690 comp rx=0 tx=0).stop 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 shutdown_connections 2024-04-22T13:41:37.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564c2f854920 0x564c2f856d40 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564c2f7dda00 0x564c2f7d7fc0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564c2f7d4e80 0x564c2f7d7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f333affd640 0 client.18507 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 --2- 172.21.15.87:0/183093811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c2f7d44d0 0x564c2f7d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 >> 172.21.15.87:0/183093811 conn(0x564c2f7ceba0 msgr2=0x564c2f7d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 shutdown_connections 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 1 -- 172.21.15.87:0/183093811 wait complete. 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f3364594740 20 client.18507 trim_cache size 0 max 16384 2024-04-22T13:41:37.567 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 msgr2=0x7f33440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 0x7f33440a61e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f335400afd0 tx=0x7f3354005e90 comp rx=0 tx=0).stop 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 msgr2=0x7f335c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f3350000c00 tx=0x7f3350004060 comp rx=0 tx=0).stop 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.564+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 shutdown_connections 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f33440a3d20 0x7f33440a61e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f335c105e20 0x7f335c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f335c102350 0x7f335c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 --2- 172.21.15.87:0/1054113984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f335c101990 0x7f335c1986f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 >> 172.21.15.87:0/1054113984 conn(0x7f335c0760e0 msgr2=0x7f335c104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 shutdown_connections 2024-04-22T13:41:37.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:37.565+0000 7f33619a4640 1 -- 172.21.15.87:0/1054113984 wait complete. 2024-04-22T13:41:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:37 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4164461853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:37 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4230100645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:37 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4164461853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:37 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4230100645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 -- 172.21.15.87:0/4195519297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4106fb0 msgr2=0x7f1ea4107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/4195519297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4106fb0 0x7f1ea4107390 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009900 tx=0x7f1e8c02bc80 comp rx=0 tx=0).stop 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 -- 172.21.15.87:0/4195519297 shutdown_connections 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/4195519297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea4101b60 0x7f1ea410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/4195519297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea41011a0 0x7f1ea4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/4195519297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4106fb0 0x7f1ea4107390 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 -- 172.21.15.87:0/4195519297 >> 172.21.15.87:0/4195519297 conn(0x7f1ea4076c50 msgr2=0x7f1ea4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 -- 172.21.15.87:0/4195519297 shutdown_connections 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 -- 172.21.15.87:0/4195519297 wait complete. 2024-04-22T13:41:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.752+0000 7f1ea92c7640 1 Processor -- start 2024-04-22T13:41:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 -- start start 2024-04-22T13:41:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea41011a0 0x7f1ea4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea4106fb0 0x7f1ea41a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea4111090 con 0x7f1ea4101b60 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea4110f10 con 0x7f1ea4106fb0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea92c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea4111210 con 0x7f1ea41011a0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57326/0 (socket says 172.21.15.87:57326) 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 -- 172.21.15.87:0/1827689995 learned_addr learned my addr 172.21.15.87:0/1827689995 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea41011a0 0x7f1ea4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea4106fb0 0x7f1ea41a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 -- 172.21.15.87:0/1827689995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea41011a0 msgr2=0x7f1ea4199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea41011a0 0x7f1ea4199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 -- 172.21.15.87:0/1827689995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea4106fb0 msgr2=0x7f1ea41a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea4106fb0 0x7f1ea41a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 -- 172.21.15.87:0/1827689995 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea41a49d0 con 0x7f1ea4101b60 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.753+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400d950 tx=0x7f1e9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.754+0000 7f1e9bfff640 1 -- 172.21.15.87:0/1827689995 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94017070 con 0x7f1ea4101b60 2024-04-22T13:41:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.754+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea41a4ba0 con 0x7f1ea4101b60 2024-04-22T13:41:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.754+0000 7f1e9bfff640 1 -- 172.21.15.87:0/1827689995 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e94009c30 con 0x7f1ea4101b60 2024-04-22T13:41:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.754+0000 7f1e9bfff640 1 -- 172.21.15.87:0/1827689995 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9400f620 con 0x7f1ea4101b60 2024-04-22T13:41:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.754+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea41a50c0 con 0x7f1ea4101b60 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.756+0000 7f1e9bfff640 1 -- 172.21.15.87:0/1827689995 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e94010460 con 0x7f1ea4101b60 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.756+0000 7f1eabf8a740 1 Processor -- start 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.756+0000 7f1eabf8a740 1 -- start start 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fc9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fd30e0 0x562e84fd54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562e84f2e8d0 con 0x562e84fca8d0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e84ecbe80 con 0x562e84fd30e0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e84f2a890 con 0x562e84fc9b60 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1e9bfff640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 0x7f1e900a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57328/0 (socket says 172.21.15.87:57328) 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 -- 172.21.15.87:0/3389906371 learned_addr learned my addr 172.21.15.87:0/3389906371 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fc9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 0x7f1e900a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1e9bfff640 1 -- 172.21.15.87:0/1827689995 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f1e940d0420 con 0x7f1ea4101b60 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fd30e0 0x562e84fd54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 -- 172.21.15.87:0/3389906371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 msgr2=0x562e84fc9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fc9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 -- 172.21.15.87:0/3389906371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fd30e0 msgr2=0x562e84fd54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fd30e0 0x562e84fd54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 -- 172.21.15.87:0/3389906371 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e84eed320 con 0x562e84fca8d0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fc9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c012c50 tx=0x7f1e9c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70873fc43cbb7653 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 0x7f1e900a6280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c010c40 tx=0x7f1e9c014000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.757+0000 7f1e997fa640 1 -- 172.21.15.87:0/3389906371 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c022070 con 0x562e84fca8d0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1e997fa640 1 -- 172.21.15.87:0/3389906371 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e9c00cab0 con 0x562e84fca8d0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1e997fa640 1 -- 172.21.15.87:0/3389906371 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0053c0 con 0x562e84fca8d0 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3389906371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 msgr2=0x562e84fc9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c012c50 tx=0x7f1e9c010740 comp rx=0 tx=0).stop 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3389906371 shutdown_connections 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fd30e0 0x562e84fd54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fc9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3389906371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fc9620 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3389906371 >> 172.21.15.87:0/3389906371 conn(0x562e84f90c90 msgr2=0x562e84fc4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3389906371 shutdown_connections 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.758+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3389906371 wait complete. 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 -- start start 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 Processor -- start 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fd2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fcfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562e84fd5db0 con 0x562e84fd30e0 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e84eb6eb0 con 0x562e84fca8d0 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e84fd5f30 con 0x562e84fc9b60 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1eabf8a740 10 client.0 _ll_register_callbacks cb 0x562e84ee3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57342/0 (socket says 172.21.15.87:57342) 2024-04-22T13:41:37.761 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 -- 172.21.15.87:0/770500593 learned_addr learned my addr 172.21.15.87:0/770500593 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fcfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fd2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 -- 172.21.15.87:0/770500593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 msgr2=0x562e84fd2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fd2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 -- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 msgr2=0x562e84fcfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fcfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 -- 172.21.15.87:0/770500593 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e84c41210 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00ce90 tx=0x7f1e9c012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af63fb085c5eab0e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.759+0000 7f1e8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c022040 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f1e9c0058d0 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e9c00cab0 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1e9c005ad0 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 10 client.18519 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562e84d0d620 con 0x562e84fd30e0 2024-04-22T13:41:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e9c00cab0 con 0x562e84fd30e0 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1e9c02d070 con 0x562e84fd30e0 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 10 client.18519 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 10 client.18519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 20 client.18519 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e84fe3080 0x562e84fe5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 4 client.18519 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e84f42880 con 0x562e84fe3080 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.760+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e84fe3080 0x562e84fe5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.761+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e84fe3080 0x562e84fe5470 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8286fe6c4f3920a6 server_cookie=6e2dd6714dfdca92 in_seq=0 out_seq=0 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.761+0000 7f1e8a7fc640 10 client.18519 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.761+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/770500593 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e84f42880 con 0x562e84fe3080 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.761+0000 7f1e8a7fc640 10 client.18519 handle_command_reply: tid=0 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.761+0000 7f1eabf8a740 1 client.18519 shutdown 2024-04-22T13:41:37.763 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 msgr2=0x562e84fd0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00ce90 tx=0x7f1e9c012510 comp rx=0 tx=0).stop 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 shutdown_connections 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e84fe3080 0x562e84fe5470 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fd30e0 0x562e84fd0460 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fcfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1e8a7fc640 0 client.18519 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/770500593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fc9b60 0x562e84fd2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 >> 172.21.15.87:0/770500593 conn(0x562e84f90c90 msgr2=0x562e84fc65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 shutdown_connections 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 1 -- 172.21.15.87:0/770500593 wait complete. 2024-04-22T13:41:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.762+0000 7f1eabf8a740 20 client.18519 trim_cache size 0 max 16384 2024-04-22T13:41:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 Processor -- start 2024-04-22T13:41:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 -- start start 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fd2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fd2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41b02c0 con 0x562e84fc9b60 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41adb90 con 0x562e84fca8d0 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.776+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41aba00 con 0x562e84fd30e0 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57350/0 (socket says 172.21.15.87:57350) 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 -- 172.21.15.87:0/674349639 learned_addr learned my addr 172.21.15.87:0/674349639 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.777 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fd2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 -- 172.21.15.87:0/674349639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 msgr2=0x562e84fd2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fd2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 -- 172.21.15.87:0/674349639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 msgr2=0x562e84fd2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fd2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 -- 172.21.15.87:0/674349639 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e84fc8e30 con 0x562e84fc9b60 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fd2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009900 tx=0x7f1e8c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3533f2cef670f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1e98ff9640 1 -- 172.21.15.87:0/674349639 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c004280 con 0x562e84fc9b60 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1e98ff9640 1 -- 172.21.15.87:0/674349639 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e8c02cc80 con 0x562e84fc9b60 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.777+0000 7f1e98ff9640 1 -- 172.21.15.87:0/674349639 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0046c0 con 0x562e84fc9b60 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- 172.21.15.87:0/674349639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 msgr2=0x562e84fdecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009900 tx=0x7f1e8c005fd0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- 172.21.15.87:0/674349639 shutdown_connections 2024-04-22T13:41:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fd2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fca8d0 0x562e84fd2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/674349639 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fc9b60 0x562e84fdecb0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- 172.21.15.87:0/674349639 >> 172.21.15.87:0/674349639 conn(0x562e84f90c90 msgr2=0x562e84fceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- 172.21.15.87:0/674349639 shutdown_connections 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- 172.21.15.87:0/674349639 wait complete. 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- start start 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 Processor -- start 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fc9b60 0x562e84fdb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fcca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41ac050 con 0x562e84fca8d0 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41ad4c0 con 0x562e84fc9b60 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea41ac720 con 0x562e84fd30e0 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1eabf8a740 10 client.0 _ll_register_callbacks cb 0x7f1e9400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1ea27fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1ea27fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57354/0 (socket says 172.21.15.87:57354) 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.778+0000 7f1ea27fc640 1 -- 172.21.15.87:0/3158245655 learned_addr learned my addr 172.21.15.87:0/3158245655 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fcca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fc9b60 0x562e84fdb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 -- 172.21.15.87:0/3158245655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 msgr2=0x562e84fcca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fcca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 -- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fc9b60 msgr2=0x562e84fdb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fc9b60 0x562e84fdb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 -- 172.21.15.87:0/3158245655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e84fccf70 con 0x562e84fca8d0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea37fe640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fcca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1ea27fc640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f1e9409c100 tx=0x7f1e9409be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57aeda27522ea675 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94020040 con 0x562e84fca8d0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f1e940b5b80 con 0x562e84fca8d0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e8c02cc80 con 0x562e84fca8d0 2024-04-22T13:41:37.780 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1e940a07b0 con 0x562e84fca8d0 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 10 client.18525 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562e85049140 con 0x562e84fca8d0 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e8c02cc80 con 0x562e84fca8d0 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1e94098150 con 0x562e84fca8d0 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 10 client.18525 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 10 client.18525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 20 client.18525 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e85049940 0x562e8504bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 4 client.18525 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.779+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e84ece3d0 con 0x562e85049940 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.780+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e85049940 0x562e8504bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.780+0000 7f1e8a7fc640 10 client.18525 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.780+0000 7f1ea2ffd640 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e85049940 0x562e8504bd60 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c38c3f14cad5127f server_cookie=d11d3375e3c9a919 in_seq=0 out_seq=0 2024-04-22T13:41:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.785+0000 7f1e8a7fc640 1 -- 172.21.15.87:0/3158245655 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e84ece3d0 con 0x562e85049940 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.785+0000 7f1e8a7fc640 10 client.18525 handle_command_reply: tid=0 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 client.18525 shutdown 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 msgr2=0x562e84fcc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f1e9409c100 tx=0x7f1e9409be90 comp rx=0 tx=0).stop 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 shutdown_connections 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562e85049940 0x562e8504bd60 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e84fd30e0 0x562e84fcca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1e8a7fc640 0 client.18525 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562e84fca8d0 0x562e84fcc4f0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 --2- 172.21.15.87:0/3158245655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e84fc9b60 0x562e84fdb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 >> 172.21.15.87:0/3158245655 conn(0x562e84f90c90 msgr2=0x562e84fc5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 shutdown_connections 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 1 -- 172.21.15.87:0/3158245655 wait complete. 2024-04-22T13:41:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1eabf8a740 20 client.18525 trim_cache size 0 max 16384 2024-04-22T13:41:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:41:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:41:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:41:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 msgr2=0x7f1e900a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 0x7f1e900a6280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c010c40 tx=0x7f1e9c014000 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 msgr2=0x7f1ea4199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400d950 tx=0x7f1e9400de10 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 shutdown_connections 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1e900a3dc0 0x7f1e900a6280 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ea4106fb0 0x7f1ea41a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ea4101b60 0x7f1ea4199950 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 --2- 172.21.15.87:0/1827689995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ea41011a0 0x7f1ea4199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.786+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 >> 172.21.15.87:0/1827689995 conn(0x7f1ea4076c50 msgr2=0x7f1ea40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.787+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 shutdown_connections 2024-04-22T13:41:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:37.787+0000 7f1ea92c7640 1 -- 172.21.15.87:0/1827689995 wait complete. 2024-04-22T13:41:38.572 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:41:38.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:41:38.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:41:38.597 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:38 smithi087 ceph-mon[27999]: pgmap v305: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:41:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:38 smithi120 ceph-mon[31101]: pgmap v305: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:41:38.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 -- 172.21.15.87:0/4113246138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb094107270 msgr2=0x7fb09410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/4113246138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb094107270 0x7fb09410ee40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009ca0 tx=0x7fb08c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 -- 172.21.15.87:0/4113246138 shutdown_connections 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/4113246138 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb094107270 0x7fb09410ee40 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/4113246138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb094102b50 0x7fb094106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/4113246138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0941021a0 0x7fb094102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 -- 172.21.15.87:0/4113246138 >> 172.21.15.87:0/4113246138 conn(0x7fb0940fd9c0 msgr2=0x7fb0940ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 -- 172.21.15.87:0/4113246138 shutdown_connections 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.691+0000 7fb09c6c8640 1 -- 172.21.15.87:0/4113246138 wait complete. 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 Processor -- start 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 -- start start 2024-04-22T13:41:38.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb094107270 0x7fb094198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb09419d010 0x7fb09419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941121e0 con 0x7fb0941021a0 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb094112060 con 0x7fb09419d010 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09c6c8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb094112360 con 0x7fb094107270 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09a464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09a464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57380/0 (socket says 172.21.15.87:57380) 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb09a464640 1 -- 172.21.15.87:0/2751529893 learned_addr learned my addr 172.21.15.87:0/2751529893 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.692+0000 7fb099c63640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb094107270 0x7fb094198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb09419d010 0x7fb09419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 -- 172.21.15.87:0/2751529893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb094107270 msgr2=0x7fb094198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb094107270 0x7fb094198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 -- 172.21.15.87:0/2751529893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb09419d010 msgr2=0x7fb09419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb09419d010 0x7fb09419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 -- 172.21.15.87:0/2751529893 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09419db00 con 0x7fb0941021a0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09a464640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c970 tx=0x7fb09000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb08b7fe640 1 -- 172.21.15.87:0/2751529893 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090007bb0 con 0x7fb0941021a0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb08b7fe640 1 -- 172.21.15.87:0/2751529893 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb090007d50 con 0x7fb0941021a0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09419ddf0 con 0x7fb0941021a0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb08b7fe640 1 -- 172.21.15.87:0/2751529893 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09000f450 con 0x7fb0941021a0 2024-04-22T13:41:38.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.693+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0941a5590 con 0x7fb0941021a0 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.696+0000 7fb08b7fe640 1 -- 172.21.15.87:0/2751529893 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb090004f20 con 0x7fb0941021a0 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.696+0000 7fb09f397740 1 Processor -- start 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.696+0000 7fb09f397740 1 -- start start 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.696+0000 7fb09f397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09f397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deef840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09f397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182deefe10 0x56182def2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09f397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56182de4af00 con 0x56182dee68e0 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09f397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56182dde8590 con 0x56182dee7290 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09f397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56182de46ec0 con 0x56182deefe10 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57390/0 (socket says 172.21.15.87:57390) 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 -- 172.21.15.87:0/1892099147 learned_addr learned my addr 172.21.15.87:0/1892099147 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb08b7fe640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 0x7fb0800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09a464640 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb099c63640 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182deefe10 0x56182def2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 0x7fb0800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 -- 172.21.15.87:0/1892099147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182deefe10 msgr2=0x56182def2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182deefe10 0x56182def2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 -- 172.21.15.87:0/1892099147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 msgr2=0x56182deef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 -- 172.21.15.87:0/1892099147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56182dc257e0 con 0x56182dee68e0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb08b7fe640 1 -- 172.21.15.87:0/2751529893 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb0900d0490 con 0x7fb0941021a0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb09ac65640 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009ca0 tx=0x7fb08c02cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3919d98fbbce517f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.697+0000 7fb088ff9640 1 -- 172.21.15.87:0/1892099147 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c039040 con 0x56182dee68e0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 0x7fb0800a61e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb08c04c000 tx=0x7fb08c045000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb088ff9640 1 -- 172.21.15.87:0/1892099147 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb08c003e20 con 0x56182dee68e0 2024-04-22T13:41:38.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb088ff9640 1 -- 172.21.15.87:0/1892099147 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c034440 con 0x56182dee68e0 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 -- 172.21.15.87:0/1892099147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 msgr2=0x56182dee6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009ca0 tx=0x7fb08c02cca0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 -- 172.21.15.87:0/1892099147 shutdown_connections 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182deefe10 0x56182def2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 --2- 172.21.15.87:0/1892099147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee68e0 0x56182dee6cc0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 -- 172.21.15.87:0/1892099147 >> 172.21.15.87:0/1892099147 conn(0x56182dee0fb0 msgr2=0x56182dee1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 -- 172.21.15.87:0/1892099147 shutdown_connections 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.698+0000 7fb09f397740 1 -- 172.21.15.87:0/1892099147 wait complete. 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 -- start start 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 Processor -- start 2024-04-22T13:41:38.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182deece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deed380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56182def2d80 con 0x56182deefe10 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56182de04580 con 0x56182dee7290 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56182def2f00 con 0x56182dee68e0 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09f397740 10 client.0 _ll_register_callbacks cb 0x56182de0a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57398/0 (socket says 172.21.15.87:57398) 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 -- 172.21.15.87:0/129387187 learned_addr learned my addr 172.21.15.87:0/129387187 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb099c63640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deed380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09a464640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182deece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 -- 172.21.15.87:0/129387187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 msgr2=0x56182deece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182deece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 -- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 msgr2=0x56182deed380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deed380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09ac65640 1 -- 172.21.15.87:0/129387187 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56182dc85f70 con 0x56182deefe10 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.699+0000 7fb09a464640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182deece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09ac65640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0049d0 tx=0x7fb08c032990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215e29ae548cbbd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c039070 con 0x56182deefe10 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb08c003970 con 0x56182deefe10 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb08c003e20 con 0x56182deefe10 2024-04-22T13:41:38.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb08c003b70 con 0x56182deefe10 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 10 client.18537 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56182db1d170 con 0x56182deefe10 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb08c003e20 con 0x56182deefe10 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb08c03f070 con 0x56182deefe10 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 10 client.18537 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 10 client.18537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 20 client.18537 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182df00230 0x56182df02620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 4 client.18537 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.700+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56182de58920 con 0x56182df00230 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.701+0000 7fb09a464640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182df00230 0x56182df02620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.701+0000 7fb09a464640 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182df00230 0x56182df02620 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4809c0bfd0b0c67 server_cookie=6b463ffe750f7159 in_seq=0 out_seq=0 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.701+0000 7fb07dffb640 10 client.18537 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb07dffb640 1 -- 172.21.15.87:0/129387187 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56182de58920 con 0x56182df00230 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb07dffb640 10 client.18537 handle_command_reply: tid=0 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 client.18537 shutdown 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 msgr2=0x56182deed8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0049d0 tx=0x7fb08c032990 comp rx=0 tx=0).stop 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 shutdown_connections 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182df00230 0x56182df02620 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182deefe10 0x56182deed8c0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb07dffb640 0 client.18537 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee7290 0x56182deed380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 --2- 172.21.15.87:0/129387187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182deece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 >> 172.21.15.87:0/129387187 conn(0x56182dee0fb0 msgr2=0x56182dee2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 shutdown_connections 2024-04-22T13:41:38.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 1 -- 172.21.15.87:0/129387187 wait complete. 2024-04-22T13:41:38.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.702+0000 7fb09f397740 20 client.18537 trim_cache size 0 max 16384 2024-04-22T13:41:38.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:38 smithi132 ceph-mon[31531]: pgmap v305: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 Processor -- start 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 -- start start 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 0x56182defbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee7290 0x56182deed510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941b0300 con 0x56182dee0fb0 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941adbd0 con 0x56182dee68e0 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.719+0000 7fb09f397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941aba40 con 0x56182dee7290 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57404/0 (socket says 172.21.15.87:57404) 2024-04-22T13:41:38.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 -- 172.21.15.87:0/2932257278 learned_addr learned my addr 172.21.15.87:0/2932257278 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 -- 172.21.15.87:0/2932257278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee7290 msgr2=0x56182deed510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee7290 0x56182deed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 -- 172.21.15.87:0/2932257278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 msgr2=0x56182defbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb099c63640 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 0x56182defbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 0x56182defbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 -- 172.21.15.87:0/2932257278 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56182def2ea0 con 0x56182dee0fb0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb099c63640 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 0x56182defbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb09a464640 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb09009c4b0 tx=0x7fb0900d29a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbae0e18d1b7930 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb07ffff640 1 -- 172.21.15.87:0/2932257278 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09001d040 con 0x56182dee0fb0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb07ffff640 1 -- 172.21.15.87:0/2932257278 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb09009c520 con 0x56182dee0fb0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.720+0000 7fb07ffff640 1 -- 172.21.15.87:0/2932257278 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0900cdea0 con 0x56182dee0fb0 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- 172.21.15.87:0/2932257278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 msgr2=0x56182deef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb09009c4b0 tx=0x7fb0900d29a0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- 172.21.15.87:0/2932257278 shutdown_connections 2024-04-22T13:41:38.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee7290 0x56182deed510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee68e0 0x56182defbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- 172.21.15.87:0/2932257278 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee0fb0 0x56182deef570 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- 172.21.15.87:0/2932257278 >> 172.21.15.87:0/2932257278 conn(0x56182dee79e0 msgr2=0x56182deece00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- 172.21.15.87:0/2932257278 shutdown_connections 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- 172.21.15.87:0/2932257278 wait complete. 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- start start 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 Processor -- start 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee0fb0 0x56182def8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182def86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941ac090 con 0x56182dee7290 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941ad500 con 0x56182dee0fb0 2024-04-22T13:41:38.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb0941ac760 con 0x56182dee68e0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.721+0000 7fb09f397740 10 client.0 _ll_register_callbacks cb 0x7fb08c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57416/0 (socket says 172.21.15.87:57416) 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 -- 172.21.15.87:0/2985600401 learned_addr learned my addr 172.21.15.87:0/2985600401 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb099c63640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182def86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 -- 172.21.15.87:0/2985600401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 msgr2=0x56182def86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182def86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09a464640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee0fb0 0x56182def8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 -- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee0fb0 msgr2=0x56182def8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee0fb0 0x56182def8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 -- 172.21.15.87:0/2985600401 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56182def4f30 con 0x56182dee7290 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb099c63640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182def86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09ac65640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb08c034490 tx=0x7fb08c032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2677f947833e4fed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb07dffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c061070 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fb08c004180 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb09009c520 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb08c06a050 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09f397740 10 client.18543 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56182df67080 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.722+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb09009c520 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb08c064020 con 0x56182dee7290 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 10 client.18543 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 10 client.18543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 20 client.18543 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182deefe10 0x56182df69690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 4 client.18543 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56182de52790 con 0x56182deefe10 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09a464640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182deefe10 0x56182df69690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb07dffb640 10 client.18543 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.723+0000 7fb09a464640 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182deefe10 0x56182df69690 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69cc73b634514960 server_cookie=13528c25539408d0 in_seq=0 out_seq=0 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb07dffb640 1 -- 172.21.15.87:0/2985600401 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56182de52790 con 0x56182deefe10 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb07dffb640 10 client.18543 handle_command_reply: tid=0 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 client.18543 shutdown 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 msgr2=0x56182def4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb08c034490 tx=0x7fb08c032c80 comp rx=0 tx=0).stop 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 shutdown_connections 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56182deefe10 0x56182df69690 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56182dee7290 0x56182def4730 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb07dffb640 0 client.18543 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56182dee68e0 0x56182def86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 --2- 172.21.15.87:0/2985600401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56182dee0fb0 0x56182def8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.724+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 >> 172.21.15.87:0/2985600401 conn(0x56182dee79e0 msgr2=0x56182dee0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 shutdown_connections 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09f397740 1 -- 172.21.15.87:0/2985600401 wait complete. 2024-04-22T13:41:38.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09f397740 20 client.18543 trim_cache size 0 max 16384 2024-04-22T13:41:38.730 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:38.730 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:41:38.730 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "2d33967f-6c6a-408f-8327-1baf563495d2", 2024-04-22T13:41:38.730 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:41:38.730 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 msgr2=0x7fb0800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 0x7fb0800a61e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb08c04c000 tx=0x7fb08c045000 comp rx=0 tx=0).stop 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 msgr2=0x7fb0941987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c970 tx=0x7fb09000ce30 comp rx=0 tx=0).stop 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 shutdown_connections 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb0800a3d20 0x7fb0800a61e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb09419d010 0x7fb09419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb094107270 0x7fb094198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 --2- 172.21.15.87:0/2751529893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0941021a0 0x7fb0941987a0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 >> 172.21.15.87:0/2751529893 conn(0x7fb0940fd9c0 msgr2=0x7fb0940fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 shutdown_connections 2024-04-22T13:41:38.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.725+0000 7fb09c6c8640 1 -- 172.21.15.87:0/2751529893 wait complete. 2024-04-22T13:41:38.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:41:38.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/2013799713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 msgr2=0x7fe6d4101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/2013799713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d4101470 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009900 tx=0x7fe6d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/2013799713 shutdown_connections 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/2013799713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/2013799713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d4101a40 0x7fe6d4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/2013799713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d4101470 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/2013799713 >> 172.21.15.87:0/2013799713 conn(0x7fe6d40771d0 msgr2=0x7fe6d40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/2013799713 shutdown_connections 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.877+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/2013799713 wait complete. 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 Processor -- start 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 -- start start 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 0x7fe6d419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41110a0 con 0x7fe6d4101090 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4110f20 con 0x7fe6d419ce90 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6dc0d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4111220 con 0x7fe6d4106160 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57430/0 (socket says 172.21.15.87:57430) 2024-04-22T13:41:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/3577647413 learned_addr learned my addr 172.21.15.87:0/3577647413 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d966d640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6da66f640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 0x7fe6d419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/3577647413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 msgr2=0x7fe6d4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/3577647413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 msgr2=0x7fe6d419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 0x7fe6d419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/3577647413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d419d980 con 0x7fe6d4101090 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d966d640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6da66f640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 0x7fe6d419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.878+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0005940 tx=0x7fe6d002c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.879+0000 7fe6caffd640 1 -- 172.21.15.87:0/3577647413 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0043020 con 0x7fe6d4101090 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.879+0000 7fe6caffd640 1 -- 172.21.15.87:0/3577647413 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6d002cc60 con 0x7fe6d4101090 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.879+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d419dc10 con 0x7fe6d4101090 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.879+0000 7fe6caffd640 1 -- 172.21.15.87:0/3577647413 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0003c30 con 0x7fe6d4101090 2024-04-22T13:41:38.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.879+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d41a5430 con 0x7fe6d4101090 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.881+0000 7fe6caffd640 1 -- 172.21.15.87:0/3577647413 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe6d0046020 con 0x7fe6d4101090 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.881+0000 7fe6deda6740 1 Processor -- start 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.881+0000 7fe6deda6740 1 -- start start 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.881+0000 7fe6caffd640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.881+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc32428540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3242ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6deda6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc323841e0 con 0x55fc3241f580 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6deda6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc32321390 con 0x55fc32428a80 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6deda6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc323801a0 con 0x55fc3241fea0 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6da66f640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57440/0 (socket says 172.21.15.87:57440) 2024-04-22T13:41:38.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6caffd640 1 -- 172.21.15.87:0/3577647413 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe6d00f2b60 con 0x7fe6d4101090 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/922706796 learned_addr learned my addr 172.21.15.87:0/922706796 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d966d640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc32428540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6da66f640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3242ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/922706796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 msgr2=0x55fc32428540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc32428540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/922706796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 msgr2=0x55fc3242ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3242ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/922706796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc321607d0 con 0x55fc3241f580 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6da66f640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3242ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6da66f640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc005cc0 tx=0x7fe6cc014a30 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c7390 tx=0x7fe6d00bea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f01777c3693ae22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6bffff640 1 -- 172.21.15.87:0/922706796 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0083bd0 con 0x55fc3241f580 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6bffff640 1 -- 172.21.15.87:0/922706796 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6d0005560 con 0x55fc3241f580 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.882+0000 7fe6bffff640 1 -- 172.21.15.87:0/922706796 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00884b0 con 0x55fc3241f580 2024-04-22T13:41:38.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- 172.21.15.87:0/922706796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 msgr2=0x55fc3241f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00c7390 tx=0x7fe6d00bea20 comp rx=0 tx=0).stop 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- 172.21.15.87:0/922706796 shutdown_connections 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3242ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc32428540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 --2- 172.21.15.87:0/922706796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3241f960 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- 172.21.15.87:0/922706796 >> 172.21.15.87:0/922706796 conn(0x55fc32419cb0 msgr2=0x55fc3241a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- 172.21.15.87:0/922706796 shutdown_connections 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- 172.21.15.87:0/922706796 wait complete. 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.883+0000 7fe6deda6740 1 -- start start 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 Processor -- start 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc324259d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc3241fea0 0x55fc32425f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fc3242b630 con 0x55fc32428a80 2024-04-22T13:41:38.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc3233d2f0 con 0x55fc3241fea0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fc3242b7b0 con 0x55fc3241f580 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6deda6740 10 client.0 _ll_register_callbacks cb 0x55fc3234a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57442/0 (socket says 172.21.15.87:57442) 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 -- 172.21.15.87:0/965832002 learned_addr learned my addr 172.21.15.87:0/965832002 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6d966d640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc3241fea0 0x55fc32425f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc324259d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 -- 172.21.15.87:0/965832002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 msgr2=0x55fc324259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc324259d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 -- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc3241fea0 msgr2=0x55fc32425f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc3241fea0 0x55fc32425f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.884+0000 7fe6da66f640 1 -- 172.21.15.87:0/965832002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc320963f0 con 0x55fc32428a80 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6da66f640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc011cf0 tx=0x7fe6cc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8b0c8e3da20eca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc024030 con 0x55fc32428a80 2024-04-22T13:41:38.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe6cc004b20 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fc32391c00 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe6cc02e050 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 10 client.18555 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc320560f0 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fc32391c00 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6cc01f480 con 0x55fc32428a80 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 10 client.18555 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 10 client.18555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 20 client.18555 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc32438740 0x55fc3243ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 4 client.18555 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.885+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc3238ba70 con 0x55fc32438740 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.886+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc32438740 0x55fc3243ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.886+0000 7fe6bd7fa640 10 client.18555 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.886+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc32438740 0x55fc3243ab60 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6219925171456b48 server_cookie=d4e5f767b81726ef in_seq=0 out_seq=0 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/965832002 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc3238ba70 con 0x55fc32438740 2024-04-22T13:41:38.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6bd7fa640 10 client.18555 handle_command_reply: tid=0 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 client.18555 shutdown 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 msgr2=0x55fc32426450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc011cf0 tx=0x7fe6cc004ab0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 shutdown_connections 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc32438740 0x55fc3243ab60 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc32428a80 0x55fc32426450 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc3241fea0 0x55fc32425f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 --2- 172.21.15.87:0/965832002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc324259d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6bd7fa640 0 client.18555 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 >> 172.21.15.87:0/965832002 conn(0x55fc32419cb0 msgr2=0x55fc3241ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 shutdown_connections 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 1 -- 172.21.15.87:0/965832002 wait complete. 2024-04-22T13:41:38.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.887+0000 7fe6deda6740 20 client.18555 trim_cache size 0 max 16384 2024-04-22T13:41:38.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 Processor -- start 2024-04-22T13:41:38.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 -- start start 2024-04-22T13:41:38.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc32427980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3241b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41b0150 con 0x55fc3241fea0 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ada20 con 0x55fc32428a80 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6deda6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ab890 con 0x55fc3241f580 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6d966d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.904+0000 7fe6d966d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57450/0 (socket says 172.21.15.87:57450) 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 -- 172.21.15.87:0/3749131573 learned_addr learned my addr 172.21.15.87:0/3749131573 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 -- 172.21.15.87:0/3749131573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 msgr2=0x55fc32427980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc32427980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 -- 172.21.15.87:0/3749131573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 msgr2=0x55fc3241b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3241b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 -- 172.21.15.87:0/3749131573 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc3241aa40 con 0x55fc3241fea0 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6d966d640 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4008dd0 tx=0x7fe6c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66547072c9f4c9ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3749131573 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4005020 con 0x55fc3241fea0 2024-04-22T13:41:38.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3749131573 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c40047a0 con 0x55fc3241fea0 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.905+0000 7fe6bf7fe640 1 -- 172.21.15.87:0/3749131573 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400f8b0 con 0x55fc3241fea0 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- 172.21.15.87:0/3749131573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 msgr2=0x55fc32425180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4008dd0 tx=0x7fe6c4011810 comp rx=0 tx=0).stop 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- 172.21.15.87:0/3749131573 shutdown_connections 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3241b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241fea0 0x55fc32425180 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- 172.21.15.87:0/3749131573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241f580 0x55fc32427980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- 172.21.15.87:0/3749131573 >> 172.21.15.87:0/3749131573 conn(0x55fc32419cb0 msgr2=0x55fc3241b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- 172.21.15.87:0/3749131573 shutdown_connections 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- 172.21.15.87:0/3749131573 wait complete. 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- start start 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 Processor -- start 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc3242cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3249dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41abee0 con 0x55fc3241f580 2024-04-22T13:41:38.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ad350 con 0x55fc32428a80 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d41ac5b0 con 0x55fc3241fea0 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6deda6740 10 client.0 _ll_register_callbacks cb 0x7fe6cc0116e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.906+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57460/0 (socket says 172.21.15.87:57460) 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/241763245 learned_addr learned my addr 172.21.15.87:0/241763245 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6da66f640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3249dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d966d640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc3242cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/241763245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 msgr2=0x55fc3242cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc3242cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 msgr2=0x55fc3249dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3249dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 -- 172.21.15.87:0/241763245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3249e1d0 con 0x55fc3241f580 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6da66f640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3249dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d966d640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc3242cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:38.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6d9e6e640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00bebe0 tx=0x7fe6d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fce561067b0069b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00887e0 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7fe6d00f0570 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c40047a0 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe6d00889c0 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6deda6740 10 client.18561 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fc3249f380 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.907+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c40047a0 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe6d00df0b0 con 0x55fc3241f580 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 10 client.18561 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 10 client.18561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 20 client.18561 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc3249f900 0x55fc324a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:38.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 4 client.18561 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc32307590 con 0x55fc3249f900 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6d966d640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc3249f900 0x55fc324a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6bd7fa640 10 client.18561 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.908+0000 7fe6d966d640 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc3249f900 0x55fc324a1d20 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a171a6abd3374e1 server_cookie=18f4f13d0e404de2 in_seq=0 out_seq=0 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6bd7fa640 1 -- 172.21.15.87:0/241763245 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fc32307590 con 0x55fc3249f900 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6bd7fa640 10 client.18561 handle_command_reply: tid=0 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 client.18561 shutdown 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 msgr2=0x55fc3242c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00bebe0 tx=0x7fe6d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 shutdown_connections 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55fc3249f900 0x55fc324a1d20 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc32428a80 0x55fc3249dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fc3241fea0 0x55fc3242cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6bd7fa640 0 client.18561 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 --2- 172.21.15.87:0/241763245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fc3241f580 0x55fc3242c760 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 >> 172.21.15.87:0/241763245 conn(0x55fc32419cb0 msgr2=0x55fc32426220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 shutdown_connections 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 1 -- 172.21.15.87:0/241763245 wait complete. 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6deda6740 20 client.18561 trim_cache size 0 max 16384 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "2d33967f-6c6a-408f-8327-1baf563495d2": { 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "2d33967f-6c6a-408f-8327-1baf563495d2", 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:38.912 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:38.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 msgr2=0x7fe6c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc005cc0 tx=0x7fe6cc014a30 comp rx=0 tx=0).stop 2024-04-22T13:41:38.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 msgr2=0x7fe6d41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:38.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0005940 tx=0x7fe6d002c860 comp rx=0 tx=0).stop 2024-04-22T13:41:38.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 shutdown_connections 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe6c00a3d20 0x7fe6c00a61e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6d419ce90 0x7fe6d419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6d4106160 0x7fe6d4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 --2- 172.21.15.87:0/3577647413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d4101090 0x7fe6d41985d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.909+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 >> 172.21.15.87:0/3577647413 conn(0x7fe6d40771d0 msgr2=0x7fe6d406a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.910+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 shutdown_connections 2024-04-22T13:41:38.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:38.910+0000 7fe6dc0d2640 1 -- 172.21.15.87:0/3577647413 wait complete. 2024-04-22T13:41:38.916 INFO:tasks.cephfs.filesystem:scrub status for tag:2d33967f-6c6a-408f-8327-1baf563495d2 - {'path': '/', 'tag': '2d33967f-6c6a-408f-8327-1baf563495d2', 'options': 'recursive,force'} 2024-04-22T13:41:39.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:39 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:41:39.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:39 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:39.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:39 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:41:39.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:39 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:39.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:39 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:41:39.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:39 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:39.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:39 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:41:39.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:39 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:39.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:39 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:41:39.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:39 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:39.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:39 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:41:39.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:39 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:41:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:40 smithi120 ceph-mon[31101]: pgmap v306: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:40 smithi132 ceph-mon[31531]: pgmap v306: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:40 smithi087 ceph-mon[27999]: pgmap v306: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:41.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:41 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:41:41.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:41 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:41:41.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:41 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:41:42.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:42 smithi120 ceph-mon[31101]: pgmap v307: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:42.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:42 smithi132 ceph-mon[31531]: pgmap v307: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:42.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:42 smithi087 ceph-mon[27999]: pgmap v307: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:41:44.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:44 smithi120 ceph-mon[31101]: pgmap v308: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:44.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:44 smithi132 ceph-mon[31531]: pgmap v308: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:44.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:44 smithi087 ceph-mon[27999]: pgmap v308: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:46.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:46 smithi120 ceph-mon[31101]: pgmap v309: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:41:46.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:46 smithi132 ceph-mon[31531]: pgmap v309: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:41:46.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:46 smithi087 ceph-mon[27999]: pgmap v309: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:41:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:48 smithi120 ceph-mon[31101]: pgmap v310: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:48.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:48 smithi132 ceph-mon[31531]: pgmap v310: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:48 smithi087 ceph-mon[27999]: pgmap v310: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:41:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:50 smithi120 ceph-mon[31101]: pgmap v311: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:41:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:50 smithi132 ceph-mon[31531]: pgmap v311: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:41:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:50 smithi087 ceph-mon[27999]: pgmap v311: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.882+0000 7f46f8f4a640 1 -- 172.21.15.87:0/2461682962 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f406aa30 msgr2=0x7f46f406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.882+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/2461682962 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f406aa30 0x7f46f406ae10 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ca0 tx=0x7f46e402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 -- 172.21.15.87:0/2461682962 shutdown_connections 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/2461682962 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f41072e0 0x7f46f410ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/2461682962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f406b3e0 0x7f46f4106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/2461682962 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f406aa30 0x7f46f406ae10 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 -- 172.21.15.87:0/2461682962 >> 172.21.15.87:0/2461682962 conn(0x7f46f4077ba0 msgr2=0x7f46f4077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 -- 172.21.15.87:0/2461682962 shutdown_connections 2024-04-22T13:41:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 -- 172.21.15.87:0/2461682962 wait complete. 2024-04-22T13:41:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 Processor -- start 2024-04-22T13:41:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.883+0000 7f46f8f4a640 1 -- start start 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 0x7f46f4101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f41072e0 0x7f46f4101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4112220 con 0x7f46f4101c80 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41120a0 con 0x7f46f41072e0 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f8f4a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41123a0 con 0x7f46f406aa30 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f1d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f41072e0 0x7f46f4101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33280/0 (socket says 172.21.15.87:33280) 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 -- 172.21.15.87:0/140787374 learned_addr learned my addr 172.21.15.87:0/140787374 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f259c640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 0x7f46f4101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.882 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 -- 172.21.15.87:0/140787374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 msgr2=0x7f46f4101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 0x7f46f4101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 -- 172.21.15.87:0/140787374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f41072e0 msgr2=0x7f46f4101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f41072e0 0x7f46f4101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 -- 172.21.15.87:0/140787374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f41a6f50 con 0x7f46f4101c80 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f259c640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 0x7f46f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d750 tx=0x7f46ec00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46e37fe640 1 -- 172.21.15.87:0/140787374 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec017070 con 0x7f46f4101c80 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.884+0000 7f46e37fe640 1 -- 172.21.15.87:0/140787374 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec0097c0 con 0x7f46f4101c80 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.885+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a7240 con 0x7f46f4101c80 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.885+0000 7f46e37fe640 1 -- 172.21.15.87:0/140787374 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec004ba0 con 0x7f46f4101c80 2024-04-22T13:41:50.883 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.885+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a7790 con 0x7f46f4101c80 2024-04-22T13:41:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.887+0000 7f46e37fe640 1 -- 172.21.15.87:0/140787374 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f46ec009c90 con 0x7f46f4101c80 2024-04-22T13:41:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.887+0000 7f46fbc0e740 1 Processor -- start 2024-04-22T13:41:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.887+0000 7f46fbc0e740 1 -- start start 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46e37fe640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 0x7f46dc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c2cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558d04b918d0 con 0x558d04c2d8d0 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558d04b2ee80 con 0x558d04c360e0 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558d04b8d890 con 0x558d04c2cb60 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 0x7f46dc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46e37fe640 1 -- 172.21.15.87:0/140787374 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f46ec0d00a0 con 0x7f46f4101c80 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33284/0 (socket says 172.21.15.87:33284) 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 -- 172.21.15.87:0/4268391263 learned_addr learned my addr 172.21.15.87:0/4268391263 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c2cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 -- 172.21.15.87:0/4268391263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 msgr2=0x558d04c2cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c2cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 -- 172.21.15.87:0/4268391263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 msgr2=0x558d04c384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 -- 172.21.15.87:0/4268391263 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d04b50320 con 0x558d04c2d8d0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 0x7f46dc0a6280 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x558d04c2e030 tx=0x7f46e8005d60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46f259c640 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ca0 tx=0x7f46e402cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef3a8e221db1a60 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.888+0000 7f46e0ff9640 1 -- 172.21.15.87:0/4268391263 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4043020 con 0x558d04c2d8d0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46e0ff9640 1 -- 172.21.15.87:0/4268391263 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46e4032970 con 0x558d04c2d8d0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46e0ff9640 1 -- 172.21.15.87:0/4268391263 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4039040 con 0x558d04c2d8d0 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 -- 172.21.15.87:0/4268391263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 msgr2=0x558d04c2c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ca0 tx=0x7f46e402cca0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 -- 172.21.15.87:0/4268391263 shutdown_connections 2024-04-22T13:41:50.892 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c2cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/4268391263 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c2c620 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 -- 172.21.15.87:0/4268391263 >> 172.21.15.87:0/4268391263 conn(0x558d04bf3c90 msgr2=0x558d04c27850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 -- 172.21.15.87:0/4268391263 shutdown_connections 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.889+0000 7f46fbc0e740 1 -- 172.21.15.87:0/4268391263 wait complete. 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 -- start start 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 Processor -- start 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558d04c38db0 con 0x558d04c360e0 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558d04b19eb0 con 0x558d04c2cb60 2024-04-22T13:41:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558d04c38f30 con 0x558d04c2d8d0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46fbc0e740 10 client.0 _ll_register_callbacks cb 0x558d04b46b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33288/0 (socket says 172.21.15.87:33288) 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:45590/0 (socket says 172.21.15.87:45590) 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 -- 172.21.15.87:0/3148889518 learned_addr learned my addr 172.21.15.87:0/3148889518 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c32dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 -- 172.21.15.87:0/3148889518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 msgr2=0x558d04c32dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c32dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 -- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 msgr2=0x558d04c35a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.890+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46f2d9d640 1 -- 172.21.15.87:0/3148889518 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d048a4210 con 0x558d04c360e0 2024-04-22T13:41:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46f259c640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f46ec09c0b0 tx=0x7f46ec09c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d810742ce0424c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec020050 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f46ec004d40 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46e4032970 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f46ec0a4d90 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 10 client.18573 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558d04970620 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46e4032970 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f46ec0a0120 con 0x558d04c360e0 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 10 client.18573 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 10 client.18573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 20 client.18573 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04c460a0 0x558d04c48490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 4 client.18573 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.891+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d04ba5880 con 0x558d04c460a0 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.892+0000 7f46f259c640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04c460a0 0x558d04c48490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.892+0000 7f46d9ffb640 10 client.18573 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.892+0000 7f46f259c640 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04c460a0 0x558d04c48490 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0c3a404357e37a9 server_cookie=f87d3f37eb93078 in_seq=0 out_seq=0 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46d9ffb640 1 -- 172.21.15.87:0/3148889518 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d04ba5880 con 0x558d04c460a0 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46d9ffb640 10 client.18573 handle_command_reply: tid=0 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 client.18573 shutdown 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 msgr2=0x558d04c33320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f46ec09c0b0 tx=0x7f46ec09c8e0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 shutdown_connections 2024-04-22T13:41:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04c460a0 0x558d04c48490 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c360e0 0x558d04c33320 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/3148889518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c2cb60 0x558d04c35a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46d9ffb640 0 client.18573 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 >> 172.21.15.87:0/3148889518 conn(0x558d04bf3c90 msgr2=0x558d04c295a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 shutdown_connections 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 1 -- 172.21.15.87:0/3148889518 wait complete. 2024-04-22T13:41:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.893+0000 7f46fbc0e740 20 client.18573 trim_cache size 0 max 16384 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 Processor -- start 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 -- start start 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c35a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41b23c0 con 0x558d04c2cb60 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41afc90 con 0x558d04c360e0 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41adb00 con 0x558d04c2d8d0 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46f259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46f259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33304/0 (socket says 172.21.15.87:33304) 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46f259c640 1 -- 172.21.15.87:0/534507948 learned_addr learned my addr 172.21.15.87:0/534507948 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.908+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c35a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 -- 172.21.15.87:0/534507948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 msgr2=0x558d04c354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c354f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 -- 172.21.15.87:0/534507948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 msgr2=0x558d04c35a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c35a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 -- 172.21.15.87:0/534507948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d04c48800 con 0x558d04c2cb60 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46f259c640 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f46e402c630 tx=0x7f46e4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116b396e676278f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46dbfff640 1 -- 172.21.15.87:0/534507948 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4043040 con 0x558d04c2cb60 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46dbfff640 1 -- 172.21.15.87:0/534507948 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46e4003970 con 0x558d04c2cb60 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46dbfff640 1 -- 172.21.15.87:0/534507948 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4039040 con 0x558d04c2cb60 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46fbc0e740 1 -- 172.21.15.87:0/534507948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 msgr2=0x558d04c41cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.909+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f46e402c630 tx=0x7f46e4003ce0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- 172.21.15.87:0/534507948 shutdown_connections 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c35a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2d8d0 0x558d04c354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/534507948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2cb60 0x558d04c41cb0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- 172.21.15.87:0/534507948 >> 172.21.15.87:0/534507948 conn(0x558d04bf3c90 msgr2=0x558d04c31690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- 172.21.15.87:0/534507948 shutdown_connections 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- 172.21.15.87:0/534507948 wait complete. 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- start start 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 Processor -- start 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c3e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c3a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ae150 con 0x558d04c2d8d0 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41af5c0 con 0x558d04c360e0 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ae820 con 0x558d04c2cb60 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46fbc0e740 10 client.0 _ll_register_callbacks cb 0x7f46ec0983c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46f1d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46f1d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33306/0 (socket says 172.21.15.87:33306) 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46f1d9b640 1 -- 172.21.15.87:0/1092512430 learned_addr learned my addr 172.21.15.87:0/1092512430 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46f2d9d640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c3a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.910+0000 7f46f259c640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c3e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 -- 172.21.15.87:0/1092512430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 msgr2=0x558d04c3e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c3e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 -- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 msgr2=0x558d04c3a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c3a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 -- 172.21.15.87:0/1092512430 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d04cab610 con 0x558d04c2d8d0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46f1d9b640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f46f4102c40 tx=0x7f46e800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169c198eea33e92e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e8012790 con 0x558d04c2d8d0 2024-04-22T13:41:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f46e8012930 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d04b313d0 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f46e8012b30 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 10 client.18579 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558d04cac870 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558d04b313d0 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f46e802a430 con 0x558d04c2d8d0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 10 client.18579 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 10 client.18579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 20 client.18579 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04cad020 0x558d04caf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 4 client.18579 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.911+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d04b9f2f0 con 0x558d04cad020 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.912+0000 7f46f259c640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04cad020 0x558d04caf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.912+0000 7f46f259c640 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04cad020 0x558d04caf440 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89fcacf7af8eff20 server_cookie=7bb2c1f13aec6b70 in_seq=0 out_seq=0 2024-04-22T13:41:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.912+0000 7f46d9ffb640 10 client.18579 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.917+0000 7f46d9ffb640 1 -- 172.21.15.87:0/1092512430 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d04b9f2f0 con 0x558d04cad020 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.917+0000 7f46d9ffb640 10 client.18579 handle_command_reply: tid=0 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.917+0000 7f46fbc0e740 1 client.18579 shutdown 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 msgr2=0x558d04c39ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f46f4102c40 tx=0x7f46e800a2d0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 shutdown_connections 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558d04cad020 0x558d04caf440 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d04c360e0 0x558d04c3a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46d9ffb640 0 client.18579 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:50.915 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558d04c2d8d0 0x558d04c39ab0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 --2- 172.21.15.87:0/1092512430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558d04c2cb60 0x558d04c3e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 >> 172.21.15.87:0/1092512430 conn(0x558d04bf3c90 msgr2=0x558d04c28c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 shutdown_connections 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 1 -- 172.21.15.87:0/1092512430 wait complete. 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46fbc0e740 20 client.18579 trim_cache size 0 max 16384 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:41:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:41:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 msgr2=0x7f46dc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 0x7f46dc0a6280 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x558d04c2e030 tx=0x7f46e8005d60 comp rx=0 tx=0).stop 2024-04-22T13:41:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 msgr2=0x7f46f41a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d750 tx=0x7f46ec00db50 comp rx=0 tx=0).stop 2024-04-22T13:41:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 shutdown_connections 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f46dc0a3dc0 0x7f46dc0a6280 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46f4101c80 0x7f46f41a68b0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46f41072e0 0x7f46f4101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 --2- 172.21.15.87:0/140787374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46f406aa30 0x7f46f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.918+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 >> 172.21.15.87:0/140787374 conn(0x7f46f4077ba0 msgr2=0x7f46f40ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.919+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 shutdown_connections 2024-04-22T13:41:50.918 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:41:50.919+0000 7f46f8f4a640 1 -- 172.21.15.87:0/140787374 wait complete. 2024-04-22T13:41:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:52 smithi120 ceph-mon[31101]: pgmap v312: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7 op/s 2024-04-22T13:41:52.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:52 smithi132 ceph-mon[31531]: pgmap v312: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7 op/s 2024-04-22T13:41:52.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:52 smithi087 ceph-mon[27999]: pgmap v312: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7 op/s 2024-04-22T13:41:54.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:54 smithi120 ceph-mon[31101]: pgmap v313: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:41:54.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:54 smithi132 ceph-mon[31531]: pgmap v313: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:41:54.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:54 smithi087 ceph-mon[27999]: pgmap v313: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:41:55.327 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.9 2024-04-22T13:41:55.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:41:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 -- 172.21.15.87:0/190919050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102b40 msgr2=0x7f4550106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 --2- 172.21.15.87:0/190919050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102b40 0x7f4550106b80 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4544009900 tx=0x7f454402bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 -- 172.21.15.87:0/190919050 shutdown_connections 2024-04-22T13:41:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 --2- 172.21.15.87:0/190919050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4550107260 0x7f455010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 --2- 172.21.15.87:0/190919050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102b40 0x7f4550106b80 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 --2- 172.21.15.87:0/190919050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550102190 0x7f4550102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.444+0000 7f4556b85640 1 -- 172.21.15.87:0/190919050 >> 172.21.15.87:0/190919050 conn(0x7f45500793e0 msgr2=0x7f45500797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- 172.21.15.87:0/190919050 shutdown_connections 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- 172.21.15.87:0/190919050 wait complete. 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 Processor -- start 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- start start 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 0x7f4550198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f455019cfc0 0x7f455019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4550112180 con 0x7f4550102190 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4550112000 con 0x7f4550107260 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4556b85640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4550112300 con 0x7f455019cfc0 2024-04-22T13:41:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4554921640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4554921640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33332/0 (socket says 172.21.15.87:33332) 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.445+0000 7f4554921640 1 -- 172.21.15.87:0/1658547444 learned_addr learned my addr 172.21.15.87:0/1658547444 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4555122640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f455019cfc0 0x7f455019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f454ffff640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 0x7f4550198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 -- 172.21.15.87:0/1658547444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f455019cfc0 msgr2=0x7f455019d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f455019cfc0 0x7f455019d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 -- 172.21.15.87:0/1658547444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 msgr2=0x7f4550198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 0x7f4550198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 -- 172.21.15.87:0/1658547444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455019dab0 con 0x7f4550102190 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f454ffff640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 0x7f4550198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4554921640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f454000d950 tx=0x7f454000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540017070 con 0x7f4550102190 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4540009c30 con 0x7f4550102190 2024-04-22T13:41:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455019dda0 con 0x7f4550102190 2024-04-22T13:41:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454000f620 con 0x7f4550102190 2024-04-22T13:41:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.446+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45501a5590 con 0x7f4550102190 2024-04-22T13:41:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.447+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4550069230 con 0x7f4550102190 2024-04-22T13:41:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.448+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4540010460 con 0x7f4550102190 2024-04-22T13:41:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.449+0000 7f454dffb640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 0x7f45380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.449+0000 7f454ffff640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 0x7f45380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.449+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f45400d04b0 con 0x7f4550102190 2024-04-22T13:41:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.449+0000 7f454ffff640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 0x7f45380a61e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4544036040 tx=0x7f4544000bc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.454+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f454009cb90 con 0x7f4550102190 2024-04-22T13:41:55.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.670+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f455010a4f0 con 0x7f4550102190 2024-04-22T13:41:55.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.670+0000 7f454dffb640 1 -- 172.21.15.87:0/1658547444 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f4540009de0 con 0x7f4550102190 2024-04-22T13:41:55.667 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:41:55.667 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:41:55.667 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.672+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 msgr2=0x7f45380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.672+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 0x7f45380a61e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4544036040 tx=0x7f4544000bc0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.672+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 msgr2=0x7f4550198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.672+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f454000d950 tx=0x7f454000de10 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 shutdown_connections 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45380a3d20 0x7f45380a61e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f455019cfc0 0x7f455019d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4550107260 0x7f4550198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 --2- 172.21.15.87:0/1658547444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102190 0x7f4550198700 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 >> 172.21.15.87:0/1658547444 conn(0x7f45500793e0 msgr2=0x7f4550100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 shutdown_connections 2024-04-22T13:41:55.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.673+0000 7f4556b85640 1 -- 172.21.15.87:0/1658547444 wait complete. 2024-04-22T13:41:55.680 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 82.9 --await' 2024-04-22T13:41:55.680 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 82.9 --await 2024-04-22T13:41:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 -- 172.21.15.87:0/2023416212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158101ac0 msgr2=0x7f3158105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 --2- 172.21.15.87:0/2023416212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158101ac0 0x7f3158105ab0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f3154009900 tx=0x7f315402bc30 comp rx=0 tx=0).stop 2024-04-22T13:41:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 -- 172.21.15.87:0/2023416212 shutdown_connections 2024-04-22T13:41:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 --2- 172.21.15.87:0/2023416212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158106190 0x7f315810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 --2- 172.21.15.87:0/2023416212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158101ac0 0x7f3158105ab0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 --2- 172.21.15.87:0/2023416212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3158101110 0x7f31581014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 -- 172.21.15.87:0/2023416212 >> 172.21.15.87:0/2023416212 conn(0x7f3158077fb0 msgr2=0x7f31580783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 -- 172.21.15.87:0/2023416212 shutdown_connections 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.791+0000 7f315f9f4640 1 -- 172.21.15.87:0/2023416212 wait complete. 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 Processor -- start 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 -- start start 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 0x7f3158199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 0x7f31581a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f31581110d0 con 0x7f3158106190 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3158110f50 con 0x7f3158101110 2024-04-22T13:41:55.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315f9f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3158111250 con 0x7f315819a020 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315cf8f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315cf8f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33352/0 (socket says 172.21.15.87:33352) 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315cf8f640 1 -- 172.21.15.87:0/626427462 learned_addr learned my addr 172.21.15.87:0/626427462 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315d790640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 0x7f3158199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.792+0000 7f315df91640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 0x7f31581a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 -- 172.21.15.87:0/626427462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 msgr2=0x7f31581a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 0x7f31581a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 -- 172.21.15.87:0/626427462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 msgr2=0x7f3158199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 0x7f3158199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 -- 172.21.15.87:0/626427462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31581a4dd0 con 0x7f3158106190 2024-04-22T13:41:55.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315d790640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 0x7f3158199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315df91640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 0x7f31581a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315cf8f640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f3154036040 tx=0x7f315402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f314e7fc640 1 -- 172.21.15.87:0/626427462 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154043020 con 0x7f3158106190 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f314e7fc640 1 -- 172.21.15.87:0/626427462 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f315402cbe0 con 0x7f3158106190 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31581a5060 con 0x7f3158106190 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f314e7fc640 1 -- 172.21.15.87:0/626427462 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154003bf0 con 0x7f3158106190 2024-04-22T13:41:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.793+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31581a55b0 con 0x7f3158106190 2024-04-22T13:41:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.795+0000 7f314e7fc640 1 -- 172.21.15.87:0/626427462 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3154046020 con 0x7f3158106190 2024-04-22T13:41:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.795+0000 7f31625eb740 1 Processor -- start 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.795+0000 7f31625eb740 1 -- start start 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc69c0 0x56449fbc6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f314e7fc640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 0x7f31440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56449fb270b0 con 0x56449fbcfec0 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56449fad8680 con 0x56449fbc72e0 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f31625eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56449fae4c60 con 0x56449fbc69c0 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33368/0 (socket says 172.21.15.87:33368) 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 -- 172.21.15.87:0/3731854450 learned_addr learned my addr 172.21.15.87:0/3731854450 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f314e7fc640 1 -- 172.21.15.87:0/626427462 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f31540f29b0 con 0x7f3158106190 2024-04-22T13:41:55.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315cf8f640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 0x7f31440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315d790640 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc69c0 0x56449fbc6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 -- 172.21.15.87:0/3731854450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc69c0 msgr2=0x56449fbc6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc69c0 0x56449fbc6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 -- 172.21.15.87:0/3731854450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 msgr2=0x56449fbcf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 -- 172.21.15.87:0/3731854450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449f909c90 con 0x56449fbcfec0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315cf8f640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 0x7f31440a61e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f31540bb470 tx=0x7f31540a9040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.796+0000 7f315df91640 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f315000af50 tx=0x7f315000a830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca621a16b3f2210 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31437fe640 1 -- 172.21.15.87:0/3731854450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150004d20 con 0x56449fbcfec0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31437fe640 1 -- 172.21.15.87:0/3731854450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3150014e50 con 0x56449fbcfec0 2024-04-22T13:41:55.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31437fe640 1 -- 172.21.15.87:0/3731854450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500054c0 con 0x56449fbcfec0 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 -- 172.21.15.87:0/3731854450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 msgr2=0x56449fbd22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f315000af50 tx=0x7f315000a830 comp rx=0 tx=0).stop 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 -- 172.21.15.87:0/3731854450 shutdown_connections 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbd22b0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 --2- 172.21.15.87:0/3731854450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc69c0 0x56449fbc6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 -- 172.21.15.87:0/3731854450 >> 172.21.15.87:0/3731854450 conn(0x56449fbc10f0 msgr2=0x56449fbc14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 -- 172.21.15.87:0/3731854450 shutdown_connections 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.797+0000 7f31625eb740 1 -- 172.21.15.87:0/3731854450 wait complete. 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 -- start start 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 Processor -- start 2024-04-22T13:41:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbcd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56449fb2a400 con 0x56449fbc69c0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56449fae6840 con 0x56449fbc72e0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56449fb2f720 con 0x56449fbcfec0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f31625eb740 10 client.0 _ll_register_callbacks cb 0x56449fb37ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33382/0 (socket says 172.21.15.87:33382) 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 -- 172.21.15.87:0/2217010454 learned_addr learned my addr 172.21.15.87:0/2217010454 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315d790640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 -- 172.21.15.87:0/2217010454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 msgr2=0x56449fbcd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbcd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 -- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 msgr2=0x56449fbcd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.798+0000 7f315df91640 1 -- 172.21.15.87:0/2217010454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449f96a420 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f315df91640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f315000a6a0 tx=0x7f3150014460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90e54f953a40e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150004cd0 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f3150022070 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3150014e50 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3150004e70 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 10 client.18597 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56449f7fd580 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3150014e50 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3150012450 con 0x56449fbc69c0 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 10 client.18597 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 10 client.18597 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 20 client.18597 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fbe1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 4 client.18597 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:41:55.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.799+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56449fb38b10 con 0x56449fbdfac0 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.800+0000 7f315d790640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fbe1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.800+0000 7f3140ff9640 10 client.18597 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.800+0000 7f315d790640 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fbe1eb0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7e336c34cb75d5c server_cookie=b7b9e9e13e6ec955 in_seq=0 out_seq=0 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f3140ff9640 1 -- 172.21.15.87:0/2217010454 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56449fb38b10 con 0x56449fbdfac0 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f3140ff9640 10 client.18597 handle_command_reply: tid=0 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 client.18597 shutdown 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 msgr2=0x56449fbccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f315000a6a0 tx=0x7f3150014460 comp rx=0 tx=0).stop 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 shutdown_connections 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fbe1eb0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f3140ff9640 0 client.18597 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbcd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbcd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 --2- 172.21.15.87:0/2217010454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbccf40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 >> 172.21.15.87:0/2217010454 conn(0x56449fbc10f0 msgr2=0x56449fbc2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 shutdown_connections 2024-04-22T13:41:55.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.801+0000 7f31625eb740 1 -- 172.21.15.87:0/2217010454 wait complete. 2024-04-22T13:41:55.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.802+0000 7f31625eb740 20 client.18597 trim_cache size 0 max 16384 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 Processor -- start 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 -- start start 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc69c0 0x56449fbcf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc72e0 0x56449fbcbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f31581adbb0 con 0x56449fbcfec0 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31581aba20 con 0x56449fbc69c0 2024-04-22T13:41:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f31625eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31581ad4e0 con 0x56449fbc72e0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.818+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33392/0 (socket says 172.21.15.87:33392) 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 -- 172.21.15.87:0/2394851514 learned_addr learned my addr 172.21.15.87:0/2394851514 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc69c0 0x56449fbcf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315d790640 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc72e0 0x56449fbcbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 -- 172.21.15.87:0/2394851514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc72e0 msgr2=0x56449fbcbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc72e0 0x56449fbcbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 -- 172.21.15.87:0/2394851514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc69c0 msgr2=0x56449fbcf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc69c0 0x56449fbcf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 -- 172.21.15.87:0/2394851514 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449fbd22b0 con 0x56449fbcfec0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f315df91640 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f31500054c0 tx=0x7f3150000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f675350f380ea47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f3142ffd640 1 -- 172.21.15.87:0/2394851514 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150033040 con 0x56449fbcfec0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f3142ffd640 1 -- 172.21.15.87:0/2394851514 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3150014e50 con 0x56449fbcfec0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.819+0000 7f3142ffd640 1 -- 172.21.15.87:0/2394851514 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150005890 con 0x56449fbcfec0 2024-04-22T13:41:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- 172.21.15.87:0/2394851514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 msgr2=0x56449fbcd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f31500054c0 tx=0x7f3150000f40 comp rx=0 tx=0).stop 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- 172.21.15.87:0/2394851514 shutdown_connections 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbcfec0 0x56449fbcd7e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbc72e0 0x56449fbcbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- 172.21.15.87:0/2394851514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc69c0 0x56449fbcf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- 172.21.15.87:0/2394851514 >> 172.21.15.87:0/2394851514 conn(0x56449fbc10f0 msgr2=0x56449fbc01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- 172.21.15.87:0/2394851514 shutdown_connections 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- 172.21.15.87:0/2394851514 wait complete. 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- start start 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 Processor -- start 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbd3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbd4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f31581ac740 con 0x56449fbc69c0 2024-04-22T13:41:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31581ac070 con 0x56449fbc72e0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.820+0000 7f31625eb740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31581ace10 con 0x56449fbcfec0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f31625eb740 10 client.0 _ll_register_callbacks cb 0x7f3150014720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33404/0 (socket says 172.21.15.87:33404) 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 -- 172.21.15.87:0/852036133 learned_addr learned my addr 172.21.15.87:0/852036133 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315d790640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbd3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 -- 172.21.15.87:0/852036133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 msgr2=0x56449fbd4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbd4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 -- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 msgr2=0x56449fbd3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbd3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 -- 172.21.15.87:0/852036133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449fc49320 con 0x56449fbc69c0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f315df91640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3150014070 tx=0x7f3150014440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aba5a257621e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f3140ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:41:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315003e060 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7f3150012a90 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3150014e50 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3150033040 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f31625eb740 10 client.18603 fetch_fsmap learned FSMap version 23 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56449fc4a540 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.821+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3150014e50 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f315000fa80 con 0x56449fbc69c0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 10 client.18603 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 10 client.18603 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 20 client.18603 populate_metadata read hostname 'smithi087' 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fc4cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 4 client.18603 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}] 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}) -- 0x56449fb32980 con 0x56449fbdfac0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f315d790640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fc4cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f3140ff9640 10 client.18603 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.822+0000 7f315d790640 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fc4cbd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0e2fc21c8a50a76 server_cookie=339f08dacd81588e in_seq=0 out_seq=0 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f3140ff9640 1 -- 172.21.15.87:0/852036133 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56449fb32980 con 0x56449fbdfac0 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f3140ff9640 10 client.18603 handle_command_reply: tid=0 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 client.18603 shutdown 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 msgr2=0x56449fbd3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3150014070 tx=0x7f3150014440 comp rx=0 tx=0).stop 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 shutdown_connections 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x56449fbdfac0 0x56449fc4cbd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56449fbcfec0 0x56449fbd4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f3140ff9640 0 client.18603 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56449fbc72e0 0x56449fbd3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 --2- 172.21.15.87:0/852036133 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56449fbc69c0 0x56449fbd3710 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 >> 172.21.15.87:0/852036133 conn(0x56449fbc10f0 msgr2=0x56449fbc2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 shutdown_connections 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.833+0000 7f31625eb740 1 -- 172.21.15.87:0/852036133 wait complete. 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f31625eb740 20 client.18603 trim_cache size 0 max 16384 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 38, 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:41:55.831 INFO:teuthology.orchestra.run.smithi087.stdout: "e83f2f35": { 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 38, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 82.9, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:55.832 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:55.833 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:55.833 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:55.833 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:41:55.833 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:41:55.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 msgr2=0x7f31440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 0x7f31440a61e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f31540bb470 tx=0x7f31540a9040 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 msgr2=0x7f3158199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f3154036040 tx=0x7f315402c7e0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 shutdown_connections 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f31440a3d20 0x7f31440a61e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f315819a020 0x7f31581a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3158106190 0x7f3158199a90 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 --2- 172.21.15.87:0/626427462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3158101110 0x7f3158199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 >> 172.21.15.87:0/626427462 conn(0x7f3158077fb0 msgr2=0x7f31580ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:41:55.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 shutdown_connections 2024-04-22T13:41:55.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:41:55.834+0000 7f315f9f4640 1 -- 172.21.15.87:0/626427462 wait complete. 2024-04-22T13:41:55.841 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: e83f2f35, quiesce took 0.5 seconds. Will release after: 22.9 2024-04-22T13:41:55.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:55 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1658547444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:56.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:55 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1658547444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:56.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:55 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1658547444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:41:56.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:56 smithi087 ceph-mon[27999]: pgmap v314: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:57.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:56 smithi120 ceph-mon[31101]: pgmap v314: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:57.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:56 smithi132 ceph-mon[31531]: pgmap v314: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:41:58.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:41:58 smithi120 ceph-mon[31101]: pgmap v315: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:58.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:41:58 smithi132 ceph-mon[31531]: pgmap v315: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:41:58.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:41:58 smithi087 ceph-mon[27999]: pgmap v315: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:00.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:00 smithi120 ceph-mon[31101]: pgmap v316: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:00.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:00 smithi132 ceph-mon[31531]: pgmap v316: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:00.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:00 smithi087 ceph-mon[27999]: pgmap v316: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:02 smithi120 ceph-mon[31101]: pgmap v317: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:02 smithi132 ceph-mon[31531]: pgmap v317: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:02.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:02 smithi087 ceph-mon[27999]: pgmap v317: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3426241363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8106e90 msgr2=0x7ff0e806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3426241363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8106e90 0x7ff0e806a730 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009ca0 tx=0x7ff0e402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3426241363 shutdown_connections 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3426241363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e806b350 0x7ff0e81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3426241363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8107300 0x7ff0e806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3426241363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8106e90 0x7ff0e806a730 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3426241363 >> 172.21.15.87:0/3426241363 conn(0x7ff0e8077c30 msgr2=0x7ff0e8078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3426241363 shutdown_connections 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3426241363 wait complete. 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 Processor -- start 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.014+0000 7ff0efd0d640 1 -- start start 2024-04-22T13:42:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8106e90 0x7ff0e81a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e819eb50 0x7ff0e819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8118810 con 0x7ff0e8107300 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8118690 con 0x7ff0e819eb50 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8118990 con 0x7ff0e8106e90 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49348/0 (socket says 172.21.15.87:49348) 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 -- 172.21.15.87:0/3593781596 learned_addr learned my addr 172.21.15.87:0/3593781596 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8106e90 0x7ff0e81a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e819eb50 0x7ff0e819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 -- 172.21.15.87:0/3593781596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8106e90 msgr2=0x7ff0e81a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8106e90 0x7ff0e81a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 -- 172.21.15.87:0/3593781596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e819eb50 msgr2=0x7ff0e819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e819eb50 0x7ff0e819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 -- 172.21.15.87:0/3593781596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e819f670 con 0x7ff0e8107300 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0ed2a8640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c970 tx=0x7ff0d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0deffd640 1 -- 172.21.15.87:0/3593781596 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8007bb0 con 0x7ff0e8107300 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0deffd640 1 -- 172.21.15.87:0/3593781596 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0d8007d50 con 0x7ff0e8107300 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e819f960 con 0x7ff0e8107300 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0deffd640 1 -- 172.21.15.87:0/3593781596 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8018730 con 0x7ff0e8107300 2024-04-22T13:42:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.015+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e81abb10 con 0x7ff0e8107300 2024-04-22T13:42:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.018+0000 7ff0deffd640 1 -- 172.21.15.87:0/3593781596 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff0d800f990 con 0x7ff0e8107300 2024-04-22T13:42:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.018+0000 7ff0f29d0740 1 Processor -- start 2024-04-22T13:42:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 -- start start 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383f8d0 0x55c47383e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c47383efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0deffd640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 0x7ff0d40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c4737a38d0 con 0x55c4738480e0 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c473740e80 con 0x55c47383eb60 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c47379f890 con 0x55c47383f8d0 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ed2a8640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 0x7ff0d40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49352/0 (socket says 172.21.15.87:49352) 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/3862492199 learned_addr learned my addr 172.21.15.87:0/3862492199 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383f8d0 0x55c47383e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c47383efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0deffd640 1 -- 172.21.15.87:0/3593781596 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7ff0d80d0810 con 0x7ff0e8107300 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/3862492199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383f8d0 msgr2=0x55c47383e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383f8d0 0x55c47383e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/3862492199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 msgr2=0x55c47383efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c47383efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.019+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/3862492199 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c473762320 con 0x55c4738480e0 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.020+0000 7ff0ed2a8640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 0x7ff0d40a6280 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff0d809c490 tx=0x7ff0d8087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.020+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0014970 tx=0x7ff0e000c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5226035d32b6eabe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.020+0000 7ff0d3fff640 1 -- 172.21.15.87:0/3862492199 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00142d0 con 0x55c4738480e0 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.020+0000 7ff0d3fff640 1 -- 172.21.15.87:0/3862492199 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e00054d0 con 0x55c4738480e0 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.020+0000 7ff0d3fff640 1 -- 172.21.15.87:0/3862492199 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0004d80 con 0x55c4738480e0 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3862492199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 msgr2=0x55c47384a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0014970 tx=0x7ff0e000c4f0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3862492199 shutdown_connections 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c4738480e0 0x55c47384a4d0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c47383efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3862492199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383f8d0 0x55c47383e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3862492199 >> 172.21.15.87:0/3862492199 conn(0x55c473805c90 msgr2=0x55c473839850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3862492199 shutdown_connections 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3862492199 wait complete. 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 -- start start 2024-04-22T13:42:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.021+0000 7ff0f29d0740 1 Processor -- start 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 0x55c473847ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4738480e0 0x55c473845460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c47384ae10 con 0x55c47383f8d0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c47372beb0 con 0x55c4738480e0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c47384af90 con 0x55c47383eb60 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 10 client.0 _ll_register_callbacks cb 0x55c473758b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49362/0 (socket says 172.21.15.87:49362) 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 -- 172.21.15.87:0/3127498166 learned_addr learned my addr 172.21.15.87:0/3127498166 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 0x55c473847ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4738480e0 0x55c473845460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 -- 172.21.15.87:0/3127498166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 msgr2=0x55c473847ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 0x55c473847ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 -- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4738480e0 msgr2=0x55c473845460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4738480e0 0x55c473845460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 -- 172.21.15.87:0/3127498166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4734b6210 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 0x55c473847ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4037040 tx=0x7ff0e402cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d66f2f28e3366d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4044070 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7ff0e40324c0 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.022+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e00054d0 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff0e4050050 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 10 client.18615 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c473582620 con 0x55c47383f8d0 2024-04-22T13:42:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e00054d0 con 0x55c47383f8d0 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff0e4004140 con 0x55c47383f8d0 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 10 client.18615 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 10 client.18615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 20 client.18615 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c473858130 0x55c47385a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 4 client.18615 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4737b7880 con 0x55c473858130 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.023+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c473858130 0x55c47385a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.024+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c473858130 0x55c47385a540 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70a9b9586ae998e4 server_cookie=5807ac325f56fefe in_seq=0 out_seq=0 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.024+0000 7ff0d17fa640 10 client.18615 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.024+0000 7ff0d17fa640 1 -- 172.21.15.87:0/3127498166 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4737b7880 con 0x55c473858130 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.024+0000 7ff0d17fa640 10 client.18615 handle_command_reply: tid=0 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.024+0000 7ff0f29d0740 1 client.18615 shutdown 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 msgr2=0x55c473844e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4037040 tx=0x7ff0e402cca0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 shutdown_connections 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c473858130 0x55c47385a540 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4738480e0 0x55c473845460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473844e50 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0d17fa640 0 client.18615 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/3127498166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c47383eb60 0x55c473847ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 >> 172.21.15.87:0/3127498166 conn(0x55c473805c90 msgr2=0x55c47383b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 shutdown_connections 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 1 -- 172.21.15.87:0/3127498166 wait complete. 2024-04-22T13:42:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.025+0000 7ff0f29d0740 20 client.18615 trim_cache size 0 max 16384 2024-04-22T13:42:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 Processor -- start 2024-04-22T13:42:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 -- start start 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383f8d0 0x55c473844df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473847580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b68d0 con 0x55c47383eb60 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b41a0 con 0x55c47383f8d0 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b2010 con 0x55c4738480e0 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49366/0 (socket says 172.21.15.87:49366) 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/2877548505 learned_addr learned my addr 172.21.15.87:0/2877548505 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383f8d0 0x55c473844df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/2877548505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 msgr2=0x55c473847580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473847580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/2877548505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383f8d0 msgr2=0x55c473844df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383f8d0 0x55c473844df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 -- 172.21.15.87:0/2877548505 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c47383de30 con 0x55c47383eb60 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0014910 tx=0x7ff0e000c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e864bc49397b889d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0d37fe640 1 -- 172.21.15.87:0/2877548505 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00142b0 con 0x55c47383eb60 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.040+0000 7ff0d37fe640 1 -- 172.21.15.87:0/2877548505 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e0027030 con 0x55c47383eb60 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0d37fe640 1 -- 172.21.15.87:0/2877548505 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e002b820 con 0x55c47383eb60 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- 172.21.15.87:0/2877548505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 msgr2=0x55c473853d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0014910 tx=0x7ff0e000c4f0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- 172.21.15.87:0/2877548505 shutdown_connections 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473847580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383f8d0 0x55c473844df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/2877548505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383eb60 0x55c473853d10 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- 172.21.15.87:0/2877548505 >> 172.21.15.87:0/2877548505 conn(0x55c473805c90 msgr2=0x55c4738446e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- 172.21.15.87:0/2877548505 shutdown_connections 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- 172.21.15.87:0/2877548505 wait complete. 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 -- start start 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.041+0000 7ff0f29d0740 1 Processor -- start 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c4738484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473848f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b2660 con 0x55c47383f8d0 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b3ad0 con 0x55c47383eb60 2024-04-22T13:42:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b2d30 con 0x55c4738480e0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0f29d0740 10 client.0 _ll_register_callbacks cb 0x7ff0e4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49380/0 (socket says 172.21.15.87:49380) 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 -- 172.21.15.87:0/1971994185 learned_addr learned my addr 172.21.15.87:0/1971994185 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c4738484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473848f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 -- 172.21.15.87:0/1971994185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 msgr2=0x55c473848f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473848f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 -- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 msgr2=0x55c4738484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c4738484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 -- 172.21.15.87:0/1971994185 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c47384f420 con 0x55c47383f8d0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c4738484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0edaa9640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009bf0 tx=0x7ff0e4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d6163c0745774 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.042+0000 7ff0d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4044070 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8455+0+0 (secure 0 0 0) 0x7ff0e40326c0 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e0027030 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff0e4039040 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 10 client.18621 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c473850710 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e0027030 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff0e40039e0 con 0x55c47383f8d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 10 client.18621 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 10 client.18621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 20 client.18621 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c4738be7d0 0x55c4738c0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 4 client.18621 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c4737433d0 con 0x55c4738be7d0 2024-04-22T13:42:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.043+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c4738be7d0 0x55c4738c0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.044+0000 7ff0d17fa640 10 client.18621 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.044+0000 7ff0ee2aa640 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c4738be7d0 0x55c4738c0bf0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=279ee39c42674b08 server_cookie=923fd17b6ea721c2 in_seq=0 out_seq=0 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.051+0000 7ff0d17fa640 1 -- 172.21.15.87:0/1971994185 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c4737433d0 con 0x55c4738be7d0 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.051+0000 7ff0d17fa640 10 client.18621 handle_command_reply: tid=0 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 client.18621 shutdown 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 msgr2=0x55c473848a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009bf0 tx=0x7ff0e4005c50 comp rx=0 tx=0).stop 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 shutdown_connections 2024-04-22T13:42:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c4738be7d0 0x55c4738c0bf0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4738480e0 0x55c473848f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c47383f8d0 0x55c473848a00 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0d17fa640 0 client.18621 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 --2- 172.21.15.87:0/1971994185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c47383eb60 0x55c4738484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 >> 172.21.15.87:0/1971994185 conn(0x55c473805c90 msgr2=0x55c473843720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 shutdown_connections 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 1 -- 172.21.15.87:0/1971994185 wait complete. 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0f29d0740 20 client.18621 trim_cache size 0 max 16384 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:42:04.049 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:42:04.050 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 msgr2=0x7ff0d40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.050 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 0x7ff0d40a6280 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff0d809c490 tx=0x7ff0d8087040 comp rx=0 tx=0).stop 2024-04-22T13:42:04.050 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 msgr2=0x7ff0e81a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.052+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c970 tx=0x7ff0d800ce30 comp rx=0 tx=0).stop 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 shutdown_connections 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0d40a3dc0 0x7ff0d40a6280 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e819eb50 0x7ff0e819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0e8107300 0x7ff0e81a4e90 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 --2- 172.21.15.87:0/3593781596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0e8106e90 0x7ff0e81a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 >> 172.21.15.87:0/3593781596 conn(0x7ff0e8077c30 msgr2=0x7ff0e80fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 shutdown_connections 2024-04-22T13:42:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:04.053+0000 7ff0efd0d640 1 -- 172.21.15.87:0/3593781596 wait complete. 2024-04-22T13:42:04.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:04 smithi120 ceph-mon[31101]: pgmap v318: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:42:04.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:04 smithi132 ceph-mon[31531]: pgmap v318: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:42:04.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:04 smithi087 ceph-mon[27999]: pgmap v318: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:42:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:06 smithi087 ceph-mon[27999]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T13:42:07.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:06 smithi087 ceph-mon[27999]: pgmap v320: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:42:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:06 smithi120 ceph-mon[31101]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T13:42:07.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:06 smithi120 ceph-mon[31101]: pgmap v320: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:42:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:06 smithi132 ceph-mon[31531]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T13:42:07.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:06 smithi132 ceph-mon[31531]: pgmap v320: 97 pgs: 97 active+clean; 27 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:42:08.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:08 smithi120 ceph-mon[31101]: pgmap v321: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:08.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:08 smithi132 ceph-mon[31531]: pgmap v321: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:08.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:08 smithi087 ceph-mon[27999]: pgmap v321: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:08.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:42:08.918 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:42:09.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 -- 172.21.15.87:0/1585995016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b410c500 msgr2=0x7fd9b410e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 --2- 172.21.15.87:0/1585995016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b410c500 0x7fd9b410e9a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac009ca0 tx=0x7fd9ac02c230 comp rx=0 tx=0).stop 2024-04-22T13:42:09.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 -- 172.21.15.87:0/1585995016 shutdown_connections 2024-04-22T13:42:09.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 --2- 172.21.15.87:0/1585995016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b410c500 0x7fd9b410e9a0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 --2- 172.21.15.87:0/1585995016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b4103ae0 0x7fd9b410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 --2- 172.21.15.87:0/1585995016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 -- 172.21.15.87:0/1585995016 >> 172.21.15.87:0/1585995016 conn(0x7fd9b4078000 msgr2=0x7fd9b4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 -- 172.21.15.87:0/1585995016 shutdown_connections 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.034+0000 7fd9bae48640 1 -- 172.21.15.87:0/1585995016 wait complete. 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 Processor -- start 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 -- start start 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b410c500 0x7fd9b419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b4114390 con 0x7fd9b4103ae0 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b4114210 con 0x7fd9b410c500 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9bae48640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b4114510 con 0x7fd9b41031c0 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49398/0 (socket says 172.21.15.87:49398) 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 -- 172.21.15.87:0/3190993999 learned_addr learned my addr 172.21.15.87:0/3190993999 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b410c500 0x7fd9b419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 -- 172.21.15.87:0/3190993999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 msgr2=0x7fd9b41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 -- 172.21.15.87:0/3190993999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b410c500 msgr2=0x7fd9b419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b410c500 0x7fd9b419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 -- 172.21.15.87:0/3190993999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b419af70 con 0x7fd9b4103ae0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00d950 tx=0x7fd99c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.035+0000 7fd9b1ffb640 1 -- 172.21.15.87:0/3190993999 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c017070 con 0x7fd9b4103ae0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.036+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b419b260 con 0x7fd9b4103ae0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.036+0000 7fd9b1ffb640 1 -- 172.21.15.87:0/3190993999 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99c009c30 con 0x7fd9b4103ae0 2024-04-22T13:42:09.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.036+0000 7fd9b1ffb640 1 -- 172.21.15.87:0/3190993999 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00f620 con 0x7fd9b4103ae0 2024-04-22T13:42:09.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.036+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b41a7570 con 0x7fd9b4103ae0 2024-04-22T13:42:09.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.038+0000 7fd9b1ffb640 1 -- 172.21.15.87:0/3190993999 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd99c010460 con 0x7fd9b4103ae0 2024-04-22T13:42:09.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.038+0000 7fd9bda30740 1 Processor -- start 2024-04-22T13:42:09.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.038+0000 7fd9bda30740 1 -- start start 2024-04-22T13:42:09.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.038+0000 7fd9b1ffb640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 0x7fd9a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827580 0x55b55e827960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e830540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 0x7fd9a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b55e78c1e0 con 0x55b55e830a80 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b55e729390 con 0x55b55e827ea0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9bda30740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b55e7881a0 con 0x55b55e827580 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b1ffb640 1 -- 172.21.15.87:0/3190993999 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fd99c0d0450 con 0x7fd9b4103ae0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49410/0 (socket says 172.21.15.87:49410) 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 -- 172.21.15.87:0/3717966636 learned_addr learned my addr 172.21.15.87:0/3717966636 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827580 0x55b55e827960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e830540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 -- 172.21.15.87:0/3717966636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827580 msgr2=0x55b55e827960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827580 0x55b55e827960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 -- 172.21.15.87:0/3717966636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 msgr2=0x55b55e830540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e830540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 -- 172.21.15.87:0/3717966636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55e5687d0 con 0x55b55e830a80 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e830540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac003bc0 tx=0x7fd9ac003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653f33ca9bc74822 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 0x7fd9a00a61e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac02c770 tx=0x7fd9ac03b000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9aaffd640 1 -- 172.21.15.87:0/3717966636 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac039070 con 0x55b55e830a80 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9aaffd640 1 -- 172.21.15.87:0/3717966636 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ac050070 con 0x55b55e830a80 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.039+0000 7fd9aaffd640 1 -- 172.21.15.87:0/3717966636 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac034440 con 0x55b55e830a80 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- 172.21.15.87:0/3717966636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 msgr2=0x55b55e832e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac003bc0 tx=0x7fd9ac003bf0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- 172.21.15.87:0/3717966636 shutdown_connections 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e832e70 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e830540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3717966636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827580 0x55b55e827960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- 172.21.15.87:0/3717966636 >> 172.21.15.87:0/3717966636 conn(0x55b55e821cb0 msgr2=0x55b55e822090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- 172.21.15.87:0/3717966636 shutdown_connections 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- 172.21.15.87:0/3717966636 wait complete. 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.040+0000 7fd9bda30740 1 -- start start 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 Processor -- start 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e82e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 0x55b55e82e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b55e833810 con 0x55b55e827580 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b55e7452f0 con 0x55b55e830a80 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b55e833990 con 0x55b55e827ea0 2024-04-22T13:42:09.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9bda30740 10 client.0 _ll_register_callbacks cb 0x55b55e752580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49418/0 (socket says 172.21.15.87:49418) 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 -- 172.21.15.87:0/1799376458 learned_addr learned my addr 172.21.15.87:0/1799376458 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e82e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 0x55b55e82e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 -- 172.21.15.87:0/1799376458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 msgr2=0x55b55e82e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e82e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 -- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 msgr2=0x55b55e82e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 0x55b55e82e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b8be4640 1 -- 172.21.15.87:0/1799376458 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b55e49e3f0 con 0x55b55e827580 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.041+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 0x55b55e82e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4011a60 tx=0x7fd9a4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfbc79a01edee1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400fdb0 con 0x55b55e827580 2024-04-22T13:42:09.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fd9a4005020 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9ac050070 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd9a4005220 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 10 client.18633 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b55e45e0f0 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ac050070 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd9a401b400 con 0x55b55e827580 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 10 client.18633 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 10 client.18633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 20 client.18633 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e840b10 0x55b55e842fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 4 client.18633 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.042+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b55e799c00 con 0x55b55e840b10 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.043+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e840b10 0x55b55e842fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.043+0000 7fd993fff640 10 client.18633 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.043+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e840b10 0x55b55e842fb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a628b2d05b46d7c0 server_cookie=2924a28123bfa9ca in_seq=0 out_seq=0 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd993fff640 1 -- 172.21.15.87:0/1799376458 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b55e799c00 con 0x55b55e840b10 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd993fff640 10 client.18633 handle_command_reply: tid=0 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 client.18633 shutdown 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 msgr2=0x55b55e82dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4011a60 tx=0x7fd9a4011e60 comp rx=0 tx=0).stop 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 shutdown_connections 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e840b10 0x55b55e842fb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e830a80 0x55b55e82e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd993fff640 0 client.18633 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e82e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1799376458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e82dae0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 >> 172.21.15.87:0/1799376458 conn(0x55b55e821cb0 msgr2=0x55b55e823a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 shutdown_connections 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 1 -- 172.21.15.87:0/1799376458 wait complete. 2024-04-22T13:42:09.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.044+0000 7fd9bda30740 20 client.18633 trim_cache size 0 max 16384 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.061+0000 7fd9bda30740 1 Processor -- start 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.061+0000 7fd9bda30740 1 -- start start 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827580 0x55b55e830210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e83c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41b24e0 con 0x55b55e830a80 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41afdb0 con 0x55b55e827580 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9bda30740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41adc20 con 0x55b55e827ea0 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49424/0 (socket says 172.21.15.87:49424) 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 -- 172.21.15.87:0/1688001680 learned_addr learned my addr 172.21.15.87:0/1688001680 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827580 0x55b55e830210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 -- 172.21.15.87:0/1688001680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 msgr2=0x55b55e83c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e83c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 -- 172.21.15.87:0/1688001680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827580 msgr2=0x55b55e830210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827580 0x55b55e830210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 -- 172.21.15.87:0/1688001680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55e83f660 con 0x55b55e830a80 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac02efd0 tx=0x7fd9ac005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e476497a11ce67bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9aa7fc640 1 -- 172.21.15.87:0/1688001680 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac039040 con 0x55b55e830a80 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9aa7fc640 1 -- 172.21.15.87:0/1688001680 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ac059070 con 0x55b55e830a80 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.062+0000 7fd9aa7fc640 1 -- 172.21.15.87:0/1688001680 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac054450 con 0x55b55e830a80 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- 172.21.15.87:0/1688001680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 msgr2=0x55b55e82cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac02efd0 tx=0x7fd9ac005d80 comp rx=0 tx=0).stop 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- 172.21.15.87:0/1688001680 shutdown_connections 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e830a80 0x55b55e82cc40 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e827ea0 0x55b55e83c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 --2- 172.21.15.87:0/1688001680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827580 0x55b55e830210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- 172.21.15.87:0/1688001680 >> 172.21.15.87:0/1688001680 conn(0x55b55e821cb0 msgr2=0x55b55e8230e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- 172.21.15.87:0/1688001680 shutdown_connections 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- 172.21.15.87:0/1688001680 wait complete. 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 -- start start 2024-04-22T13:42:09.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.063+0000 7fd9bda30740 1 Processor -- start 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e834ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 0x55b55e8a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41ae270 con 0x55b55e827580 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41af6e0 con 0x55b55e827ea0 2024-04-22T13:42:09.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41ae940 con 0x55b55e830a80 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 10 client.0 _ll_register_callbacks cb 0x7fd9a400c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49436/0 (socket says 172.21.15.87:49436) 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 -- 172.21.15.87:0/3658631107 learned_addr learned my addr 172.21.15.87:0/3658631107 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e834ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 0x55b55e8a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 -- 172.21.15.87:0/3658631107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 msgr2=0x55b55e8a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 0x55b55e8a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 -- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 msgr2=0x55b55e834ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e834ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 -- 172.21.15.87:0/3658631107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b55e835730 con 0x55b55e827580 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b93e5640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 0x55b55e8a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9b8be4640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a401b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef89b88ab75789e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd993fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4030060 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fd9a400fbb0 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.064+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9ac059070 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd9a4005ea0 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 10 client.18639 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b55e8a7770 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ac059070 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd9a4004660 con 0x55b55e827580 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 10 client.18639 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 10 client.18639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 20 client.18639 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e8a7ef0 0x55b55e8aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 4 client.18639 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b55e793a70 con 0x55b55e8a7ef0 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.065+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e8a7ef0 0x55b55e8aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd993fff640 10 client.18639 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9b3fff640 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e8a7ef0 0x55b55e8aa310 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9a846e0c72038b server_cookie=7419c02abcca951 in_seq=0 out_seq=0 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd993fff640 1 -- 172.21.15.87:0/3658631107 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b55e793a70 con 0x55b55e8a7ef0 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd993fff640 10 client.18639 handle_command_reply: tid=0 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 client.18639 shutdown 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 msgr2=0x55b55e834960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a401b6b0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 shutdown_connections 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55b55e8a7ef0 0x55b55e8aa310 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b55e830a80 0x55b55e8a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b55e827ea0 0x55b55e834ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd993fff640 0 client.18639 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 --2- 172.21.15.87:0/3658631107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b55e827580 0x55b55e834960 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 >> 172.21.15.87:0/3658631107 conn(0x55b55e821cb0 msgr2=0x55b55e8230e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 shutdown_connections 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 1 -- 172.21.15.87:0/3658631107 wait complete. 2024-04-22T13:42:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.066+0000 7fd9bda30740 20 client.18639 trim_cache size 0 max 16384 2024-04-22T13:42:09.070 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:09.070 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:42:09.070 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:42:09.070 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 msgr2=0x7fd9a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 0x7fd9a00a61e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac02c770 tx=0x7fd9ac03b000 comp rx=0 tx=0).stop 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 msgr2=0x7fd9b41a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00d950 tx=0x7fd99c00de10 comp rx=0 tx=0).stop 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 shutdown_connections 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd9a00a3d20 0x7fd9a00a61e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9b410c500 0x7fd9b419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9b4103ae0 0x7fd9b41a0b30 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 --2- 172.21.15.87:0/3190993999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9b41031c0 0x7fd9b41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 >> 172.21.15.87:0/3190993999 conn(0x7fd9b4078000 msgr2=0x7fd9b40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 shutdown_connections 2024-04-22T13:42:09.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.067+0000 7fd9bae48640 1 -- 172.21.15.87:0/3190993999 wait complete. 2024-04-22T13:42:09.074 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:42:09.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:42:09.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- 172.21.15.87:0/2869029654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec101a50 msgr2=0x7f64ec105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- 172.21.15.87:0/2869029654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec101a50 0x7f64ec105a90 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009900 tx=0x7f64e002bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:09.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- 172.21.15.87:0/2869029654 shutdown_connections 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- 172.21.15.87:0/2869029654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106170 0x7f64ec10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- 172.21.15.87:0/2869029654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec101a50 0x7f64ec105a90 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- 172.21.15.87:0/2869029654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec1010a0 0x7f64ec101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- 172.21.15.87:0/2869029654 >> 172.21.15.87:0/2869029654 conn(0x7f64ec0771d0 msgr2=0x7f64ec0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- 172.21.15.87:0/2869029654 shutdown_connections 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- 172.21.15.87:0/2869029654 wait complete. 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 Processor -- start 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- start start 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec106170 0x7f64ec197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 0x7f64ec19c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111090 con 0x7f64ec1010a0 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec110f10 con 0x7f64ec19beb0 2024-04-22T13:42:09.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.175+0000 7f64f3350640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111210 con 0x7f64ec106170 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49446/0 (socket says 172.21.15.87:49446) 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 -- 172.21.15.87:0/1736320955 learned_addr learned my addr 172.21.15.87:0/1736320955 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f08eb640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec106170 0x7f64ec197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f18ed640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 0x7f64ec19c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 -- 172.21.15.87:0/1736320955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec106170 msgr2=0x7f64ec197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec106170 0x7f64ec197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 -- 172.21.15.87:0/1736320955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 msgr2=0x7f64ec19c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 0x7f64ec19c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 -- 172.21.15.87:0/1736320955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec19c9a0 con 0x7f64ec1010a0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f18ed640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 0x7f64ec19c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f10ec640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f64e400ebc0 tx=0x7f64e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4010630 con 0x7f64ec1010a0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64e4010c80 con 0x7f64ec1010a0 2024-04-22T13:42:09.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4018b60 con 0x7f64ec1010a0 2024-04-22T13:42:09.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec19cc90 con 0x7f64ec1010a0 2024-04-22T13:42:09.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.176+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a44a0 con 0x7f64ec1010a0 2024-04-22T13:42:09.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.177+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec069230 con 0x7f64ec1010a0 2024-04-22T13:42:09.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.178+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64e4018d00 con 0x7f64ec1010a0 2024-04-22T13:42:09.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.179+0000 7f64ea7fc640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 0x7f64d40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.179+0000 7f64f08eb640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 0x7f64d40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.179+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f64e40d1260 con 0x7f64ec1010a0 2024-04-22T13:42:09.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.179+0000 7f64f08eb640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 0x7f64d40a6260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009230 tx=0x7f64e0005fd0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.183+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64e409d910 con 0x7f64ec1010a0 2024-04-22T13:42:09.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.400+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64ec198940 con 0x7f64ec1010a0 2024-04-22T13:42:09.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.401+0000 7f64ea7fc640 1 -- 172.21.15.87:0/1736320955 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f64ec198940 con 0x7f64ec1010a0 2024-04-22T13:42:09.399 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:42:09.399 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:42:09.400 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:42:09.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 msgr2=0x7f64d40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 0x7f64d40a6260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009230 tx=0x7f64e0005fd0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 msgr2=0x7f64ec1975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f64e400ebc0 tx=0x7f64e400c6a0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 shutdown_connections 2024-04-22T13:42:09.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f64d40a3da0 0x7f64d40a6260 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec19beb0 0x7f64ec19c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec106170 0x7f64ec197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 --2- 172.21.15.87:0/1736320955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64ec1010a0 0x7f64ec1975f0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 >> 172.21.15.87:0/1736320955 conn(0x7f64ec0771d0 msgr2=0x7f64ec0feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 shutdown_connections 2024-04-22T13:42:09.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.403+0000 7f64f3350640 1 -- 172.21.15.87:0/1736320955 wait complete. 2024-04-22T13:42:09.411 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:42:09.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 -- 172.21.15.87:0/3625452158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 msgr2=0x7fa530101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 --2- 172.21.15.87:0/3625452158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa530101600 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fa524009930 tx=0x7fa52402bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 -- 172.21.15.87:0/3625452158 shutdown_connections 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 --2- 172.21.15.87:0/3625452158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa530101b40 0x7fa53010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 --2- 172.21.15.87:0/3625452158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa530101600 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 --2- 172.21.15.87:0/3625452158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530107030 0x7fa530107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 -- 172.21.15.87:0/3625452158 >> 172.21.15.87:0/3625452158 conn(0x7fa530076c50 msgr2=0x7fa530077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 -- 172.21.15.87:0/3625452158 shutdown_connections 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.514+0000 7fa536359640 1 -- 172.21.15.87:0/3625452158 wait complete. 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 Processor -- start 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 -- start start 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 0x7fa53019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa530107030 0x7fa5301985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa530112260 con 0x7fa5301011a0 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301120e0 con 0x7fa530101b40 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa536359640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301123e0 con 0x7fa530107030 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49478/0 (socket says 172.21.15.87:49478) 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.515+0000 7fa52ffff640 1 -- 172.21.15.87:0/3466443792 learned_addr learned my addr 172.21.15.87:0/3466443792 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 0x7fa53019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 -- 172.21.15.87:0/3466443792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa530107030 msgr2=0x7fa5301985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa530107030 0x7fa5301985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 -- 172.21.15.87:0/3466443792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 msgr2=0x7fa53019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 0x7fa53019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 -- 172.21.15.87:0/3466443792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa530198d20 con 0x7fa5301011a0 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 0x7fa53019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d950 tx=0x7fa52000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52d7fa640 1 -- 172.21.15.87:0/3466443792 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520017070 con 0x7fa5301011a0 2024-04-22T13:42:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52d7fa640 1 -- 172.21.15.87:0/3466443792 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa520009c30 con 0x7fa5301011a0 2024-04-22T13:42:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa52d7fa640 1 -- 172.21.15.87:0/3466443792 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52000f620 con 0x7fa5301011a0 2024-04-22T13:42:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa530199010 con 0x7fa5301011a0 2024-04-22T13:42:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.516+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5301a52b0 con 0x7fa5301011a0 2024-04-22T13:42:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.518+0000 7fa52d7fa640 1 -- 172.21.15.87:0/3466443792 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa520010460 con 0x7fa5301011a0 2024-04-22T13:42:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 Processor -- start 2024-04-22T13:42:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 -- start start 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1d28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56138b1371f0 con 0x56138b1d2e80 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56138b0d44b0 con 0x56138b1dba00 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa53902d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56138b1331b0 con 0x56138b1d24d0 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49484/0 (socket says 172.21.15.87:49484) 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 -- 172.21.15.87:0/2662966709 learned_addr learned my addr 172.21.15.87:0/2662966709 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa5348f6640 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1d28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52d7fa640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 0x7fa5180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 -- 172.21.15.87:0/2662966709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 msgr2=0x56138b1d28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1d28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 -- 172.21.15.87:0/2662966709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 msgr2=0x56138b1dde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa52ffff640 1 -- 172.21.15.87:0/2662966709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56138b0c61b0 con 0x56138b1d2e80 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.519+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 0x7fa5180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa52d7fa640 1 -- 172.21.15.87:0/3466443792 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fa5200d04e0 con 0x7fa5301011a0 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa52ffff640 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cdbb0 tx=0x7fa5200c75a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2da5444cc5eaef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa5167fc640 1 -- 172.21.15.87:0/2662966709 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520020070 con 0x56138b1d2e80 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa5167fc640 1 -- 172.21.15.87:0/2662966709 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5200a0380 con 0x56138b1d2e80 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa5167fc640 1 -- 172.21.15.87:0/2662966709 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520009de0 con 0x56138b1d2e80 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 0x7fa5180a61e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa528007750 tx=0x7fa5280076e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 -- 172.21.15.87:0/2662966709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 msgr2=0x56138b1db430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cdbb0 tx=0x7fa5200c75a0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 -- 172.21.15.87:0/2662966709 shutdown_connections 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d2e80 0x56138b1db430 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.520+0000 7fa53902d740 1 --2- 172.21.15.87:0/2662966709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1d28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- 172.21.15.87:0/2662966709 >> 172.21.15.87:0/2662966709 conn(0x56138b1ccba0 msgr2=0x56138b1ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- 172.21.15.87:0/2662966709 shutdown_connections 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- 172.21.15.87:0/2662966709 wait complete. 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- start start 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 Processor -- start 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d24d0 0x56138b1db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56138b1de8b0 con 0x56138b1dba00 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56138b0f0500 con 0x56138b1d24d0 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56138b1dea30 con 0x56138b1d2e80 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa53902d740 10 client.0 _ll_register_callbacks cb 0x56138b0beb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa5348f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa5348f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49488/0 (socket says 172.21.15.87:49488) 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa5348f6640 1 -- 172.21.15.87:0/1797301538 learned_addr learned my addr 172.21.15.87:0/1797301538 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.521+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa52ffff640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d24d0 0x56138b1db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 -- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 msgr2=0x56138b1d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 -- 172.21.15.87:0/1797301538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d24d0 msgr2=0x56138b1db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d24d0 0x56138b1db270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 -- 172.21.15.87:0/1797301538 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56138ae493f0 con 0x56138b1dba00 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa5348f6640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa5280119c0 tx=0x7fa5280074a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9954b8172cb3a61e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52801cd80 con 0x56138b1dba00 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fa52801a450 con 0x56138b1dba00 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5200a0380 con 0x56138b1dba00 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa52801a690 con 0x56138b1dba00 2024-04-22T13:42:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa53902d740 10 client.18657 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56138af137d0 con 0x56138b1dba00 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.522+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5200a0380 con 0x56138b1dba00 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa528023070 con 0x56138b1dba00 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 10 client.18657 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 10 client.18657 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 20 client.18657 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b1ebb60 0x56138b1edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 4 client.18657 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56138b0d6930 con 0x56138b1ebb60 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa52ffff640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b1ebb60 0x56138b1edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa50f7fe640 10 client.18657 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.523+0000 7fa52ffff640 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b1ebb60 0x56138b1edfd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6fdcebed62987624 server_cookie=8de85baeec176ab1 in_seq=0 out_seq=0 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa50f7fe640 1 -- 172.21.15.87:0/1797301538 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56138b0d6930 con 0x56138b1ebb60 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa50f7fe640 10 client.18657 handle_command_reply: tid=0 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 client.18657 shutdown 2024-04-22T13:42:09.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 msgr2=0x56138b1d8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa5280119c0 tx=0x7fa5280074a0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 shutdown_connections 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b1ebb60 0x56138b1edfd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1d8b40 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa50f7fe640 0 client.18657 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 --2- 172.21.15.87:0/1797301538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d24d0 0x56138b1db270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 >> 172.21.15.87:0/1797301538 conn(0x56138b1ccba0 msgr2=0x56138b1ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 shutdown_connections 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 1 -- 172.21.15.87:0/1797301538 wait complete. 2024-04-22T13:42:09.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.524+0000 7fa53902d740 20 client.18657 trim_cache size 0 max 16384 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 Processor -- start 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 -- start start 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301b0430 con 0x56138b1d24d0 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301add00 con 0x56138b1dba00 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.537+0000 7fa53902d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301abb70 con 0x56138b1d2e80 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49490/0 (socket says 172.21.15.87:49490) 2024-04-22T13:42:09.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 -- 172.21.15.87:0/3034139930 learned_addr learned my addr 172.21.15.87:0/3034139930 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 -- 172.21.15.87:0/3034139930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 msgr2=0x56138b1d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 -- 172.21.15.87:0/3034139930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 msgr2=0x56138b1dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 -- 172.21.15.87:0/3034139930 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56138b1d9140 con 0x56138b1d24d0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa520021000 tx=0x7fa52000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a104a5c5eebf6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa515ffb640 1 -- 172.21.15.87:0/3034139930 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520020040 con 0x56138b1d24d0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa515ffb640 1 -- 172.21.15.87:0/3034139930 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa520009de0 con 0x56138b1d24d0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.538+0000 7fa515ffb640 1 -- 172.21.15.87:0/3034139930 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5200a0840 con 0x56138b1d24d0 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- 172.21.15.87:0/3034139930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 msgr2=0x56138b1e7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fa520021000 tx=0x7fa52000bf60 comp rx=0 tx=0).stop 2024-04-22T13:42:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- 172.21.15.87:0/3034139930 shutdown_connections 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1dba00 0x56138b1dad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d2e80 0x56138b1d3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- 172.21.15.87:0/3034139930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1d24d0 0x56138b1e7770 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- 172.21.15.87:0/3034139930 >> 172.21.15.87:0/3034139930 conn(0x56138b1ccba0 msgr2=0x56138b1d3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- 172.21.15.87:0/3034139930 shutdown_connections 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- 172.21.15.87:0/3034139930 wait complete. 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- start start 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 Processor -- start 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 0x56138b1df2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301ac1c0 con 0x56138b1dba00 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301ad630 con 0x56138b1d2e80 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5301ac890 con 0x56138b1d24d0 2024-04-22T13:42:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.539+0000 7fa53902d740 10 client.0 _ll_register_callbacks cb 0x7fa528011670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49494/0 (socket says 172.21.15.87:49494) 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 -- 172.21.15.87:0/3895516122 learned_addr learned my addr 172.21.15.87:0/3895516122 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 0x56138b1df2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 -- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 msgr2=0x56138b1e3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1e3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 -- 172.21.15.87:0/3895516122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 msgr2=0x56138b1df2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 0x56138b1df2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 -- 172.21.15.87:0/3895516122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56138b1dff60 con 0x56138b1dba00 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa52f7fe640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 0x56138b1df2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa5348f6640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa528007610 tx=0x7fa528005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c5e7023a6d225d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528025440 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7fa5280255e0 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa520009de0 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa52801cb80 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa53902d740 10 client.18663 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56138b2563e0 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.540+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa520009de0 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa528023040 con 0x56138b1dba00 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 10 client.18663 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 10 client.18663 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 20 client.18663 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b256b30 0x56138b258f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 4 client.18663 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56138b1d8ed0 con 0x56138b256b30 2024-04-22T13:42:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b256b30 0x56138b258f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa50f7fe640 10 client.18663 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa52ffff640 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b256b30 0x56138b258f50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=688c3158f5c88236 server_cookie=8f650ee4a9d79a2e in_seq=0 out_seq=0 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa50f7fe640 1 -- 172.21.15.87:0/3895516122 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56138b1d8ed0 con 0x56138b256b30 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa50f7fe640 10 client.18663 handle_command_reply: tid=0 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.541+0000 7fa53902d740 1 client.18663 shutdown 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 msgr2=0x56138b1df820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fa528007610 tx=0x7fa528005bd0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 shutdown_connections 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56138b256b30 0x56138b258f50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56138b1dba00 0x56138b1df820 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56138b1d2e80 0x56138b1df2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa50f7fe640 0 client.18663 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 --2- 172.21.15.87:0/3895516122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56138b1d24d0 0x56138b1e3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 >> 172.21.15.87:0/3895516122 conn(0x56138b1ccba0 msgr2=0x56138b1d3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 shutdown_connections 2024-04-22T13:42:09.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 1 -- 172.21.15.87:0/3895516122 wait complete. 2024-04-22T13:42:09.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa53902d740 20 client.18663 trim_cache size 0 max 16384 2024-04-22T13:42:09.546 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 msgr2=0x7fa5180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 0x7fa5180a61e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa528007750 tx=0x7fa5280076e0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 msgr2=0x7fa53019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d950 tx=0x7fa52000de10 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 shutdown_connections 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa5180a3d20 0x7fa5180a61e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa530107030 0x7fa5301985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa530101b40 0x7fa53019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 --2- 172.21.15.87:0/3466443792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5301011a0 0x7fa53019e3f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 >> 172.21.15.87:0/3466443792 conn(0x7fa530076c50 msgr2=0x7fa5300fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 shutdown_connections 2024-04-22T13:42:09.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.542+0000 7fa536359640 1 -- 172.21.15.87:0/3466443792 wait complete. 2024-04-22T13:42:09.550 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 -- 172.21.15.87:0/3913031057 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 msgr2=0x7f8fd810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 --2- 172.21.15.87:0/3913031057 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd810dd40 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009ca0 tx=0x7f8fd002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 -- 172.21.15.87:0/3913031057 shutdown_connections 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 --2- 172.21.15.87:0/3913031057 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd810dd40 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 --2- 172.21.15.87:0/3913031057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd8101a50 0x7f8fd8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 --2- 172.21.15.87:0/3913031057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd81010a0 0x7f8fd8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 -- 172.21.15.87:0/3913031057 >> 172.21.15.87:0/3913031057 conn(0x7f8fd8077eb0 msgr2=0x7f8fd80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 -- 172.21.15.87:0/3913031057 shutdown_connections 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.653+0000 7f8fddc31640 1 -- 172.21.15.87:0/3913031057 wait complete. 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 Processor -- start 2024-04-22T13:42:09.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 -- start start 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd81010a0 0x7f8fd8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd819cf80 0x7f8fd819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81121f0 con 0x7f8fd8106170 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd8112070 con 0x7f8fd81010a0 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fddc31640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd8112370 con 0x7f8fd819cf80 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fd6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fd6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50424/0 (socket says 172.21.15.87:50424) 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/2827893371 learned_addr learned my addr 172.21.15.87:0/2827893371 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd81010a0 0x7f8fd8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.654+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd819cf80 0x7f8fd819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/2827893371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd819cf80 msgr2=0x7f8fd819d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd819cf80 0x7f8fd819d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/2827893371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd81010a0 msgr2=0x7f8fd8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd81010a0 0x7f8fd8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/2827893371 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd819da70 con 0x7f8fd8106170 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005fd0 tx=0x7f8fc000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd4ff9640 1 -- 172.21.15.87:0/2827893371 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0017070 con 0x7f8fd8106170 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd4ff9640 1 -- 172.21.15.87:0/2827893371 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc0009c30 con 0x7f8fd8106170 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd819dd60 con 0x7f8fd8106170 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fd4ff9640 1 -- 172.21.15.87:0/2827893371 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0003d30 con 0x7f8fd8106170 2024-04-22T13:42:09.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.655+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd81a5550 con 0x7f8fd8106170 2024-04-22T13:42:09.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.657+0000 7f8fd4ff9640 1 -- 172.21.15.87:0/2827893371 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8fc0003ed0 con 0x7f8fd8106170 2024-04-22T13:42:09.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 Processor -- start 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 -- start start 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x563686941430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x563686943e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56368689d1f0 con 0x5636869384d0 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56368683a4b0 con 0x563686938e80 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fe0829740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5636868991b0 con 0x563686941a00 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50432/0 (socket says 172.21.15.87:50432) 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 -- 172.21.15.87:0/333505110 learned_addr learned my addr 172.21.15.87:0/333505110 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x563686941430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x563686943e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd4ff9640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 0x7f8fc40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 -- 172.21.15.87:0/333505110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 msgr2=0x563686943e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x563686943e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 -- 172.21.15.87:0/333505110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 msgr2=0x563686941430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x563686941430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 -- 172.21.15.87:0/333505110 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56368682c1b0 con 0x5636869384d0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 0x7f8fc40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.658+0000 7f8fd4ff9640 1 -- 172.21.15.87:0/2827893371 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f8fc00d0320 con 0x7f8fd8106170 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0037040 tx=0x7f8fd0005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba5445ccfc320f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fcdffb640 1 -- 172.21.15.87:0/333505110 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0039040 con 0x5636869384d0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fcdffb640 1 -- 172.21.15.87:0/333505110 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd0003e20 con 0x5636869384d0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fcdffb640 1 -- 172.21.15.87:0/333505110 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0034440 con 0x5636869384d0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 -- 172.21.15.87:0/333505110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 msgr2=0x5636869388b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 0x7f8fc40a61e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8fd004c000 tx=0x7f8fd0045000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0037040 tx=0x7f8fd0005f70 comp rx=0 tx=0).stop 2024-04-22T13:42:09.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 -- 172.21.15.87:0/333505110 shutdown_connections 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x563686943e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x563686941430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 --2- 172.21.15.87:0/333505110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x5636869388b0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 -- 172.21.15.87:0/333505110 >> 172.21.15.87:0/333505110 conn(0x563686932ba0 msgr2=0x563686932f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 -- 172.21.15.87:0/333505110 shutdown_connections 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.659+0000 7f8fe0829740 1 -- 172.21.15.87:0/333505110 wait complete. 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 -- start start 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 Processor -- start 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5636869384d0 0x563686941350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x56368693e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563686944970 con 0x563686941a00 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563686856500 con 0x563686938e80 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563686944af0 con 0x5636869384d0 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fe0829740 10 client.0 _ll_register_callbacks cb 0x563686824b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50448/0 (socket says 172.21.15.87:50448) 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fd7fff640 1 -- 172.21.15.87:0/2828694222 learned_addr learned my addr 172.21.15.87:0/2828694222 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x56368693e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.660+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5636869384d0 0x563686941350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 -- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5636869384d0 msgr2=0x563686941350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5636869384d0 0x563686941350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 -- 172.21.15.87:0/2828694222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 msgr2=0x56368693e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x56368693e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 -- 172.21.15.87:0/2828694222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636865af3f0 con 0x563686941a00 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x56368693e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00093a0 tx=0x7f8fd0005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca683a13d807fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0039070 con 0x563686941a00 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f8fd0004040 con 0x563686941a00 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd0003e20 con 0x563686941a00 2024-04-22T13:42:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8fd0034440 con 0x563686941a00 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fe0829740 10 client.18675 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5636866797d0 con 0x563686941a00 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd0003e20 con 0x563686941a00 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8fd003f040 con 0x563686941a00 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fe0829740 10 client.18675 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.661+0000 7f8fe0829740 10 client.18675 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fe0829740 20 client.18675 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x563686951ae0 0x563686953ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fe0829740 4 client.18675 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56368683c930 con 0x563686951ae0 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x563686951ae0 0x563686953ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x563686951ae0 0x563686953ed0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3972ec39ea1ab4fe server_cookie=ed979c6c42e24110 in_seq=0 out_seq=0 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.662+0000 7f8fb6ffd640 10 client.18675 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/2828694222 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56368683c930 con 0x563686951ae0 2024-04-22T13:42:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fb6ffd640 10 client.18675 handle_command_reply: tid=0 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 client.18675 shutdown 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 msgr2=0x56368693ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00093a0 tx=0x7f8fd0005c30 comp rx=0 tx=0).stop 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 shutdown_connections 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x563686951ae0 0x563686953ed0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686941a00 0x56368693ecf0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686938e80 0x56368693e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fb6ffd640 0 client.18675 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 --2- 172.21.15.87:0/2828694222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5636869384d0 0x563686941350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 >> 172.21.15.87:0/2828694222 conn(0x563686932ba0 msgr2=0x5636869348f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 shutdown_connections 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.663+0000 7f8fe0829740 1 -- 172.21.15.87:0/2828694222 wait complete. 2024-04-22T13:42:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.664+0000 7f8fe0829740 20 client.18675 trim_cache size 0 max 16384 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.676+0000 7f8fe0829740 1 Processor -- start 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.676+0000 7f8fe0829740 1 -- start start 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686938e80 0x563686941350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 0x563686956ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81b0330 con 0x5636869384d0 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81adc00 con 0x563686941a00 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fe0829740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81aba70 con 0x563686938e80 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50456/0 (socket says 172.21.15.87:50456) 2024-04-22T13:42:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 -- 172.21.15.87:0/3064597331 learned_addr learned my addr 172.21.15.87:0/3064597331 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 0x563686956ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686938e80 0x563686941350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 -- 172.21.15.87:0/3064597331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686938e80 msgr2=0x563686941350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686938e80 0x563686941350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 -- 172.21.15.87:0/3064597331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 msgr2=0x563686956ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 0x563686956ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 -- 172.21.15.87:0/3064597331 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636869334e0 con 0x5636869384d0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 0x563686956ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009d20 tx=0x7f8fc8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e903575011cd88 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fcd7fa640 1 -- 172.21.15.87:0/3064597331 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80042f0 con 0x5636869384d0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fcd7fa640 1 -- 172.21.15.87:0/3064597331 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc8004490 con 0x5636869384d0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.677+0000 7f8fcd7fa640 1 -- 172.21.15.87:0/3064597331 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8004a00 con 0x5636869384d0 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- 172.21.15.87:0/3064597331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 msgr2=0x563686940e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009d20 tx=0x7f8fc8005f90 comp rx=0 tx=0).stop 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- 172.21.15.87:0/3064597331 shutdown_connections 2024-04-22T13:42:09.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563686941a00 0x563686956ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686938e80 0x563686941350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 --2- 172.21.15.87:0/3064597331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5636869384d0 0x563686940e10 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- 172.21.15.87:0/3064597331 >> 172.21.15.87:0/3064597331 conn(0x563686932ba0 msgr2=0x56368693d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- 172.21.15.87:0/3064597331 shutdown_connections 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- 172.21.15.87:0/3064597331 wait complete. 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 -- start start 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.678+0000 7f8fe0829740 1 Processor -- start 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 0x563686954950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x5636869553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81ac0c0 con 0x563686938e80 2024-04-22T13:42:09.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81ad530 con 0x5636869384d0 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd81ac790 con 0x563686941a00 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 10 client.0 _ll_register_callbacks cb 0x7f8fd0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50458/0 (socket says 172.21.15.87:50458) 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/332134027 learned_addr learned my addr 172.21.15.87:0/332134027 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x5636869553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 0x563686954950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 msgr2=0x5636869553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x5636869553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/332134027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 msgr2=0x563686954950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 0x563686954950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 -- 172.21.15.87:0/332134027 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636869b6cf0 con 0x563686938e80 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd7fff640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x5636869553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:09.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 0x563686954950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fd6ffd640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0009610 tx=0x7f8fc009be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a122f1cb7801e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0009de0 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f8fc009a5d0 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc8004490 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8fc009c2d0 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.679+0000 7f8fe0829740 10 client.18681 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5636869b8060 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc8004490 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8fc00d3cc0 con 0x563686938e80 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 10 client.18681 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 10 client.18681 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 20 client.18681 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5636869b8740 0x5636869bab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 4 client.18681 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:09.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636868aac10 con 0x5636869b8740 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.680+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5636869b8740 0x5636869bab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fd77fe640 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5636869b8740 0x5636869bab60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=577a0488b998caee server_cookie=1b4903233fb67067 in_seq=0 out_seq=0 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fb6ffd640 10 client.18681 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fb6ffd640 1 -- 172.21.15.87:0/332134027 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636868aac10 con 0x5636869b8740 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fb6ffd640 10 client.18681 handle_command_reply: tid=0 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 client.18681 shutdown 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 msgr2=0x563686954e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0009610 tx=0x7f8fc009be10 comp rx=0 tx=0).stop 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 shutdown_connections 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x5636869b8740 0x5636869bab60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563686941a00 0x5636869553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563686938e80 0x563686954e90 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fb6ffd640 0 client.18681 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 --2- 172.21.15.87:0/332134027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5636869384d0 0x563686954950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 >> 172.21.15.87:0/332134027 conn(0x563686932ba0 msgr2=0x563686931370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 shutdown_connections 2024-04-22T13:42:09.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 1 -- 172.21.15.87:0/332134027 wait complete. 2024-04-22T13:42:09.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.681+0000 7f8fe0829740 20 client.18681 trim_cache size 0 max 16384 2024-04-22T13:42:09.685 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 msgr2=0x7f8fc40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 0x7f8fc40a61e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8fd004c000 tx=0x7f8fd0045000 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 msgr2=0x7f8fd8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0005fd0 tx=0x7f8fc000dde0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 shutdown_connections 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f8fc40a3d20 0x7f8fc40a61e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8fd819cf80 0x7f8fd819d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fd8106170 0x7f8fd8198c50 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 --2- 172.21.15.87:0/2827893371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fd81010a0 0x7f8fd8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 >> 172.21.15.87:0/2827893371 conn(0x7f8fd8077eb0 msgr2=0x7f8fd80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 shutdown_connections 2024-04-22T13:42:09.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.682+0000 7f8fddc31640 1 -- 172.21.15.87:0/2827893371 wait complete. 2024-04-22T13:42:09.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:09 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1736320955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:09.689 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:42:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:09 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1736320955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:09.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 -- 172.21.15.87:0/2867398544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101a40 msgr2=0x7f5104105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2867398544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101a40 0x7f5104105a80 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009900 tx=0x7f50f402bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:09.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 -- 172.21.15.87:0/2867398544 shutdown_connections 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2867398544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 0x7f510410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2867398544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101a40 0x7f5104105a80 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2867398544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5104101090 0x7f5104101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 -- 172.21.15.87:0/2867398544 >> 172.21.15.87:0/2867398544 conn(0x7f5104077f50 msgr2=0x7f5104078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 -- 172.21.15.87:0/2867398544 shutdown_connections 2024-04-22T13:42:09.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.792+0000 7f5108e8c640 1 -- 172.21.15.87:0/2867398544 wait complete. 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 Processor -- start 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 -- start start 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 0x7f5104198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f510419cfc0 0x7f510419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f5104111080 con 0x7f5104101090 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5104110f00 con 0x7f5104106160 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5108e8c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5104111200 con 0x7f510419cfc0 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50482/0 (socket says 172.21.15.87:50482) 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 -- 172.21.15.87:0/2554385662 learned_addr learned my addr 172.21.15.87:0/2554385662 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f5101d9b640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 0x7f5104198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 -- 172.21.15.87:0/2554385662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f510419cfc0 msgr2=0x7f510419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f510419cfc0 0x7f510419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 -- 172.21.15.87:0/2554385662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 msgr2=0x7f5104198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 0x7f5104198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.793+0000 7f510259c640 1 -- 172.21.15.87:0/2554385662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510419dab0 con 0x7f5104101090 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f510259c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f50f800c920 tx=0x7f50f800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f50f37fe640 1 -- 172.21.15.87:0/2554385662 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8007c70 con 0x7f5104101090 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f50f37fe640 1 -- 172.21.15.87:0/2554385662 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50f8007e10 con 0x7f5104101090 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510419dda0 con 0x7f5104101090 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f50f37fe640 1 -- 172.21.15.87:0/2554385662 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8018820 con 0x7f5104101090 2024-04-22T13:42:09.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.794+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51041a5590 con 0x7f5104101090 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.796+0000 7f50f37fe640 1 -- 172.21.15.87:0/2554385662 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f50f800f450 con 0x7f5104101090 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.796+0000 7f510ba80740 1 Processor -- start 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 -- start start 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 0x55cf95c6b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 0x55cf95c74430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95c76e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cf95bd01f0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cf95b6d4b0 con 0x55cf95c6be80 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510ba80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cf95bcc1b0 con 0x55cf95c74a00 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f50f37fe640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 0x7f50ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5102d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 0x55cf95c6b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5102d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 0x55cf95c6b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50494/0 (socket says 172.21.15.87:50494) 2024-04-22T13:42:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5102d9d640 1 -- 172.21.15.87:0/1511469199 learned_addr learned my addr 172.21.15.87:0/1511469199 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95c76e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5102d9d640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 0x7f50ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510259c640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 0x55cf95c74430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f50f37fe640 1 -- 172.21.15.87:0/2554385662 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f50f80d09c0 con 0x7f5104101090 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 -- 172.21.15.87:0/1511469199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 msgr2=0x55cf95c74430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 0x55cf95c74430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 -- 172.21.15.87:0/1511469199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 msgr2=0x55cf95c6b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 0x55cf95c6b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5101d9b640 1 -- 172.21.15.87:0/1511469199 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf95b5f1b0 con 0x55cf95c74a00 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f510259c640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 0x55cf95c74430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.797+0000 7f5102d9d640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 0x7f50ec0a61e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f50fc012a30 tx=0x7f50fc009840 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95c76e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5104069a70 tx=0x7f50f4005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ccee2457198c0db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f50f0ff9640 1 -- 172.21.15.87:0/1511469199 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4043020 con 0x55cf95c74a00 2024-04-22T13:42:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f50f0ff9640 1 -- 172.21.15.87:0/1511469199 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50f402c7e0 con 0x55cf95c74a00 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f50f0ff9640 1 -- 172.21.15.87:0/1511469199 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4005410 con 0x55cf95c74a00 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 -- 172.21.15.87:0/1511469199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 msgr2=0x55cf95c76e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95c76e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5104069a70 tx=0x7f50f4005e00 comp rx=0 tx=0).stop 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 -- 172.21.15.87:0/1511469199 shutdown_connections 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95c76e50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6be80 0x55cf95c74430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 --2- 172.21.15.87:0/1511469199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6b4d0 0x55cf95c6b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.798+0000 7f510ba80740 1 -- 172.21.15.87:0/1511469199 >> 172.21.15.87:0/1511469199 conn(0x55cf95c65ba0 msgr2=0x55cf95c65f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- 172.21.15.87:0/1511469199 shutdown_connections 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- 172.21.15.87:0/1511469199 wait complete. 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- start start 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 Processor -- start 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6be80 0x55cf95c716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 0x55cf95c71de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cf95c77910 con 0x55cf95c74a00 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cf95b89500 con 0x55cf95c6b4d0 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cf95c77a90 con 0x55cf95c6be80 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510ba80740 10 client.0 _ll_register_callbacks cb 0x55cf95b57b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f5102d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 0x55cf95c71de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:35298/0 (socket says 172.21.15.87:35298) 2024-04-22T13:42:09.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f510259c640 1 -- 172.21.15.87:0/2021174397 learned_addr learned my addr 172.21.15.87:0/2021174397 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.799+0000 7f5101d9b640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6be80 0x55cf95c716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 -- 172.21.15.87:0/2021174397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6be80 msgr2=0x55cf95c716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6be80 0x55cf95c716c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 -- 172.21.15.87:0/2021174397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 msgr2=0x55cf95c71de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 0x55cf95c71de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 -- 172.21.15.87:0/2021174397 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf958e23f0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f5102d9d640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 0x55cf95c71de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510259c640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50f809c900 tx=0x7f50f801dcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d57c5e425bbfec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f50e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f809c970 con 0x55cf95c6b4d0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f402c7e0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f50f801dd40 con 0x55cf95c6b4d0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f50f80a0d20 con 0x55cf95c6b4d0 2024-04-22T13:42:09.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510ba80740 10 client.27425 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cf959ac7d0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.800+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50f402c7e0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f50f8098fe0 con 0x55cf95c6b4d0 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 10 client.27425 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 10 client.27425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 20 client.27425 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95c84c70 0x55cf95c870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 4 client.27425 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf95b6f930 con 0x55cf95c84c70 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f5101d9b640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95c84c70 0x55cf95c870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f50e9ffb640 10 client.27425 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.801+0000 7f5101d9b640 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95c84c70 0x55cf95c870a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca5b5f51c033a21d server_cookie=670fe1236acdf0b9 in_seq=0 out_seq=0 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f50e9ffb640 1 -- 172.21.15.87:0/2021174397 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf95b6f930 con 0x55cf95c84c70 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f50e9ffb640 10 client.27425 handle_command_reply: tid=0 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 client.27425 shutdown 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 msgr2=0x55cf95c743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50f809c900 tx=0x7f50f801dcd0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 shutdown_connections 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95c84c70 0x55cf95c870a0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c74a00 0x55cf95c71de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f50e9ffb640 0 client.27425 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6be80 0x55cf95c716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 --2- 172.21.15.87:0/2021174397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c743d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 >> 172.21.15.87:0/2021174397 conn(0x55cf95c65ba0 msgr2=0x55cf95c678f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 shutdown_connections 2024-04-22T13:42:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 1 -- 172.21.15.87:0/2021174397 wait complete. 2024-04-22T13:42:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.802+0000 7f510ba80740 20 client.27425 trim_cache size 0 max 16384 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.815+0000 7f510ba80740 1 Processor -- start 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.815+0000 7f510ba80740 1 -- start start 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 0x55cf95c807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c74a00 0x55cf95c67700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51041b0450 con 0x55cf95c6be80 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51041add20 con 0x55cf95c74a00 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510ba80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51041abb90 con 0x55cf95c6b4d0 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50520/0 (socket says 172.21.15.87:50520) 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 -- 172.21.15.87:0/1774655597 learned_addr learned my addr 172.21.15.87:0/1774655597 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5102d9d640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c74a00 0x55cf95c67700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510259c640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 0x55cf95c807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 -- 172.21.15.87:0/1774655597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 msgr2=0x55cf95c807d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 0x55cf95c807d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 -- 172.21.15.87:0/1774655597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c74a00 msgr2=0x55cf95c67700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c74a00 0x55cf95c67700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 -- 172.21.15.87:0/1774655597 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf95c6a790 con 0x55cf95c6be80 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f510259c640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 0x55cf95c807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f5101d9b640 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009900 tx=0x7f50f4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc29da3b0ede42e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f50ebfff640 1 -- 172.21.15.87:0/1774655597 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4043040 con 0x55cf95c6be80 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f50ebfff640 1 -- 172.21.15.87:0/1774655597 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50f402c5d0 con 0x55cf95c6be80 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.816+0000 7f50ebfff640 1 -- 172.21.15.87:0/1774655597 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40046c0 con 0x55cf95c6be80 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- 172.21.15.87:0/1774655597 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 msgr2=0x55cf95c70820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009900 tx=0x7f50f4005fd0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- 172.21.15.87:0/1774655597 shutdown_connections 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c74a00 0x55cf95c67700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c70820 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- 172.21.15.87:0/1774655597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c6b4d0 0x55cf95c807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- 172.21.15.87:0/1774655597 >> 172.21.15.87:0/1774655597 conn(0x55cf95c65ba0 msgr2=0x55cf95c66ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- 172.21.15.87:0/1774655597 shutdown_connections 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- 172.21.15.87:0/1774655597 wait complete. 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- start start 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 Processor -- start 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c77ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95ce91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f51041ac1e0 con 0x55cf95c6be80 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51041ad650 con 0x55cf95c6b4d0 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51041ac8b0 con 0x55cf95c74a00 2024-04-22T13:42:09.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f510ba80740 10 client.0 _ll_register_callbacks cb 0x7f50f800c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.817+0000 7f5101d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50530/0 (socket says 172.21.15.87:50530) 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 -- 172.21.15.87:0/3388577868 learned_addr learned my addr 172.21.15.87:0/3388577868 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f510259c640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c77ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 -- 172.21.15.87:0/3388577868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 msgr2=0x55cf95ce91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95ce91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 -- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 msgr2=0x55cf95c77ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c77ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 -- 172.21.15.87:0/3388577868 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf95ce97e0 con 0x55cf95c6be80 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f5101d9b640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f50f402bc70 tx=0x7f50f402cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b61d8675c41d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f50e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4058020 con 0x55cf95c6be80 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f50f4004240 con 0x55cf95c6be80 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f402c5d0 con 0x55cf95c6be80 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f50f4063050 con 0x55cf95c6be80 2024-04-22T13:42:09.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f510ba80740 10 client.18699 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cf95ceaa50 con 0x55cf95c6be80 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.818+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50f402c5d0 con 0x55cf95c6be80 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f50f4003710 con 0x55cf95c6be80 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 10 client.18699 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 10 client.18699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 20 client.18699 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95ceacf0 0x55cf95ced110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 4 client.18699 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf95bddc10 con 0x55cf95ceacf0 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510259c640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95ceacf0 0x55cf95ced110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510259c640 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95ceacf0 0x55cf95ced110 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daf7e07d238a8de0 server_cookie=7c52df2a582a938 in_seq=0 out_seq=0 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f50e9ffb640 10 client.18699 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f50e9ffb640 1 -- 172.21.15.87:0/3388577868 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf95bddc10 con 0x55cf95ceacf0 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f50e9ffb640 10 client.18699 handle_command_reply: tid=0 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.819+0000 7f510ba80740 1 client.18699 shutdown 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 msgr2=0x55cf95c78430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f50f402bc70 tx=0x7f50f402cc20 comp rx=0 tx=0).stop 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 shutdown_connections 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cf95ceacf0 0x55cf95ced110 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf95c74a00 0x55cf95ce91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cf95c6be80 0x55cf95c78430 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 --2- 172.21.15.87:0/3388577868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf95c6b4d0 0x55cf95c77ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f50e9ffb640 0 client.18699 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 >> 172.21.15.87:0/3388577868 conn(0x55cf95c65ba0 msgr2=0x55cf95c67880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 shutdown_connections 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 1 -- 172.21.15.87:0/3388577868 wait complete. 2024-04-22T13:42:09.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f510ba80740 20 client.18699 trim_cache size 0 max 16384 2024-04-22T13:42:09.824 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:09.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 msgr2=0x7f50ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 0x7f50ec0a61e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f50fc012a30 tx=0x7f50fc009840 comp rx=0 tx=0).stop 2024-04-22T13:42:09.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 msgr2=0x7f5104198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f50f800c920 tx=0x7f50f800cde0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 shutdown_connections 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f50ec0a3d20 0x7f50ec0a61e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f510419cfc0 0x7f510419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5104106160 0x7f5104198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 --2- 172.21.15.87:0/2554385662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5104101090 0x7f5104198700 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 >> 172.21.15.87:0/2554385662 conn(0x7f5104077f50 msgr2=0x7f5104100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 shutdown_connections 2024-04-22T13:42:09.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:09.820+0000 7f5108e8c640 1 -- 172.21.15.87:0/2554385662 wait complete. 2024-04-22T13:42:09.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:09 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1736320955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:10.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:10 smithi132 ceph-mon[31531]: pgmap v322: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:10.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:10 smithi087 ceph-mon[27999]: pgmap v322: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:10.829 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:42:10.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:42:10.829 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:42:10.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:10 smithi120 ceph-mon[31101]: pgmap v322: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 23 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T13:42:10.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 -- 172.21.15.87:0/893551427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff43406b170 msgr2=0x7ff434106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 --2- 172.21.15.87:0/893551427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff43406b170 0x7ff434106ba0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009900 tx=0x7ff41c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:10.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 -- 172.21.15.87:0/893551427 shutdown_connections 2024-04-22T13:42:10.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 --2- 172.21.15.87:0/893551427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434107280 0x7ff43410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 --2- 172.21.15.87:0/893551427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff43406b170 0x7ff434106ba0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 --2- 172.21.15.87:0/893551427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43406a7c0 0x7ff43406aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 -- 172.21.15.87:0/893551427 >> 172.21.15.87:0/893551427 conn(0x7ff434076c50 msgr2=0x7ff434077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 -- 172.21.15.87:0/893551427 shutdown_connections 2024-04-22T13:42:10.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.947+0000 7ff438950640 1 -- 172.21.15.87:0/893551427 wait complete. 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 Processor -- start 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 -- start start 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43406a7c0 0x7ff434198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43419d000 0x7ff43419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341121a0 con 0x7ff434107280 2024-04-22T13:42:10.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112020 con 0x7ff43406a7c0 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff438950640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff434112320 con 0x7ff43419d000 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50538/0 (socket says 172.21.15.87:50538) 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 -- 172.21.15.87:0/2974170511 learned_addr learned my addr 172.21.15.87:0/2974170511 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff43259c640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43406a7c0 0x7ff434198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43419d000 0x7ff43419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 -- 172.21.15.87:0/2974170511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43419d000 msgr2=0x7ff43419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43419d000 0x7ff43419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 -- 172.21.15.87:0/2974170511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43406a7c0 msgr2=0x7ff434198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43406a7c0 0x7ff434198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.948+0000 7ff429d9b640 1 -- 172.21.15.87:0/2974170511 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43419daf0 con 0x7ff434107280 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff429d9b640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff41c036040 tx=0x7ff41c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff42bfff640 1 -- 172.21.15.87:0/2974170511 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c043020 con 0x7ff434107280 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43419dd80 con 0x7ff434107280 2024-04-22T13:42:10.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff42bfff640 1 -- 172.21.15.87:0/2974170511 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff41c02cbe0 con 0x7ff434107280 2024-04-22T13:42:10.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff42bfff640 1 -- 172.21.15.87:0/2974170511 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c003c30 con 0x7ff434107280 2024-04-22T13:42:10.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.949+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a55d0 con 0x7ff434107280 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.951+0000 7ff42bfff640 1 -- 172.21.15.87:0/2974170511 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff41c046020 con 0x7ff434107280 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 Processor -- start 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 -- start start 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1b9e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563ae1af9f00 con 0x563ae1b958e0 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ae1a97590 con 0x563ae1b9ee10 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43b61b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563ae1af5ec0 con 0x563ae1b96290 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff42bfff640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 0x7ff4200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff432d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff432d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50540/0 (socket says 172.21.15.87:50540) 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff432d9d640 1 -- 172.21.15.87:0/2437060376 learned_addr learned my addr 172.21.15.87:0/2437060376 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 0x7ff4200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff43259c640 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1b9e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff42bfff640 1 -- 172.21.15.87:0/2974170511 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7ff41c0f2930 con 0x7ff434107280 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.952+0000 7ff429d9b640 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 -- 172.21.15.87:0/2437060376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 msgr2=0x563ae1b9e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1b9e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 -- 172.21.15.87:0/2437060376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 msgr2=0x563ae1ba1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 -- 172.21.15.87:0/2437060376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae18d47e0 con 0x563ae1b958e0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff42c012d00 tx=0x7ff42c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df0505be097116d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff432d9d640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 0x7ff4200a61e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c017000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff428d99640 1 -- 172.21.15.87:0/2437060376 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c022070 con 0x563ae1b958e0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff428d99640 1 -- 172.21.15.87:0/2437060376 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0124f0 con 0x563ae1b958e0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.953+0000 7ff428d99640 1 -- 172.21.15.87:0/2437060376 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c004d20 con 0x563ae1b958e0 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- 172.21.15.87:0/2437060376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 msgr2=0x563ae1b95cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff42c012d00 tx=0x7ff42c010740 comp rx=0 tx=0).stop 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- 172.21.15.87:0/2437060376 shutdown_connections 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1b9e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 --2- 172.21.15.87:0/2437060376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b95cc0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- 172.21.15.87:0/2437060376 >> 172.21.15.87:0/2437060376 conn(0x563ae1b8ffb0 msgr2=0x563ae1b90390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- 172.21.15.87:0/2437060376 shutdown_connections 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- 172.21.15.87:0/2437060376 wait complete. 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 -- start start 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.954+0000 7ff43b61b740 1 Processor -- start 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b958e0 0x563ae1b9bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1b9c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563ae1ba1d20 con 0x563ae1b96290 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ae1ab3580 con 0x563ae1b9ee10 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563ae1ba1ea0 con 0x563ae1b958e0 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 10 client.0 _ll_register_callbacks cb 0x563ae1ab9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50542/0 (socket says 172.21.15.87:50542) 2024-04-22T13:42:10.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 -- 172.21.15.87:0/129226130 learned_addr learned my addr 172.21.15.87:0/129226130 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff432d9d640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1b9c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43259c640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b958e0 0x563ae1b9bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 -- 172.21.15.87:0/129226130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b958e0 msgr2=0x563ae1b9bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b958e0 0x563ae1b9bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 -- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 msgr2=0x563ae1b9c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1b9c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 -- 172.21.15.87:0/129226130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae1934f70 con 0x563ae1b96290 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff429d9b640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff41c004650 tx=0x7ff41c003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883e6fbf553d6181 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0be350 con 0x563ae1b96290 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7ff41c0bb060 con 0x563ae1b96290 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff42c0124f0 con 0x563ae1b96290 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff41c0d9070 con 0x563ae1b96290 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 10 client.18711 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:10.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.955+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563ae17cc170 con 0x563ae1b96290 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0124f0 con 0x563ae1b96290 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff41c0bec50 con 0x563ae1b96290 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 10 client.18711 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 10 client.18711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 20 client.18711 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1baf400 0x563ae1bb1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 4 client.18711 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ae1b07920 con 0x563ae1baf400 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43259c640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1baf400 0x563ae1bb1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff419ffb640 10 client.18711 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.956+0000 7ff43259c640 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1baf400 0x563ae1bb1810 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5e86e2285c046c server_cookie=4f973fd46763892a in_seq=0 out_seq=0 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff419ffb640 1 -- 172.21.15.87:0/129226130 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ae1b07920 con 0x563ae1baf400 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff419ffb640 10 client.18711 handle_command_reply: tid=0 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 client.18711 shutdown 2024-04-22T13:42:10.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 msgr2=0x563ae1b9c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff41c004650 tx=0x7ff41c003760 comp rx=0 tx=0).stop 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 shutdown_connections 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1baf400 0x563ae1bb1810 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1b9c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b96290 0x563ae1b9c400 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 --2- 172.21.15.87:0/129226130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b958e0 0x563ae1b9bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff419ffb640 0 client.18711 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 >> 172.21.15.87:0/129226130 conn(0x563ae1b8ffb0 msgr2=0x563ae1b91d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 shutdown_connections 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 1 -- 172.21.15.87:0/129226130 wait complete. 2024-04-22T13:42:10.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.957+0000 7ff43b61b740 20 client.18711 trim_cache size 0 max 16384 2024-04-22T13:42:10.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 Processor -- start 2024-04-22T13:42:10.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 -- start start 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b96290 0x563ae1baabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b9ee10 0x563ae1bb7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341b0350 con 0x563ae1b958e0 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341adc20 con 0x563ae1b96290 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43b61b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341aba90 con 0x563ae1b9ee10 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50552/0 (socket says 172.21.15.87:50552) 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.975+0000 7ff43259c640 1 -- 172.21.15.87:0/763627493 learned_addr learned my addr 172.21.15.87:0/763627493 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 -- 172.21.15.87:0/763627493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b9ee10 msgr2=0x563ae1bb7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b9ee10 0x563ae1bb7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 -- 172.21.15.87:0/763627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b96290 msgr2=0x563ae1baabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b96290 0x563ae1baabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 -- 172.21.15.87:0/763627493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae1bb1e90 con 0x563ae1b958e0 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff43259c640 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x563ae1b953b0 tx=0x7ff42400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faababfc4ea5bfc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff41bfff640 1 -- 172.21.15.87:0/763627493 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424005020 con 0x563ae1b958e0 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff41bfff640 1 -- 172.21.15.87:0/763627493 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4240047a0 con 0x563ae1b958e0 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.976+0000 7ff41bfff640 1 -- 172.21.15.87:0/763627493 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424020410 con 0x563ae1b958e0 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- 172.21.15.87:0/763627493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 msgr2=0x563ae1b9e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x563ae1b953b0 tx=0x7ff42400b390 comp rx=0 tx=0).stop 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- 172.21.15.87:0/763627493 shutdown_connections 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b9ee10 0x563ae1bb7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b96290 0x563ae1baabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- 172.21.15.87:0/763627493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1b9e5f0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- 172.21.15.87:0/763627493 >> 172.21.15.87:0/763627493 conn(0x563ae1b8ffb0 msgr2=0x563ae1b9c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- 172.21.15.87:0/763627493 shutdown_connections 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- 172.21.15.87:0/763627493 wait complete. 2024-04-22T13:42:10.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- start start 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 Processor -- start 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1ba3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ac0e0 con 0x563ae1b958e0 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ad550 con 0x563ae1b9ee10 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ac7b0 con 0x563ae1b96290 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43b61b740 10 client.0 _ll_register_callbacks cb 0x7ff41c0bac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50556/0 (socket says 172.21.15.87:50556) 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.977+0000 7ff43259c640 1 -- 172.21.15.87:0/706945730 learned_addr learned my addr 172.21.15.87:0/706945730 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff432d9d640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff429d9b640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1ba3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 -- 172.21.15.87:0/706945730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 msgr2=0x563ae1ba3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1ba3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 -- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 msgr2=0x563ae1ba3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 -- 172.21.15.87:0/706945730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae1ba44b0 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43259c640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff424005880 tx=0x7ff42400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3569e3aaa057826 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42402e020 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7ff4240203b0 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4240047a0 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff424039050 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 10 client.18717 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563ae1c1ad30 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4240047a0 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff42402a440 con 0x563ae1b958e0 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 10 client.18717 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 10 client.18717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:10.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.978+0000 7ff43b61b740 20 client.18717 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1c1b470 0x563ae1c1d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff43b61b740 4 client.18717 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563ae1b01790 con 0x563ae1c1b470 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff429d9b640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1c1b470 0x563ae1c1d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff419ffb640 10 client.18717 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:10.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.979+0000 7ff429d9b640 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1c1b470 0x563ae1c1d890 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aff3eec8afcebdc server_cookie=24926069f9a6c574 in_seq=0 out_seq=0 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.980+0000 7ff419ffb640 1 -- 172.21.15.87:0/706945730 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563ae1b01790 con 0x563ae1c1b470 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.980+0000 7ff419ffb640 10 client.18717 handle_command_reply: tid=0 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 client.18717 shutdown 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 msgr2=0x563ae1ba73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff424005880 tx=0x7ff42400bea0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 shutdown_connections 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ae1c1b470 0x563ae1c1d890 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ae1b9ee10 0x563ae1ba3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ae1b96290 0x563ae1ba3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff419ffb640 0 client.18717 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 --2- 172.21.15.87:0/706945730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ae1b958e0 0x563ae1ba73f0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 >> 172.21.15.87:0/706945730 conn(0x563ae1b8ffb0 msgr2=0x563ae1b8e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 shutdown_connections 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 1 -- 172.21.15.87:0/706945730 wait complete. 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff43b61b740 20 client.18717 trim_cache size 0 max 16384 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:10.983 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:42:10.984 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "4adb5be7-52c0-49ba-b9bc-6b5949107a37", 2024-04-22T13:42:10.984 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:42:10.984 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 msgr2=0x7ff4200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 0x7ff4200a61e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c017000 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 msgr2=0x7ff434198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff41c036040 tx=0x7ff41c02c7e0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 shutdown_connections 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff4200a3d20 0x7ff4200a61e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff43419d000 0x7ff43419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff434107280 0x7ff434198c80 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 --2- 172.21.15.87:0/2974170511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43406a7c0 0x7ff434198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 >> 172.21.15.87:0/2974170511 conn(0x7ff434076c50 msgr2=0x7ff4340ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 shutdown_connections 2024-04-22T13:42:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:10.981+0000 7ff438950640 1 -- 172.21.15.87:0/2974170511 wait complete. 2024-04-22T13:42:10.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:42:10.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:42:11.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 -- 172.21.15.87:0/2136193236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 msgr2=0x7f3d4c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/2136193236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c102580 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009ca0 tx=0x7f3d4802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 -- 172.21.15.87:0/2136193236 shutdown_connections 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/2136193236 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 0x7f3d4c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/2136193236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c102b50 0x7f3d4c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/2136193236 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c102580 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 -- 172.21.15.87:0/2136193236 >> 172.21.15.87:0/2136193236 conn(0x7f3d4c0793e0 msgr2=0x7f3d4c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 -- 172.21.15.87:0/2136193236 shutdown_connections 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.092+0000 7f3d53c8c640 1 -- 172.21.15.87:0/2136193236 wait complete. 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 Processor -- start 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 -- start start 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 0x7f3d4c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c19cf60 0x7f3d4c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1121b0 con 0x7f3d4c1021a0 2024-04-22T13:42:11.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c112030 con 0x7f3d4c19cf60 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d53c8c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c112330 con 0x7f3d4c107270 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d51a28640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d51a28640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50574/0 (socket says 172.21.15.87:50574) 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d51a28640 1 -- 172.21.15.87:0/620572444 learned_addr learned my addr 172.21.15.87:0/620572444 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.093+0000 7f3d51227640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 0x7f3d4c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d52229640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c19cf60 0x7f3d4c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 -- 172.21.15.87:0/620572444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 msgr2=0x7f3d4c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 0x7f3d4c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 -- 172.21.15.87:0/620572444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c19cf60 msgr2=0x7f3d4c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c19cf60 0x7f3d4c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 -- 172.21.15.87:0/620572444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4c19da50 con 0x7f3d4c1021a0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d51a28640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f3d48005e30 tx=0x7f3d48034780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d42ffd640 1 -- 172.21.15.87:0/620572444 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48039040 con 0x7f3d4c1021a0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d42ffd640 1 -- 172.21.15.87:0/620572444 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d48044070 con 0x7f3d4c1021a0 2024-04-22T13:42:11.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4c19dce0 con 0x7f3d4c1021a0 2024-04-22T13:42:11.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d42ffd640 1 -- 172.21.15.87:0/620572444 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48003e60 con 0x7f3d4c1021a0 2024-04-22T13:42:11.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.094+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4c1a54a0 con 0x7f3d4c1021a0 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.096+0000 7f3d42ffd640 1 -- 172.21.15.87:0/620572444 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3d480324c0 con 0x7f3d4c1021a0 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 Processor -- start 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 -- start start 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995c2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d42ffd640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 0x7f3d380a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995cba80 0x556d995cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556d995271e0 con 0x556d995c2ea0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d994c4390 con 0x556d995cba80 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d56883740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556d995231a0 con 0x556d995c2580 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d52229640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 0x7f3d380a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50576/0 (socket says 172.21.15.87:50576) 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 -- 172.21.15.87:0/802812703 learned_addr learned my addr 172.21.15.87:0/802812703 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d42ffd640 1 -- 172.21.15.87:0/620572444 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f3d480f2a60 con 0x7f3d4c1021a0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51a28640 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d52229640 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995cba80 0x556d995cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 -- 172.21.15.87:0/802812703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 msgr2=0x556d995c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 -- 172.21.15.87:0/802812703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995cba80 msgr2=0x556d995cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995cba80 0x556d995cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.097+0000 7f3d51227640 1 -- 172.21.15.87:0/802812703 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d993037d0 con 0x556d995c2ea0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d52229640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 0x7f3d380a6260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400be00 tx=0x7f3d44014840 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d51227640 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00bc40 tx=0x7f3d3c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71dca9fb54c6923d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d37fff640 1 -- 172.21.15.87:0/802812703 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c015070 con 0x556d995c2ea0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d37fff640 1 -- 172.21.15.87:0/802812703 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d3c01a910 con 0x556d995c2ea0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d37fff640 1 -- 172.21.15.87:0/802812703 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c011800 con 0x556d995c2ea0 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.098+0000 7f3d56883740 1 -- 172.21.15.87:0/802812703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 msgr2=0x556d995cb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00bc40 tx=0x7f3d3c00f730 comp rx=0 tx=0).stop 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- 172.21.15.87:0/802812703 shutdown_connections 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995cba80 0x556d995cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995cb540 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- 172.21.15.87:0/802812703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- 172.21.15.87:0/802812703 >> 172.21.15.87:0/802812703 conn(0x556d995bccb0 msgr2=0x556d995bd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- 172.21.15.87:0/802812703 shutdown_connections 2024-04-22T13:42:11.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- 172.21.15.87:0/802812703 wait complete. 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- start start 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 Processor -- start 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2580 0x556d995c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x556d995ce870 con 0x556d995c2ea0 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d994e02f0 con 0x556d995c2580 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556d995ce9f0 con 0x556d995cba80 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.099+0000 7f3d56883740 10 client.0 _ll_register_callbacks cb 0x556d994ed580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50590/0 (socket says 172.21.15.87:50590) 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 -- 172.21.15.87:0/1146983919 learned_addr learned my addr 172.21.15.87:0/1146983919 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d52229640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51a28640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2580 0x556d995c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 -- 172.21.15.87:0/1146983919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 msgr2=0x556d995c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 -- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2580 msgr2=0x556d995c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2580 0x556d995c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 -- 172.21.15.87:0/1146983919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d992393f0 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d51227640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00bc10 tx=0x7f3d3c01a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1448446c57e7a352 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d357fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c028060 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f3d3c01aac0 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d3c01a910 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3d3c015070 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d56883740 10 client.18729 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.100+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556d991f90f0 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d3c01a910 con 0x556d995c2ea0 2024-04-22T13:42:11.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3d3c0048b0 con 0x556d995c2ea0 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 10 client.18729 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 10 client.18729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 20 client.18729 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d995dbb50 0x556d995ddf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 4 client.18729 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.101+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d99534c00 con 0x556d995dbb50 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.102+0000 7f3d51a28640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d995dbb50 0x556d995ddf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.102+0000 7f3d357fa640 10 client.18729 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.102+0000 7f3d51a28640 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d995dbb50 0x556d995ddf40 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6879a27873a9fd server_cookie=80fae1a7aa0040c3 in_seq=0 out_seq=0 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d357fa640 1 -- 172.21.15.87:0/1146983919 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d99534c00 con 0x556d995dbb50 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d357fa640 10 client.18729 handle_command_reply: tid=0 2024-04-22T13:42:11.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 client.18729 shutdown 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 msgr2=0x556d995c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00bc10 tx=0x7f3d3c01a480 comp rx=0 tx=0).stop 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 shutdown_connections 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d995dbb50 0x556d995ddf40 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d357fa640 0 client.18729 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2ea0 0x556d995c9040 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 --2- 172.21.15.87:0/1146983919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2580 0x556d995c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 >> 172.21.15.87:0/1146983919 conn(0x556d995bccb0 msgr2=0x556d995bea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 shutdown_connections 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 1 -- 172.21.15.87:0/1146983919 wait complete. 2024-04-22T13:42:11.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.103+0000 7f3d56883740 20 client.18729 trim_cache size 0 max 16384 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.120+0000 7f3d56883740 1 Processor -- start 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.120+0000 7f3d56883740 1 -- start start 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.120+0000 7f3d56883740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.120+0000 7f3d56883740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d56883740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d56883740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1b02d0 con 0x556d995c2580 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d56883740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1adba0 con 0x556d995c2ea0 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d56883740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1aba10 con 0x556d995cba80 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50606/0 (socket says 172.21.15.87:50606) 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 -- 172.21.15.87:0/337812602 learned_addr learned my addr 172.21.15.87:0/337812602 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:11.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 -- 172.21.15.87:0/337812602 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 msgr2=0x556d995c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 -- 172.21.15.87:0/337812602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 msgr2=0x556d995c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 -- 172.21.15.87:0/337812602 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d995c46d0 con 0x556d995c2580 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d51a28640 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f3d480bb4f0 tx=0x7f3d4807e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b968a9bea7209996 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d377fe640 1 -- 172.21.15.87:0/337812602 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d480bf220 con 0x556d995c2580 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d377fe640 1 -- 172.21.15.87:0/337812602 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d480bfe50 con 0x556d995c2580 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.121+0000 7f3d377fe640 1 -- 172.21.15.87:0/337812602 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d480bcbc0 con 0x556d995c2580 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- 172.21.15.87:0/337812602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 msgr2=0x556d995d7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f3d480bb4f0 tx=0x7f3d4807e640 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- 172.21.15.87:0/337812602 shutdown_connections 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995cba80 0x556d995c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- 172.21.15.87:0/337812602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995c2580 0x556d995d7730 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- 172.21.15.87:0/337812602 >> 172.21.15.87:0/337812602 conn(0x556d995bccb0 msgr2=0x556d995c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- 172.21.15.87:0/337812602 shutdown_connections 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- 172.21.15.87:0/337812602 wait complete. 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 -- start start 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 Processor -- start 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.122+0000 7f3d56883740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1ac060 con 0x556d995cba80 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1ad4d0 con 0x556d995c2ea0 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1ac730 con 0x556d995c2580 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 10 client.0 _ll_register_callbacks cb 0x7f3d3c00ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50614/0 (socket says 172.21.15.87:50614) 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 -- 172.21.15.87:0/424613174 learned_addr learned my addr 172.21.15.87:0/424613174 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:11.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d51a28640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d51227640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 -- 172.21.15.87:0/424613174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 msgr2=0x556d995d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995d40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 -- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 msgr2=0x556d995cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 -- 172.21.15.87:0/424613174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d995d0230 con 0x556d995cba80 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d51227640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d52229640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f3d44005fd0 tx=0x7f3d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27fe3288a392e16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d357fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d44026070 con 0x556d995cba80 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8499+0+0 (secure 0 0 0) 0x7f3d44005220 con 0x556d995cba80 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d480bfe50 con 0x556d995cba80 2024-04-22T13:42:11.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3d44031050 con 0x556d995cba80 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 10 client.18735 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.123+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556d99641ec0 con 0x556d995cba80 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d480bfe50 con 0x556d995cba80 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3d4401d050 con 0x556d995cba80 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 10 client.18735 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 10 client.18735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 20 client.18735 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d99642670 0x556d99644a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 4 client.18735 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556d9952ea70 con 0x556d99642670 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d51a28640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d99642670 0x556d99644a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d357fa640 10 client.18735 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.124+0000 7f3d51a28640 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d99642670 0x556d99644a90 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1d603b46c01b85 server_cookie=f6fa6c0709ae9e7f in_seq=0 out_seq=0 2024-04-22T13:42:11.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d357fa640 1 -- 172.21.15.87:0/424613174 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556d9952ea70 con 0x556d99642670 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d357fa640 10 client.18735 handle_command_reply: tid=0 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 client.18735 shutdown 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 msgr2=0x556d995cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f3d44005fd0 tx=0x7f3d440046c0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 shutdown_connections 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x556d99642670 0x556d99644a90 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x556d995cba80 0x556d995cfa60 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d995c2ea0 0x556d995cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d357fa640 0 client.18735 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 --2- 172.21.15.87:0/424613174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d995c2580 0x556d995d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 >> 172.21.15.87:0/424613174 conn(0x556d995bccb0 msgr2=0x556d995be890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 shutdown_connections 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 1 -- 172.21.15.87:0/424613174 wait complete. 2024-04-22T13:42:11.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d56883740 20 client.18735 trim_cache size 0 max 16384 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "4adb5be7-52c0-49ba-b9bc-6b5949107a37": { 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "4adb5be7-52c0-49ba-b9bc-6b5949107a37", 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:11.129 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 msgr2=0x7f3d380a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 0x7f3d380a6260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400be00 tx=0x7f3d44014840 comp rx=0 tx=0).stop 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 msgr2=0x7f3d4c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f3d48005e30 tx=0x7f3d48034780 comp rx=0 tx=0).stop 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 shutdown_connections 2024-04-22T13:42:11.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3d380a3da0 0x7f3d380a6260 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d4c19cf60 0x7f3d4c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c107270 0x7f3d4c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 --2- 172.21.15.87:0/620572444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d4c1021a0 0x7f3d4c1986f0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.125+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 >> 172.21.15.87:0/620572444 conn(0x7f3d4c0793e0 msgr2=0x7f3d4c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.126+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 shutdown_connections 2024-04-22T13:42:11.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:11.126+0000 7f3d53c8c640 1 -- 172.21.15.87:0/620572444 wait complete. 2024-04-22T13:42:11.134 INFO:tasks.cephfs.filesystem:scrub status for tag:4adb5be7-52c0-49ba-b9bc-6b5949107a37 - {'path': '/', 'tag': '4adb5be7-52c0-49ba-b9bc-6b5949107a37', 'options': 'recursive,force'} 2024-04-22T13:42:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:11 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:42:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:11 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:11 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:42:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:11 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:11 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:42:11.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:11 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:42:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:11 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:11 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:42:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:11 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:11 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:42:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:11 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:42:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:11 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:11 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:42:11.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:11 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:11.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:11 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:42:12.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:12 smithi132 ceph-mon[31531]: pgmap v323: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-04-22T13:42:12.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:12 smithi087 ceph-mon[27999]: pgmap v323: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-04-22T13:42:12.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:12 smithi120 ceph-mon[31101]: pgmap v323: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-04-22T13:42:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.172 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:13 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:13 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:14.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:13 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:42:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:42:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:42:15.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: pgmap v324: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:15.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:14 smithi120 ceph-mon[31101]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T13:42:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:42:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:42:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:42:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:42:15.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: pgmap v324: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:15.214 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:14 smithi132 ceph-mon[31531]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi120 to 176.2M 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi120 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi087 to 176.2M 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi087 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-04-22T13:42:15.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: Unable to set osd_memory_target on smithi132 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-22T13:42:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:42:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:42:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:42:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: pgmap v324: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:15.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:14 smithi087 ceph-mon[27999]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T13:42:16.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:16 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:42:16.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:16 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:42:16.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:16 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:42:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 -- 172.21.15.87:0/2572203889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 msgr2=0x7f2bb01014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/2572203889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01014d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009900 tx=0x7f2ba002bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 -- 172.21.15.87:0/2572203889 shutdown_connections 2024-04-22T13:42:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/2572203889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 0x7f2bb010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.145 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/2572203889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb0101aa0 0x7f2bb0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/2572203889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01014d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 -- 172.21.15.87:0/2572203889 >> 172.21.15.87:0/2572203889 conn(0x7f2bb0077fd0 msgr2=0x7f2bb00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 -- 172.21.15.87:0/2572203889 shutdown_connections 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.147+0000 7f2bb68c1640 1 -- 172.21.15.87:0/2572203889 wait complete. 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 Processor -- start 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 -- start start 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 0x7f2bb0198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb019cf80 0x7f2bb019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01110d0 con 0x7f2bb01010f0 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0110f50 con 0x7f2bb0106170 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2bb68c1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0111250 con 0x7f2bb019cf80 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.146 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50632/0 (socket says 172.21.15.87:50632) 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2baffff640 1 -- 172.21.15.87:0/3017356817 learned_addr learned my addr 172.21.15.87:0/3017356817 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.148+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 0x7f2bb0198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb019cf80 0x7f2bb019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 -- 172.21.15.87:0/3017356817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb019cf80 msgr2=0x7f2bb019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb019cf80 0x7f2bb019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 -- 172.21.15.87:0/3017356817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 msgr2=0x7f2bb0198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 0x7f2bb0198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 -- 172.21.15.87:0/3017356817 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb019da70 con 0x7f2bb01010f0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2baffff640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0005940 tx=0x7f2ba002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bad7fa640 1 -- 172.21.15.87:0/3017356817 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0043020 con 0x7f2bb01010f0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb019dd00 con 0x7f2bb01010f0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bad7fa640 1 -- 172.21.15.87:0/3017356817 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba002ca70 con 0x7f2bb01010f0 2024-04-22T13:42:17.147 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb01a5550 con 0x7f2bb01010f0 2024-04-22T13:42:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.149+0000 7f2bad7fa640 1 -- 172.21.15.87:0/3017356817 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0003990 con 0x7f2bb01010f0 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.151+0000 7f2bad7fa640 1 -- 172.21.15.87:0/3017356817 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ba0046020 con 0x7f2bb01010f0 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bad7fa640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 0x7f2b980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 Processor -- start 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 -- start start 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 0x7f2b980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bad7fa640 1 -- 172.21.15.87:0/3017356817 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f2ba00f27a0 con 0x7f2bb01010f0 2024-04-22T13:42:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea7094620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea7094b60 0x563ea7094fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563ea6ff98d0 con 0x563ea709e0e0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ea6f96e80 con 0x563ea7094b60 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb958d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563ea6ff5890 con 0x563ea70958d0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50648/0 (socket says 172.21.15.87:50648) 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 -- 172.21.15.87:0/4289570154 learned_addr learned my addr 172.21.15.87:0/4289570154 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea7094b60 0x563ea7094fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2baffff640 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea7094620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 0x7f2b980a6280 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0105c40 tx=0x7f2ba4011440 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 -- 172.21.15.87:0/4289570154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 msgr2=0x563ea7094620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea7094620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 -- 172.21.15.87:0/4289570154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea7094b60 msgr2=0x563ea7094fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea7094b60 0x563ea7094fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.152+0000 7f2bb4e5e640 1 -- 172.21.15.87:0/4289570154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea6fb8320 con 0x563ea709e0e0 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009e30 tx=0x7f2ba800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b679bc910ab92ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2b967fc640 1 -- 172.21.15.87:0/4289570154 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8011440 con 0x563ea709e0e0 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2b967fc640 1 -- 172.21.15.87:0/4289570154 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba8011a90 con 0x563ea709e0e0 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2b967fc640 1 -- 172.21.15.87:0/4289570154 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba801bad0 con 0x563ea709e0e0 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 -- 172.21.15.87:0/4289570154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 msgr2=0x563ea70a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009e30 tx=0x7f2ba800f730 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 -- 172.21.15.87:0/4289570154 shutdown_connections 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea709e0e0 0x563ea70a04d0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea7094b60 0x563ea7094fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 --2- 172.21.15.87:0/4289570154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea7094620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 -- 172.21.15.87:0/4289570154 >> 172.21.15.87:0/4289570154 conn(0x563ea705bc90 msgr2=0x563ea708f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 -- 172.21.15.87:0/4289570154 shutdown_connections 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.153+0000 7f2bb958d740 1 -- 172.21.15.87:0/4289570154 wait complete. 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 -- start start 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 Processor -- start 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea7094b60 0x563ea709da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x563ea70a0e10 con 0x563ea70958d0 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ea6f81eb0 con 0x563ea709e0e0 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563ea70a0f90 con 0x563ea7094b60 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2bb958d740 10 client.0 _ll_register_callbacks cb 0x563ea6faeb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2baf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2baf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50664/0 (socket says 172.21.15.87:50664) 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.154+0000 7f2baf7fe640 1 -- 172.21.15.87:0/52725733 learned_addr learned my addr 172.21.15.87:0/52725733 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:17.157 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baffff640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea7094b60 0x563ea709da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 -- 172.21.15.87:0/52725733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea7094b60 msgr2=0x563ea709da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea7094b60 0x563ea709da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 -- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 msgr2=0x563ea709b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 -- 172.21.15.87:0/52725733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea6d0c210 con 0x563ea70958d0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400fce0 tx=0x7f2ba4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c0c9392eb4ad4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2b8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4017030 con 0x563ea70958d0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f2ba40047a0 con 0x563ea70958d0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba8011a90 con 0x563ea70958d0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2ba40049a0 con 0x563ea70958d0 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2bb958d740 10 client.18747 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:17.158 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.155+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563ea6dd8620 con 0x563ea70958d0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba8011a90 con 0x563ea70958d0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2ba4025030 con 0x563ea70958d0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 10 client.18747 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 10 client.18747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 20 client.18747 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea70ae3b0 0x563ea70b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 4 client.18747 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ea700d880 con 0x563ea70ae3b0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2baffff640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea70ae3b0 0x563ea70b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.156+0000 7f2baffff640 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea70ae3b0 0x563ea70b07f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c89e7dd31836b65 server_cookie=f261452b56870bd5 in_seq=0 out_seq=0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2b8f7fe640 10 client.18747 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/52725733 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ea700d880 con 0x563ea70ae3b0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2b8f7fe640 10 client.18747 handle_command_reply: tid=0 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 client.18747 shutdown 2024-04-22T13:42:17.159 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 msgr2=0x563ea709ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400fce0 tx=0x7f2ba4004730 comp rx=0 tx=0).stop 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 shutdown_connections 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea70ae3b0 0x563ea70b07f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2b8f7fe640 0 client.18747 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea70958d0 0x563ea709ae20 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 --2- 172.21.15.87:0/52725733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea7094b60 0x563ea709da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.157+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 >> 172.21.15.87:0/52725733 conn(0x563ea705bc90 msgr2=0x563ea70915a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.158+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 shutdown_connections 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.158+0000 7f2bb958d740 1 -- 172.21.15.87:0/52725733 wait complete. 2024-04-22T13:42:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.158+0000 7f2bb958d740 20 client.18747 trim_cache size 0 max 16384 2024-04-22T13:42:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.172+0000 7f2bb958d740 1 Processor -- start 2024-04-22T13:42:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.172+0000 7f2bb958d740 1 -- start start 2024-04-22T13:42:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.170 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea709d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01b02d0 con 0x563ea7094b60 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01adba0 con 0x563ea709e0e0 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb958d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01aba10 con 0x563ea70958d0 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50670/0 (socket says 172.21.15.87:50670) 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 -- 172.21.15.87:0/2062064596 learned_addr learned my addr 172.21.15.87:0/2062064596 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea709d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 -- 172.21.15.87:0/2062064596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 msgr2=0x563ea709d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea709d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 -- 172.21.15.87:0/2062064596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 msgr2=0x563ea709da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.171 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 -- 172.21.15.87:0/2062064596 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea70acd70 con 0x563ea7094b60 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2baffff640 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00f00f0 tx=0x7f2ba00f2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a16ec03256ec5f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2b95ffb640 1 -- 172.21.15.87:0/2062064596 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00c2c40 con 0x563ea7094b60 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.173+0000 7f2b95ffb640 1 -- 172.21.15.87:0/2062064596 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba00c7500 con 0x563ea7094b60 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2b95ffb640 1 -- 172.21.15.87:0/2062064596 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00c72d0 con 0x563ea7094b60 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 -- 172.21.15.87:0/2062064596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 msgr2=0x563ea70a9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00f00f0 tx=0x7f2ba00f2040 comp rx=0 tx=0).stop 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 -- 172.21.15.87:0/2062064596 shutdown_connections 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea709e0e0 0x563ea709da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea70958d0 0x563ea709d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 --2- 172.21.15.87:0/2062064596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea70a9d40 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 -- 172.21.15.87:0/2062064596 >> 172.21.15.87:0/2062064596 conn(0x563ea705bc90 msgr2=0x563ea708f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 -- 172.21.15.87:0/2062064596 shutdown_connections 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.174+0000 7f2bb958d740 1 -- 172.21.15.87:0/2062064596 wait complete. 2024-04-22T13:42:17.172 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 -- start start 2024-04-22T13:42:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 Processor -- start 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea70958d0 0x563ea709f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea709e0e0 0x563ea70a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01ac060 con 0x563ea7094b60 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01ad4d0 con 0x563ea70958d0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01ac730 con 0x563ea709e0e0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb958d740 10 client.0 _ll_register_callbacks cb 0x7f2ba400f3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2baffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50676/0 (socket says 172.21.15.87:50676) 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2baffff640 1 -- 172.21.15.87:0/3107387685 learned_addr learned my addr 172.21.15.87:0/3107387685 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea70958d0 0x563ea709f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.175+0000 7f2bb4e5e640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea709e0e0 0x563ea70a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 -- 172.21.15.87:0/3107387685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea709e0e0 msgr2=0x563ea70a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea709e0e0 0x563ea70a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 -- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea70958d0 msgr2=0x563ea709f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea70958d0 0x563ea709f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 -- 172.21.15.87:0/3107387685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea7093e30 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2baffff640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006010 tx=0x7f2ba00df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8693ae8f79063991 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0088060 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f2ba00c7070 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba00c7500 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2ba00c2c40 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 10 client.18753 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563ea7114e80 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba00c7500 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2ba0089250 con 0x563ea7094b60 2024-04-22T13:42:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 10 client.18753 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 10 client.18753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 20 client.18753 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.176+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea7115320 0x563ea7117740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.177+0000 7f2bb958d740 4 client.18753 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.177+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ea6f993d0 con 0x563ea7115320 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.177+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea7115320 0x563ea7117740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.177+0000 7f2b8f7fe640 10 client.18753 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.177+0000 7f2baf7fe640 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea7115320 0x563ea7117740 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c298bcc1e8c0ec87 server_cookie=215dfb5dbccc270d in_seq=0 out_seq=0 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.184+0000 7f2b8f7fe640 1 -- 172.21.15.87:0/3107387685 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ea6f993d0 con 0x563ea7115320 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.184+0000 7f2b8f7fe640 10 client.18753 handle_command_reply: tid=0 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.184+0000 7f2bb958d740 1 client.18753 shutdown 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 msgr2=0x563ea709f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006010 tx=0x7f2ba00df040 comp rx=0 tx=0).stop 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 shutdown_connections 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x563ea7115320 0x563ea7117740 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ea709e0e0 0x563ea70a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2b8f7fe640 0 client.18753 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ea70958d0 0x563ea709f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 --2- 172.21.15.87:0/3107387685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x563ea7094b60 0x563ea709f0a0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 >> 172.21.15.87:0/3107387685 conn(0x563ea705bc90 msgr2=0x563ea708e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 shutdown_connections 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 1 -- 172.21.15.87:0/3107387685 wait complete. 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb958d740 20 client.18753 trim_cache size 0 max 16384 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:42:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:42:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 msgr2=0x7f2b980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 0x7f2b980a6280 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0105c40 tx=0x7f2ba4011440 comp rx=0 tx=0).stop 2024-04-22T13:42:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 msgr2=0x7f2bb01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0005940 tx=0x7f2ba002c5f0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 shutdown_connections 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2b980a3dc0 0x7f2b980a6280 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bb019cf80 0x7f2bb019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bb0106170 0x7f2bb0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 --2- 172.21.15.87:0/3017356817 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bb01010f0 0x7f2bb01986c0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.185+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 >> 172.21.15.87:0/3017356817 conn(0x7f2bb0077fd0 msgr2=0x7f2bb0069960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.186+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 shutdown_connections 2024-04-22T13:42:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:17.186+0000 7f2bb68c1640 1 -- 172.21.15.87:0/3017356817 wait complete. 2024-04-22T13:42:17.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:17 smithi087 ceph-mon[27999]: pgmap v326: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:17.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:17 smithi120 ceph-mon[31101]: pgmap v326: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:17.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:17 smithi132 ceph-mon[31531]: pgmap v326: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 10 op/s 2024-04-22T13:42:18.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:18 smithi120 ceph-mon[31101]: pgmap v327: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:18.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:18 smithi132 ceph-mon[31531]: pgmap v327: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:18.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:18 smithi087 ceph-mon[27999]: pgmap v327: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:18.742 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.848+0000 7f9569ffb640 1 -- 172.21.15.87:0/3772621755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95640690e0 msgr2=0x7f9564069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.848+0000 7f9569ffb640 1 --2- 172.21.15.87:0/3772621755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95640690e0 0x7f9564069540 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f954c009900 tx=0x7f954c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- 172.21.15.87:0/3772621755 shutdown_connections 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- 172.21.15.87:0/3772621755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564069a80 0x7f956410eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- 172.21.15.87:0/3772621755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95640690e0 0x7f9564069540 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- 172.21.15.87:0/3772621755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9564107fe0 0x7f95641083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- 172.21.15.87:0/3772621755 >> 172.21.15.87:0/3772621755 conn(0x7f9564077e30 msgr2=0x7f9564078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- 172.21.15.87:0/3772621755 shutdown_connections 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- 172.21.15.87:0/3772621755 wait complete. 2024-04-22T13:42:18.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 Processor -- start 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- start start 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 0x7f9564198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564107fe0 0x7f956419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9564112140 con 0x7f9564069a80 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9564111fc0 con 0x7f95640690e0 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.849+0000 7f9569ffb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95641122c0 con 0x7f9564107fe0 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50682/0 (socket says 172.21.15.87:50682) 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 -- 172.21.15.87:0/1325172545 learned_addr learned my addr 172.21.15.87:0/1325172545 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9563fff640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564107fe0 0x7f956419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f95637fe640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 0x7f9564198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 -- 172.21.15.87:0/1325172545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564107fe0 msgr2=0x7f956419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:18.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564107fe0 0x7f956419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 -- 172.21.15.87:0/1325172545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 msgr2=0x7f9564198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 0x7f9564198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 -- 172.21.15.87:0/1325172545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956419d670 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f95637fe640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 0x7f9564198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9562ffd640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f954c006010 tx=0x7f954c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c043020 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f954c004440 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956419d900 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c003b50 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.850+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95641a51f0 con 0x7f9564069a80 2024-04-22T13:42:18.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.851+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9564068de0 con 0x7f9564069a80 2024-04-22T13:42:18.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.852+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f954c046020 con 0x7f9564069a80 2024-04-22T13:42:18.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.853+0000 7f9560ff9640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 0x7f95500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:18.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.853+0000 7f95637fe640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 0x7f95500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:18.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.853+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f954c0f21c0 con 0x7f9564069a80 2024-04-22T13:42:18.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.853+0000 7f95637fe640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 0x7f95500a6260 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f955c005cc0 tx=0x7f955c0082d0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:18.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:18.858+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f954c04d070 con 0x7f9564069a80 2024-04-22T13:42:19.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.072+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f956402d5c0 con 0x7f9564069a80 2024-04-22T13:42:19.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.073+0000 7f9560ff9640 1 -- 172.21.15.87:0/1325172545 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f954c0b8e20 con 0x7f9564069a80 2024-04-22T13:42:19.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:42:19.069 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:42:19.070 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:42:19.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 msgr2=0x7f95500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 0x7f95500a6260 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f955c005cc0 tx=0x7f955c0082d0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 msgr2=0x7f9564198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f954c006010 tx=0x7f954c004040 comp rx=0 tx=0).stop 2024-04-22T13:42:19.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 shutdown_connections 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f95500a3da0 0x7f95500a6260 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9564107fe0 0x7f956419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9564069a80 0x7f9564198bc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 --2- 172.21.15.87:0/1325172545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95640690e0 0x7f9564198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 >> 172.21.15.87:0/1325172545 conn(0x7f9564077e30 msgr2=0x7f95640ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 shutdown_connections 2024-04-22T13:42:19.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.075+0000 7f9569ffb640 1 -- 172.21.15.87:0/1325172545 wait complete. 2024-04-22T13:42:19.083 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id e83f2f35 --release --await' 2024-04-22T13:42:19.083 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id e83f2f35 --release --await 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- 172.21.15.87:0/467945619 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101090 msgr2=0x7f19e0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 --2- 172.21.15.87:0/467945619 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101090 0x7f19e0101470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009900 tx=0x7f19d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- 172.21.15.87:0/467945619 shutdown_connections 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 --2- 172.21.15.87:0/467945619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 0x7f19e010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 --2- 172.21.15.87:0/467945619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101a40 0x7f19e0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 --2- 172.21.15.87:0/467945619 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e0101090 0x7f19e0101470 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- 172.21.15.87:0/467945619 >> 172.21.15.87:0/467945619 conn(0x7f19e0077f50 msgr2=0x7f19e0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- 172.21.15.87:0/467945619 shutdown_connections 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- 172.21.15.87:0/467945619 wait complete. 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 Processor -- start 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.193+0000 7f19e56e4640 1 -- start start 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101090 0x7f19e0197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 0x7f19e0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01110d0 con 0x7f19e019bf30 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0110f50 con 0x7f19e0101090 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0111250 con 0x7f19e0106160 2024-04-22T13:42:19.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50694/0 (socket says 172.21.15.87:50694) 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 -- 172.21.15.87:0/984930364 learned_addr learned my addr 172.21.15.87:0/984930364 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19deffd640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101090 0x7f19e0197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19de7fc640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 0x7f19e0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 -- 172.21.15.87:0/984930364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 msgr2=0x7f19e0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 0x7f19e0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 -- 172.21.15.87:0/984930364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101090 msgr2=0x7f19e0197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101090 0x7f19e0197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 -- 172.21.15.87:0/984930364 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e019ca20 con 0x7f19e019bf30 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19df7fe640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f19d800cad0 tx=0x7f19d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19c7fff640 1 -- 172.21.15.87:0/984930364 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d800ae20 con 0x7f19e019bf30 2024-04-22T13:42:19.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e019cd10 con 0x7f19e019bf30 2024-04-22T13:42:19.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19c7fff640 1 -- 172.21.15.87:0/984930364 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d8005020 con 0x7f19e019bf30 2024-04-22T13:42:19.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19c7fff640 1 -- 172.21.15.87:0/984930364 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d800f6b0 con 0x7f19e019bf30 2024-04-22T13:42:19.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.194+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a4500 con 0x7f19e019bf30 2024-04-22T13:42:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.196+0000 7f19c7fff640 1 -- 172.21.15.87:0/984930364 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19d80051d0 con 0x7f19e019bf30 2024-04-22T13:42:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 Processor -- start 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 -- start start 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f79779a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f79779a8f0 0x55f7977a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f7976ff500 con 0x55f7977a5a20 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f7976a56a0 con 0x55f797799fd0 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19e83b2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f7976fb4c0 con 0x55f79779a8f0 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19c7fff640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 0x7f19c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50700/0 (socket says 172.21.15.87:50700) 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 -- 172.21.15.87:0/140384147 learned_addr learned my addr 172.21.15.87:0/140384147 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19df7fe640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 0x7f19c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19deffd640 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f79779a8f0 0x55f7977a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f79779a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19c7fff640 1 -- 172.21.15.87:0/984930364 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f19d80d0790 con 0x7f19e019bf30 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 -- 172.21.15.87:0/140384147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f79779a8f0 msgr2=0x55f7977a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f79779a8f0 0x55f7977a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 -- 172.21.15.87:0/140384147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 msgr2=0x55f79779a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f79779a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.197+0000 7f19de7fc640 1 -- 172.21.15.87:0/140384147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7976c45e0 con 0x55f7977a5a20 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19df7fe640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 0x7f19c80a61e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f19d8099140 tx=0x7f19d8087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19de7fc640 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f19d4014980 tx=0x7f19d400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54c1eb11bb8dd71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19c57fa640 1 -- 172.21.15.87:0/140384147 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d40142d0 con 0x55f7977a5a20 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19c57fa640 1 -- 172.21.15.87:0/140384147 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d40054d0 con 0x55f7977a5a20 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19c57fa640 1 -- 172.21.15.87:0/140384147 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d400f760 con 0x55f7977a5a20 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 -- 172.21.15.87:0/140384147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 msgr2=0x55f7977a7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f19d4014980 tx=0x7f19d400c4b0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 -- 172.21.15.87:0/140384147 shutdown_connections 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f7977a5a20 0x55f7977a7e10 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f79779a8f0 0x55f7977a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 --2- 172.21.15.87:0/140384147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f79779a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.198+0000 7f19e83b2740 1 -- 172.21.15.87:0/140384147 >> 172.21.15.87:0/140384147 conn(0x55f797794a40 msgr2=0x55f797794e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- 172.21.15.87:0/140384147 shutdown_connections 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- 172.21.15.87:0/140384147 wait complete. 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- start start 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 Processor -- start 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977a5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55f7977a8810 con 0x55f79779a8f0 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f7976b8a60 con 0x55f797799fd0 2024-04-22T13:42:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f7977a8990 con 0x55f7977a5a20 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19e83b2740 10 client.0 _ll_register_callbacks cb 0x55f797411110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50706/0 (socket says 172.21.15.87:50706) 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 -- 172.21.15.87:0/3918217084 learned_addr learned my addr 172.21.15.87:0/3918217084 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19de7fc640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19de7fc640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 -- 172.21.15.87:0/3918217084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 msgr2=0x55f7977a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 -- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 msgr2=0x55f7977a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.199+0000 7f19deffd640 1 -- 172.21.15.87:0/3918217084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f797411920 con 0x55f79779a8f0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19deffd640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009900 tx=0x7f19d002c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a0efbf2534c58b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0005030 con 0x55f79779a8f0 2024-04-22T13:42:19.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f19d00051d0 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d40054d0 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f19d00053d0 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 10 client.18771 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f7974ddd30 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d40054d0 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f19d00037b0 con 0x55f79779a8f0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 10 client.18771 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 10 client.18771 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 20 client.18771 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f7977b5ac0 0x55f7977b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 4 client.18771 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.200+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7976d2a10 con 0x55f7977b5ac0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.201+0000 7f19de7fc640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f7977b5ac0 0x55f7977b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.201+0000 7f19c2ff5640 10 client.18771 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.201+0000 7f19de7fc640 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f7977b5ac0 0x55f7977b7f20 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c35ae095f94f2e6 server_cookie=27516f0c895e2abe in_seq=0 out_seq=0 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19c2ff5640 1 -- 172.21.15.87:0/3918217084 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7976d2a10 con 0x55f7977b5ac0 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19c2ff5640 10 client.18771 handle_command_reply: tid=0 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 client.18771 shutdown 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 msgr2=0x55f7977a27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009900 tx=0x7f19d002c9f0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 shutdown_connections 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f7977b5ac0 0x55f7977b7f20 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a27d0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19c2ff5640 0 client.18771 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 --2- 172.21.15.87:0/3918217084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 >> 172.21.15.87:0/3918217084 conn(0x55f797794a40 msgr2=0x55f797796790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 shutdown_connections 2024-04-22T13:42:19.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 1 -- 172.21.15.87:0/3918217084 wait complete. 2024-04-22T13:42:19.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.202+0000 7f19e83b2740 20 client.18771 trim_cache size 0 max 16384 2024-04-22T13:42:19.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 Processor -- start 2024-04-22T13:42:19.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 -- start start 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977bd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01ae1f0 con 0x55f79779a8f0 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01abac0 con 0x55f797799fd0 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19e83b2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01a9930 con 0x55f7977a5a20 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50708/0 (socket says 172.21.15.87:50708) 2024-04-22T13:42:19.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.219+0000 7f19deffd640 1 -- 172.21.15.87:0/1352593331 learned_addr learned my addr 172.21.15.87:0/1352593331 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 -- 172.21.15.87:0/1352593331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 msgr2=0x55f7977bd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977bd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 -- 172.21.15.87:0/1352593331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 msgr2=0x55f7977b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19de7fc640 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 -- 172.21.15.87:0/1352593331 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f797796d40 con 0x55f79779a8f0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19deffd640 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f19d0003d70 tx=0x7f19d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245aef5b8ac5ab8a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19c4ff9640 1 -- 172.21.15.87:0/1352593331 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0005850 con 0x55f79779a8f0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19c4ff9640 1 -- 172.21.15.87:0/1352593331 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d0058070 con 0x55f79779a8f0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19c4ff9640 1 -- 172.21.15.87:0/1352593331 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0053840 con 0x55f79779a8f0 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 -- 172.21.15.87:0/1352593331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 msgr2=0x55f797796470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f19d0003d70 tx=0x7f19d002bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 -- 172.21.15.87:0/1352593331 shutdown_connections 2024-04-22T13:42:19.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7977a5a20 0x55f7977bd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f797796470 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 --2- 172.21.15.87:0/1352593331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f797799fd0 0x55f7977b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 -- 172.21.15.87:0/1352593331 >> 172.21.15.87:0/1352593331 conn(0x55f797794a40 msgr2=0x55f797793b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 -- 172.21.15.87:0/1352593331 shutdown_connections 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.220+0000 7f19e83b2740 1 -- 172.21.15.87:0/1352593331 wait complete. 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 -- start start 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 Processor -- start 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f797799fd0 0x55f7977a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7977a5a20 0x55f7977a9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01a9f80 con 0x55f79779a8f0 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01ab3f0 con 0x55f7977a5a20 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01aa650 con 0x55f797799fd0 2024-04-22T13:42:19.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19e83b2740 10 client.0 _ll_register_callbacks cb 0x7f19d002c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50716/0 (socket says 172.21.15.87:50716) 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 -- 172.21.15.87:0/815766700 learned_addr learned my addr 172.21.15.87:0/815766700 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19de7fc640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f797799fd0 0x55f7977a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 -- 172.21.15.87:0/815766700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f797799fd0 msgr2=0x55f7977a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f797799fd0 0x55f7977a8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 -- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7977a5a20 msgr2=0x55f7977a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7977a5a20 0x55f7977a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 -- 172.21.15.87:0/815766700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f79781c0b0 con 0x55f79779a8f0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19deffd640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f19d002ce10 tx=0x7f19d002c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d4911381b046b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.221+0000 7f19c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0003980 con 0x55f79779a8f0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f19d0034440 con 0x55f79779a8f0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d0058070 con 0x55f79779a8f0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f19d006a050 con 0x55f79779a8f0 2024-04-22T13:42:19.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 10 client.18777 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f79781d3f0 con 0x55f79779a8f0 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d0058070 con 0x55f79779a8f0 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f19d0053880 con 0x55f79779a8f0 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 10 client.18777 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 10 client.18777 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 20 client.18777 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f79781da70 0x55f79781fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 4 client.18777 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e83f2f35", "release": true, "await": true}] 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e83f2f35", "release": true, "await": true}) -- 0x55f79770cf20 con 0x55f79781da70 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.222+0000 7f19de7fc640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f79781da70 0x55f79781fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.223+0000 7f19de7fc640 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f79781da70 0x55f79781fe90 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6f6e9327cbe165 server_cookie=db003339e848ede5 in_seq=0 out_seq=0 2024-04-22T13:42:19.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.223+0000 7f19c2ff5640 10 client.18777 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:19.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19c2ff5640 1 -- 172.21.15.87:0/815766700 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55f79770cf20 con 0x55f79781da70 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19c2ff5640 10 client.18777 handle_command_reply: tid=0 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 client.18777 shutdown 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 msgr2=0x55f7977a9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f19d002ce10 tx=0x7f19d002c330 comp rx=0 tx=0).stop 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 shutdown_connections 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55f79781da70 0x55f79781fe90 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7977a5a20 0x55f7977a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19c2ff5640 0 client.18777 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55f79779a8f0 0x55f7977a9120 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 --2- 172.21.15.87:0/815766700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f797799fd0 0x55f7977a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.225+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 >> 172.21.15.87:0/815766700 conn(0x55f797794a40 msgr2=0x55f797793b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 shutdown_connections 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e83b2740 1 -- 172.21.15.87:0/815766700 wait complete. 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e83b2740 20 client.18777 trim_cache size 0 max 16384 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 40, 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "e83f2f35": { 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 40, 2024-04-22T13:42:19.227 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 82.9, 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:19.228 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:19.229 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:19.229 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:19.229 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:19.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 msgr2=0x7f19c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 0x7f19c80a61e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f19d8099140 tx=0x7f19d8087040 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 msgr2=0x7f19e019c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f19d800cad0 tx=0x7f19d800cf90 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 shutdown_connections 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f19c80a3d20 0x7f19c80a61e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19e019bf30 0x7f19e019c3e0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19e0106160 0x7f19e0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 --2- 172.21.15.87:0/984930364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19e0101090 0x7f19e0197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 >> 172.21.15.87:0/984930364 conn(0x7f19e0077f50 msgr2=0x7f19e00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 shutdown_connections 2024-04-22T13:42:19.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:19.226+0000 7f19e56e4640 1 -- 172.21.15.87:0/984930364 wait complete. 2024-04-22T13:42:19.233 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: e83f2f35, seconds elapsed: 23.9 2024-04-22T13:42:19.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:19 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1325172545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:19.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:19 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1325172545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:19.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:19 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1325172545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:20.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:20 smithi120 ceph-mon[31101]: pgmap v328: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:20.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:20 smithi132 ceph-mon[31531]: pgmap v328: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:20.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:20 smithi087 ceph-mon[27999]: pgmap v328: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:22.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:22 smithi120 ceph-mon[31101]: pgmap v329: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T13:42:22.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:22 smithi132 ceph-mon[31531]: pgmap v329: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T13:42:22.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:22 smithi087 ceph-mon[27999]: pgmap v329: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T13:42:24.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:24 smithi120 ceph-mon[31101]: pgmap v330: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:42:24.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:24 smithi132 ceph-mon[31531]: pgmap v330: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:42:24.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:24 smithi087 ceph-mon[27999]: pgmap v330: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T13:42:26.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:26 smithi120 ceph-mon[31101]: pgmap v331: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:42:26.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:26 smithi132 ceph-mon[31531]: pgmap v331: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:42:26.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:26 smithi087 ceph-mon[27999]: pgmap v331: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 349 B/s wr, 6 op/s 2024-04-22T13:42:28.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:28 smithi120 ceph-mon[31101]: pgmap v332: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:42:28.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:28 smithi132 ceph-mon[31531]: pgmap v332: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:42:28.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:28 smithi087 ceph-mon[27999]: pgmap v332: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:42:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 -- 172.21.15.87:0/3066795444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 msgr2=0x7f0cd0110d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/3066795444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd0110d10 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009900 tx=0x7f0cc802bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 -- 172.21.15.87:0/3066795444 shutdown_connections 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/3066795444 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd0110d10 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/3066795444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/3066795444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd0101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 -- 172.21.15.87:0/3066795444 >> 172.21.15.87:0/3066795444 conn(0x7f0cd0077fa0 msgr2=0x7f0cd00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 -- 172.21.15.87:0/3066795444 shutdown_connections 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 -- 172.21.15.87:0/3066795444 wait complete. 2024-04-22T13:42:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.282+0000 7f0cd58a5640 1 Processor -- start 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 -- start start 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd0114430 con 0x7f0cd010e920 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01142b0 con 0x7f0cd0100d70 2024-04-22T13:42:30.281 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cd58a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01145b0 con 0x7f0cd0101690 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37092/0 (socket says 172.21.15.87:37092) 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 -- 172.21.15.87:0/2702009966 learned_addr learned my addr 172.21.15.87:0/2702009966 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cceffd640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 -- 172.21.15.87:0/2702009966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 msgr2=0x7f0cd01a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd01a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 -- 172.21.15.87:0/2702009966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 msgr2=0x7f0cd01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0ccf7fe640 1 -- 172.21.15.87:0/2702009966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd019af60 con 0x7f0cd010e920 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.283+0000 7f0cceffd640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0ccf7fe640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f0cc802c170 tx=0x7f0cc80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0cd48a3640 1 -- 172.21.15.87:0/2702009966 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8043020 con 0x7f0cd010e920 2024-04-22T13:42:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0cd48a3640 1 -- 172.21.15.87:0/2702009966 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc802cbb0 con 0x7f0cd010e920 2024-04-22T13:42:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd019b1f0 con 0x7f0cd010e920 2024-04-22T13:42:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0cd48a3640 1 -- 172.21.15.87:0/2702009966 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8003ab0 con 0x7f0cd010e920 2024-04-22T13:42:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.284+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd01a7470 con 0x7f0cd010e920 2024-04-22T13:42:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.286+0000 7f0cd48a3640 1 -- 172.21.15.87:0/2702009966 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0cc8046020 con 0x7f0cd010e920 2024-04-22T13:42:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.286+0000 7f0cd8575740 1 Processor -- start 2024-04-22T13:42:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.286+0000 7f0cd8575740 1 -- start start 2024-04-22T13:42:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e338d0 0x560d81e32620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 0x560d81e3e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd48a3640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 0x7f0cb80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560d81d978d0 con 0x560d81e32b60 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560d81d34e80 con 0x560d81e3c0e0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd8575740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560d81d93890 con 0x560d81e338d0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0ccf7fe640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 0x7f0cb80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37098/0 (socket says 172.21.15.87:37098) 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 -- 172.21.15.87:0/1987052510 learned_addr learned my addr 172.21.15.87:0/1987052510 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 0x560d81e3e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e338d0 0x560d81e32620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cd48a3640 1 -- 172.21.15.87:0/2702009966 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f0cc80f2a00 con 0x7f0cd010e920 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 -- 172.21.15.87:0/1987052510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e338d0 msgr2=0x560d81e32620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e338d0 0x560d81e32620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 -- 172.21.15.87:0/1987052510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 msgr2=0x560d81e3e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 0x560d81e3e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 -- 172.21.15.87:0/1987052510 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d81d56320 con 0x560d81e32b60 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 0x560d81e3e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00104a0 tx=0x7f0cc0010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de31d807bbffb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.287+0000 7f0ccf7fe640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 0x7f0cb80a6280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80bea80 tx=0x7f0cc8090040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cb5ffb640 1 -- 172.21.15.87:0/1987052510 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0017070 con 0x560d81e32b60 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cb5ffb640 1 -- 172.21.15.87:0/1987052510 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc0011bb0 con 0x560d81e32b60 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cb5ffb640 1 -- 172.21.15.87:0/1987052510 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000ecd0 con 0x560d81e32b60 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 -- 172.21.15.87:0/1987052510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 msgr2=0x560d81e32fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00104a0 tx=0x7f0cc0010960 comp rx=0 tx=0).stop 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 -- 172.21.15.87:0/1987052510 shutdown_connections 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e3c0e0 0x560d81e3e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e32b60 0x560d81e32fe0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1987052510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e338d0 0x560d81e32620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 -- 172.21.15.87:0/1987052510 >> 172.21.15.87:0/1987052510 conn(0x560d81df9c90 msgr2=0x560d81e2d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 -- 172.21.15.87:0/1987052510 shutdown_connections 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.288+0000 7f0cd8575740 1 -- 172.21.15.87:0/1987052510 wait complete. 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 -- start start 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 Processor -- start 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e32b60 0x560d81e3ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 0x560d81e38df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x560d81e3ee10 con 0x560d81e3c0e0 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560d81d1feb0 con 0x560d81e338d0 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560d81e3ef90 con 0x560d81e32b60 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cd8575740 10 client.0 _ll_register_callbacks cb 0x560d81d4cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cce7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cce7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37104/0 (socket says 172.21.15.87:37104) 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cce7fc640 1 -- 172.21.15.87:0/3228457237 learned_addr learned my addr 172.21.15.87:0/3228457237 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cceffd640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 0x560d81e38df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.289+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e32b60 0x560d81e3ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 -- 172.21.15.87:0/3228457237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e32b60 msgr2=0x560d81e3ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e32b60 0x560d81e3ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 -- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 msgr2=0x560d81e38df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 0x560d81e38df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 -- 172.21.15.87:0/3228457237 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d81aaa210 con 0x560d81e3c0e0 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cceffd640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 0x560d81e38df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f0cd019bae0 tx=0x7f0cc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef3df415dc0d4ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0caeffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40119a0 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f0cc4011b40 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc0011bb0 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0cc4026050 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cd8575740 10 client.18789 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560d81b76620 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.290+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc0011bb0 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0cc400f3f0 con 0x560d81e3c0e0 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 10 client.18789 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 10 client.18789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 20 client.18789 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81e4c130 0x560d81e4e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 4 client.18789 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d81dab880 con 0x560d81e4c130 2024-04-22T13:42:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cceffd640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81e4c130 0x560d81e4e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0caeffd640 10 client.18789 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.291+0000 7f0cceffd640 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81e4c130 0x560d81e4e520 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c795b38bcc29a609 server_cookie=ad1d2cd6f863b9e9 in_seq=0 out_seq=0 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0caeffd640 1 -- 172.21.15.87:0/3228457237 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d81dab880 con 0x560d81e4c130 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0caeffd640 10 client.18789 handle_command_reply: tid=0 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 client.18789 shutdown 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 msgr2=0x560d81e39370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f0cd019bae0 tx=0x7f0cc40046c0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 shutdown_connections 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81e4c130 0x560d81e4e520 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e3c0e0 0x560d81e39370 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e338d0 0x560d81e38df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0caeffd640 0 client.18789 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 --2- 172.21.15.87:0/3228457237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e32b60 0x560d81e3ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.292+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 >> 172.21.15.87:0/3228457237 conn(0x560d81df9c90 msgr2=0x560d81e2f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.293+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 shutdown_connections 2024-04-22T13:42:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.293+0000 7f0cd8575740 1 -- 172.21.15.87:0/3228457237 wait complete. 2024-04-22T13:42:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.293+0000 7f0cd8575740 20 client.18789 trim_cache size 0 max 16384 2024-04-22T13:42:30.305 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 Processor -- start 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 -- start start 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e47d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e3ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01b25a0 con 0x560d81e338d0 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01afe70 con 0x560d81e32b60 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cd8575740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01adce0 con 0x560d81e3c0e0 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37110/0 (socket says 172.21.15.87:37110) 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 -- 172.21.15.87:0/1002948936 learned_addr learned my addr 172.21.15.87:0/1002948936 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e47d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 -- 172.21.15.87:0/1002948936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 msgr2=0x560d81e3ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e3ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 -- 172.21.15.87:0/1002948936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 msgr2=0x560d81e47d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e47d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.308+0000 7f0cceffd640 1 -- 172.21.15.87:0/1002948936 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d81e31e30 con 0x560d81e338d0 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cceffd640 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009600 tx=0x7f0cc0009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f3c55260709a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cb57fa640 1 -- 172.21.15.87:0/1002948936 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0017040 con 0x560d81e338d0 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cb57fa640 1 -- 172.21.15.87:0/1002948936 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc0029070 con 0x560d81e338d0 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cb57fa640 1 -- 172.21.15.87:0/1002948936 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0003bd0 con 0x560d81e338d0 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 -- 172.21.15.87:0/1002948936 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 msgr2=0x560d81e3b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009600 tx=0x7f0cc0009590 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 -- 172.21.15.87:0/1002948936 shutdown_connections 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e3ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e3b520 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 --2- 172.21.15.87:0/1002948936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e47d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 -- 172.21.15.87:0/1002948936 >> 172.21.15.87:0/1002948936 conn(0x560d81df9c90 msgr2=0x560d81e37700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 -- 172.21.15.87:0/1002948936 shutdown_connections 2024-04-22T13:42:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.309+0000 7f0cd8575740 1 -- 172.21.15.87:0/1002948936 wait complete. 2024-04-22T13:42:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 -- start start 2024-04-22T13:42:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 Processor -- start 2024-04-22T13:42:30.310 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e446b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e36810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01ae330 con 0x560d81e338d0 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01af7a0 con 0x560d81e32b60 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01aea00 con 0x560d81e3c0e0 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cd8575740 10 client.0 _ll_register_callbacks cb 0x7f0cc40136a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cceffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:37114/0 (socket says 172.21.15.87:37114) 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cceffd640 1 -- 172.21.15.87:0/2525098114 learned_addr learned my addr 172.21.15.87:0/2525098114 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.310+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 -- 172.21.15.87:0/2525098114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 msgr2=0x560d81e36810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e36810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 -- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 msgr2=0x560d81e446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 -- 172.21.15.87:0/2525098114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d81e3cc60 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cceffd640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0006010 tx=0x7f0cc000e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca18dd2d24853ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0017070 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f0cc0003b70 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc0029070 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0cc0003d70 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 10 client.18795 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560d81eb2890 con 0x560d81e338d0 2024-04-22T13:42:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc0029070 con 0x560d81e338d0 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0cc003c080 con 0x560d81e338d0 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 10 client.18795 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 10 client.18795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 20 client.18795 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81eb2de0 0x560d81eb5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 4 client.18795 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.311+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d81d373d0 con 0x560d81eb2de0 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.312+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81eb2de0 0x560d81eb5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.312+0000 7f0caeffd640 10 client.18795 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.312+0000 7f0cce7fc640 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81eb2de0 0x560d81eb5200 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d70ac1f4aa5fdce4 server_cookie=8b3974bea55b94eb in_seq=0 out_seq=0 2024-04-22T13:42:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0caeffd640 1 -- 172.21.15.87:0/2525098114 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d81d373d0 con 0x560d81eb2de0 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0caeffd640 10 client.18795 handle_command_reply: tid=0 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 client.18795 shutdown 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 msgr2=0x560d81e36200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0006010 tx=0x7f0cc000e5a0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 shutdown_connections 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x560d81eb2de0 0x560d81eb5200 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560d81e3c0e0 0x560d81e36810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x560d81e338d0 0x560d81e36200 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0caeffd640 0 client.18795 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 --2- 172.21.15.87:0/2525098114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d81e32b60 0x560d81e446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 >> 172.21.15.87:0/2525098114 conn(0x560d81df9c90 msgr2=0x560d81e2c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 shutdown_connections 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.317+0000 7f0cd8575740 1 -- 172.21.15.87:0/2525098114 wait complete. 2024-04-22T13:42:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd8575740 20 client.18795 trim_cache size 0 max 16384 2024-04-22T13:42:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:42:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:42:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:42:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 msgr2=0x7f0cb80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 0x7f0cb80a6280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80bea80 tx=0x7f0cc8090040 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 msgr2=0x7f0cd019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f0cc802c170 tx=0x7f0cc80045e0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 shutdown_connections 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0cb80a3dc0 0x7f0cb80a6280 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0cd010e920 0x7f0cd019a790 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0cd0101690 0x7f0cd01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 --2- 172.21.15.87:0/2702009966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cd0100d70 0x7f0cd01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 >> 172.21.15.87:0/2702009966 conn(0x7f0cd0077fa0 msgr2=0x7f0cd010fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 shutdown_connections 2024-04-22T13:42:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:30.318+0000 7f0cd58a5640 1 -- 172.21.15.87:0/2702009966 wait complete. 2024-04-22T13:42:30.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:30 smithi120 ceph-mon[31101]: pgmap v333: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:30.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:30 smithi132 ceph-mon[31531]: pgmap v333: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:30.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:30 smithi087 ceph-mon[27999]: pgmap v333: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:32.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:32 smithi120 ceph-mon[31101]: pgmap v334: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:42:32.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:32 smithi132 ceph-mon[31531]: pgmap v334: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:42:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:32 smithi087 ceph-mon[27999]: pgmap v334: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:42:34.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:34 smithi120 ceph-mon[31101]: pgmap v335: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:34.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:34 smithi132 ceph-mon[31531]: pgmap v335: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:34 smithi087 ceph-mon[27999]: pgmap v335: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:36.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:36 smithi120 ceph-mon[31101]: pgmap v336: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:36.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:36 smithi132 ceph-mon[31531]: pgmap v336: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:36.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:36 smithi087 ceph-mon[27999]: pgmap v336: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:38.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:38 smithi120 ceph-mon[31101]: pgmap v337: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:38.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:38 smithi132 ceph-mon[31531]: pgmap v337: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:38.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:38 smithi087 ceph-mon[27999]: pgmap v337: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:40.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:40 smithi120 ceph-mon[31101]: pgmap v338: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:40.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:40 smithi132 ceph-mon[31531]: pgmap v338: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:40.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:40 smithi087 ceph-mon[27999]: pgmap v338: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:41.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:42:41.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:42:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 -- 172.21.15.87:0/1692518721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8100d30 msgr2=0x7fb2c8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/1692518721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8100d30 0x7fb2c8101110 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009900 tx=0x7fb2b002bc50 comp rx=0 tx=0).stop 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 -- 172.21.15.87:0/1692518721 shutdown_connections 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/1692518721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c810e8e0 0x7fb2c8110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/1692518721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c8101650 0x7fb2c810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/1692518721 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8100d30 0x7fb2c8101110 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 -- 172.21.15.87:0/1692518721 >> 172.21.15.87:0/1692518721 conn(0x7fb2c80771d0 msgr2=0x7fb2c80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 -- 172.21.15.87:0/1692518721 shutdown_connections 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.253+0000 7fb2ccbce640 1 -- 172.21.15.87:0/1692518721 wait complete. 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 Processor -- start 2024-04-22T13:42:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 -- start start 2024-04-22T13:42:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c8100d30 0x7fb2c81a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 0x7fb2c819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8114440 con 0x7fb2c8101650 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81142c0 con 0x7fb2c810e8e0 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2ccbce640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81145c0 con 0x7fb2c8100d30 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43486/0 (socket says 172.21.15.87:43486) 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2032115099 learned_addr learned my addr 172.21.15.87:0/2032115099 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 0x7fb2c819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.254+0000 7fb2c659c640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c8100d30 0x7fb2c81a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2032115099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c8100d30 msgr2=0x7fb2c81a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c8100d30 0x7fb2c81a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2032115099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 msgr2=0x7fb2c819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 0x7fb2c819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2032115099 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c819b010 con 0x7fb2c8101650 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 0x7fb2c819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800d950 tx=0x7fb2b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2bf7fe640 1 -- 172.21.15.87:0/2032115099 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8017070 con 0x7fb2c8101650 2024-04-22T13:42:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c819b300 con 0x7fb2c8101650 2024-04-22T13:42:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2bf7fe640 1 -- 172.21.15.87:0/2032115099 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b8009c30 con 0x7fb2c8101650 2024-04-22T13:42:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2bf7fe640 1 -- 172.21.15.87:0/2032115099 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800f620 con 0x7fb2c8101650 2024-04-22T13:42:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.255+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81a7520 con 0x7fb2c8101650 2024-04-22T13:42:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.257+0000 7fb2bf7fe640 1 -- 172.21.15.87:0/2032115099 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb2b8010460 con 0x7fb2c8101650 2024-04-22T13:42:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 Processor -- start 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 -- start start 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35ba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2bf7fe640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a2f351f1e0 con 0x55a2f35c3a80 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a2f34bc390 con 0x55a2f35ba580 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a2f351b1a0 con 0x55a2f35baea0 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43492/0 (socket says 172.21.15.87:43492) 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/3045149730 learned_addr learned my addr 172.21.15.87:0/3045149730 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2bf7fe640 1 -- 172.21.15.87:0/2032115099 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fb2b80d04b0 con 0x7fb2c8101650 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.258+0000 7fb2c659c640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/3045149730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 msgr2=0x55a2f35c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/3045149730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 msgr2=0x55a2f35ba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35ba960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/3045149730 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f32fb7d0 con 0x55a2f35c3a80 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35ba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80988f0 tx=0x7fb2b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b48268daed5b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009440 tx=0x7fb2c0009600 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2bcff9640 1 -- 172.21.15.87:0/3045149730 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8020040 con 0x55a2f35c3a80 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2bcff9640 1 -- 172.21.15.87:0/3045149730 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b809cb90 con 0x55a2f35c3a80 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.259+0000 7fb2bcff9640 1 -- 172.21.15.87:0/3045149730 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80a5100 con 0x55a2f35c3a80 2024-04-22T13:42:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/3045149730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 msgr2=0x55a2f35c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80988f0 tx=0x7fb2b8005880 comp rx=0 tx=0).stop 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/3045149730 shutdown_connections 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c5e70 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/3045149730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35ba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/3045149730 >> 172.21.15.87:0/3045149730 conn(0x55a2f35b4cb0 msgr2=0x55a2f35b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/3045149730 shutdown_connections 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/3045149730 wait complete. 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- start start 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 Processor -- start 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a2f35c6810 con 0x55a2f35baea0 2024-04-22T13:42:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a2f34d82f0 con 0x55a2f35c3a80 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.260+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a2f35c6990 con 0x55a2f35ba580 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2cf7c1740 10 client.0 _ll_register_callbacks cb 0x55a2f34e5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43498/0 (socket says 172.21.15.87:43498) 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2065685096 learned_addr learned my addr 172.21.15.87:0/2065685096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c659c640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2065685096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 msgr2=0x55a2f35c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35c0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35c3a80 msgr2=0x55a2f35c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 -- 172.21.15.87:0/2065685096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f32313f0 con 0x55a2f35baea0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80ce240 tx=0x7fb2b8066f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45454ccc7c93704 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2adffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8020070 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fb2b80cdec0 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2b809cb90 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb2b80a5100 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2cf7c1740 10 client.18807 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a2f31f10f0 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.261+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b809cb90 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb2b80a4c70 con 0x55a2f35baea0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 10 client.18807 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 10 client.18807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 20 client.18807 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f35d38c0 0x55a2f35d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 4 client.18807 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2f352cc00 con 0x55a2f35d38c0 2024-04-22T13:42:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2c659c640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f35d38c0 0x55a2f35d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2adffb640 10 client.18807 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.262+0000 7fb2c659c640 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f35d38c0 0x55a2f35d5cb0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c95d7cff0c137b1 server_cookie=be0730249bb9a9cf in_seq=0 out_seq=0 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2adffb640 1 -- 172.21.15.87:0/2065685096 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2f352cc00 con 0x55a2f35d38c0 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2adffb640 10 client.18807 handle_command_reply: tid=0 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 client.18807 shutdown 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 msgr2=0x55a2f35c1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80ce240 tx=0x7fb2b8066f30 comp rx=0 tx=0).stop 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 shutdown_connections 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f35d38c0 0x55a2f35d5cb0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35baea0 0x55a2f35c1000 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/2065685096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35c0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2adffb640 0 client.18807 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 >> 172.21.15.87:0/2065685096 conn(0x55a2f35b4cb0 msgr2=0x55a2f35b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 shutdown_connections 2024-04-22T13:42:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.263+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/2065685096 wait complete. 2024-04-22T13:42:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.264+0000 7fb2cf7c1740 20 client.18807 trim_cache size 0 max 16384 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 Processor -- start 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 -- start start 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35baea0 0x55a2f35cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81b2610 con 0x55a2f35c3a80 2024-04-22T13:42:41.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81afee0 con 0x55a2f35baea0 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81add50 con 0x55a2f35ba580 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43508/0 (socket says 172.21.15.87:43508) 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/212540663 learned_addr learned my addr 172.21.15.87:0/212540663 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c659c640 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/212540663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 msgr2=0x55a2f35bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35bfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35baea0 0x55a2f35cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/212540663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35baea0 msgr2=0x55a2f35cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35baea0 0x55a2f35cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/212540663 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f35c5e70 con 0x55a2f35c3a80 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000b010 tx=0x7fb2c00093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60638941cc03dc8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.281+0000 7fb2affff640 1 -- 172.21.15.87:0/212540663 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0029070 con 0x55a2f35c3a80 2024-04-22T13:42:41.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2affff640 1 -- 172.21.15.87:0/212540663 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0011d60 con 0x55a2f35c3a80 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2affff640 1 -- 172.21.15.87:0/212540663 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0024650 con 0x55a2f35c3a80 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/212540663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 msgr2=0x55a2f35c1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000b010 tx=0x7fb2c00093d0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/212540663 shutdown_connections 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c1360 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35baea0 0x55a2f35cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/212540663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35ba580 0x55a2f35bfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/212540663 >> 172.21.15.87:0/212540663 conn(0x55a2f35b4cb0 msgr2=0x55a2f35b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/212540663 shutdown_connections 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/212540663 wait complete. 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 -- start start 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.282+0000 7fb2cf7c1740 1 Processor -- start 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81ae3a0 con 0x55a2f35c3a80 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81af810 con 0x55a2f35ba580 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c81aea70 con 0x55a2f35baea0 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 10 client.0 _ll_register_callbacks cb 0x7fb2b804bf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43512/0 (socket says 172.21.15.87:43512) 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/789488796 learned_addr learned my addr 172.21.15.87:0/789488796 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c5d9b640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c659c640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/789488796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 msgr2=0x55a2f35c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 msgr2=0x55a2f35c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 -- 172.21.15.87:0/789488796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f3638d60 con 0x55a2f35c3a80 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c659c640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2c6d9d640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0018e40 tx=0x7fb2c0025630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5a148cf490744b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2adffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0029030 con 0x55a2f35c3a80 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fb2c00259b0 con 0x55a2f35c3a80 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c0011d60 con 0x55a2f35c3a80 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb2c0025bb0 con 0x55a2f35c3a80 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 10 client.18813 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.283+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a2f363a070 con 0x55a2f35c3a80 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0011d60 con 0x55a2f35c3a80 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb2c0023070 con 0x55a2f35c3a80 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 10 client.18813 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 10 client.18813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 20 client.18813 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f363a750 0x55a2f363cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 4 client.18813 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2f3526a70 con 0x55a2f363a750 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.284+0000 7fb2c659c640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f363a750 0x55a2f363cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2c659c640 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f363a750 0x55a2f363cb70 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f518461944e71d6a server_cookie=c55871bd4dc7f28c in_seq=0 out_seq=0 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2adffb640 10 client.18813 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2adffb640 1 -- 172.21.15.87:0/789488796 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a2f3526a70 con 0x55a2f363a750 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2adffb640 10 client.18813 handle_command_reply: tid=0 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 client.18813 shutdown 2024-04-22T13:42:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 msgr2=0x55a2f35c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0018e40 tx=0x7fb2c0025630 comp rx=0 tx=0).stop 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 shutdown_connections 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a2f363a750 0x55a2f363cb70 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a2f35c3a80 0x55a2f35c8320 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2f35baea0 0x55a2f35c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2adffb640 0 client.18813 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 --2- 172.21.15.87:0/789488796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35ba580 0x55a2f35c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 >> 172.21.15.87:0/789488796 conn(0x55a2f35b4cb0 msgr2=0x55a2f35b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 shutdown_connections 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 1 -- 172.21.15.87:0/789488796 wait complete. 2024-04-22T13:42:41.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2cf7c1740 20 client.18813 trim_cache size 0 max 16384 2024-04-22T13:42:41.289 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:41.289 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:42:41.289 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:42:41.289 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 msgr2=0x7fb2b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009440 tx=0x7fb2c0009600 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 msgr2=0x7fb2c81a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.285+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800d950 tx=0x7fb2b800de10 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 shutdown_connections 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb2b40a3d20 0x7fb2b40a61e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2c810e8e0 0x7fb2c819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2c8101650 0x7fb2c81a0c10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 --2- 172.21.15.87:0/2032115099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2c8100d30 0x7fb2c81a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 >> 172.21.15.87:0/2032115099 conn(0x7fb2c80771d0 msgr2=0x7fb2c810ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 shutdown_connections 2024-04-22T13:42:41.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.286+0000 7fb2ccbce640 1 -- 172.21.15.87:0/2032115099 wait complete. 2024-04-22T13:42:41.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:42:41.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:42:41.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 -- 172.21.15.87:0/1229630628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d10101b20 msgr2=0x7f9d1010dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 --2- 172.21.15.87:0/1229630628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d10101b20 0x7f9d1010dd00 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009cd0 tx=0x7f9d0802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 -- 172.21.15.87:0/1229630628 shutdown_connections 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 --2- 172.21.15.87:0/1229630628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d10101b20 0x7f9d1010dd00 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 --2- 172.21.15.87:0/1229630628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d101011a0 0x7f9d101015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 --2- 172.21.15.87:0/1229630628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10105ed0 0x7f9d101062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 -- 172.21.15.87:0/1229630628 >> 172.21.15.87:0/1229630628 conn(0x7f9d10077e30 msgr2=0x7f9d10078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 -- 172.21.15.87:0/1229630628 shutdown_connections 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.395+0000 7f9d182a5640 1 -- 172.21.15.87:0/1229630628 wait complete. 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 Processor -- start 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 -- start start 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 0x7f9d10198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d10105ed0 0x7f9d1019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9d101121f0 con 0x7f9d101011a0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10112070 con 0x7f9d10105ed0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d182a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10112370 con 0x7f9d10101b20 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43518/0 (socket says 172.21.15.87:43518) 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 -- 172.21.15.87:0/2255659158 learned_addr learned my addr 172.21.15.87:0/2255659158 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16842640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d10105ed0 0x7f9d1019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d15840640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 0x7f9d10198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 -- 172.21.15.87:0/2255659158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 msgr2=0x7f9d10198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 0x7f9d10198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 -- 172.21.15.87:0/2255659158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d10105ed0 msgr2=0x7f9d1019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d10105ed0 0x7f9d1019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d16041640 1 -- 172.21.15.87:0/2255659158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1019d6a0 con 0x7f9d101011a0 2024-04-22T13:42:41.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.396+0000 7f9d15840640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 0x7f9d10198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d16041640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00d770 tx=0x7f9d0c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c017070 con 0x7f9d101011a0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1019d990 con 0x7f9d101011a0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d0c0054d0 con 0x7f9d101011a0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c004c30 con 0x7f9d101011a0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.397+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d101a5220 con 0x7f9d101011a0 2024-04-22T13:42:41.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.398+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d10069230 con 0x7f9d101011a0 2024-04-22T13:42:41.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.399+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d0c004e10 con 0x7f9d101011a0 2024-04-22T13:42:41.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.399+0000 7f9d077fe640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.400+0000 7f9d15840640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.400+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f9d0c0d0340 con 0x7f9d101011a0 2024-04-22T13:42:41.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.400+0000 7f9d15840640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9d10100d70 tx=0x7f9d00005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.404+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d0c09ca20 con 0x7f9d101011a0 2024-04-22T13:42:41.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.618+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d10199a00 con 0x7f9d101011a0 2024-04-22T13:42:41.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.619+0000 7f9d077fe640 1 -- 172.21.15.87:0/2255659158 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f9d0c09c320 con 0x7f9d101011a0 2024-04-22T13:42:41.615 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:42:41.615 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:42:41.616 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:42:41.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 msgr2=0x7f9cfc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9d10100d70 tx=0x7f9d00005f90 comp rx=0 tx=0).stop 2024-04-22T13:42:41.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 msgr2=0x7f9d101986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00d770 tx=0x7f9d0c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:42:41.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 shutdown_connections 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f9cfc0a3d20 0x7f9cfc0a61e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d10105ed0 0x7f9d1019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d10101b20 0x7f9d10198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 --2- 172.21.15.87:0/2255659158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d101011a0 0x7f9d101986b0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 >> 172.21.15.87:0/2255659158 conn(0x7f9d10077e30 msgr2=0x7f9d100fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 shutdown_connections 2024-04-22T13:42:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.621+0000 7f9d182a5640 1 -- 172.21.15.87:0/2255659158 wait complete. 2024-04-22T13:42:41.628 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 -- 172.21.15.87:0/951543980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 msgr2=0x7fc044106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 --2- 172.21.15.87:0/951543980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc044106200 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc034009930 tx=0x7fc03402bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 -- 172.21.15.87:0/951543980 shutdown_connections 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 --2- 172.21.15.87:0/951543980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 0x7fc04410dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 --2- 172.21.15.87:0/951543980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 0x7fc044101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 --2- 172.21.15.87:0/951543980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc044106200 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 -- 172.21.15.87:0/951543980 >> 172.21.15.87:0/951543980 conn(0x7fc0440760e0 msgr2=0x7fc0440764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 -- 172.21.15.87:0/951543980 shutdown_connections 2024-04-22T13:42:41.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 -- 172.21.15.87:0/951543980 wait complete. 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.735+0000 7fc048806640 1 Processor -- start 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 -- start start 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 0x7fc044197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 0x7fc044197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc044111100 con 0x7fc044105e20 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc044110f80 con 0x7fc044102350 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc048806640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc044111280 con 0x7fc044101990 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43534/0 (socket says 172.21.15.87:43534) 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 -- 172.21.15.87:0/371746670 learned_addr learned my addr 172.21.15.87:0/371746670 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc04259c640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 0x7fc044197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc041d9b640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 0x7fc044197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 -- 172.21.15.87:0/371746670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 msgr2=0x7fc044197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 0x7fc044197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 -- 172.21.15.87:0/371746670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 msgr2=0x7fc044197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 0x7fc044197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 -- 172.21.15.87:0/371746670 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04419c630 con 0x7fc044105e20 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc042d9d640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00b970 tx=0x7fc03c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.736+0000 7fc0337fe640 1 -- 172.21.15.87:0/371746670 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c019070 con 0x7fc044105e20 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.737+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc04419c920 con 0x7fc044105e20 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.737+0000 7fc0337fe640 1 -- 172.21.15.87:0/371746670 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc03c014440 con 0x7fc044105e20 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.737+0000 7fc0337fe640 1 -- 172.21.15.87:0/371746670 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0075d0 con 0x7fc044105e20 2024-04-22T13:42:41.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.737+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0441a41b0 con 0x7fc044105e20 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc0337fe640 1 -- 172.21.15.87:0/371746670 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc03c01c070 con 0x7fc044105e20 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 Processor -- start 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 -- start start 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ed0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bea2e251f0 con 0x55bea2ec00e0 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bea2dc24b0 con 0x55bea2ec0a00 2024-04-22T13:42:41.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bea2e211b0 con 0x55bea2ecdf70 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc0337fe640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 0x7fc02c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc041d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc041d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43544/0 (socket says 172.21.15.87:43544) 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.739+0000 7fc041d9b640 1 -- 172.21.15.87:0/2670536742 learned_addr learned my addr 172.21.15.87:0/2670536742 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc042d9d640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 0x7fc02c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc04259c640 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 -- 172.21.15.87:0/2670536742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 msgr2=0x55bea2ed0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ed0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 -- 172.21.15.87:0/2670536742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 msgr2=0x55bea2ecda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc0337fe640 1 -- 172.21.15.87:0/371746670 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fc03c0cfc60 con 0x7fc044105e20 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 -- 172.21.15.87:0/2670536742 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea2db41b0 con 0x55bea2ec00e0 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc041d9b640 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc03800bdc0 tx=0x7fc0380127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfba14774446769c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc042d9d640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 0x7fc02c0a61e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0987d0 tx=0x7fc03c06d840 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc030ff9640 1 -- 172.21.15.87:0/2670536742 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038005020 con 0x55bea2ec00e0 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc030ff9640 1 -- 172.21.15.87:0/2670536742 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc038016070 con 0x55bea2ec00e0 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.740+0000 7fc030ff9640 1 -- 172.21.15.87:0/2670536742 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038004e20 con 0x55bea2ec00e0 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- 172.21.15.87:0/2670536742 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 msgr2=0x55bea2ec04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc03800bdc0 tx=0x7fc0380127c0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- 172.21.15.87:0/2670536742 shutdown_connections 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ed0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/2670536742 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec00e0 0x55bea2ec04c0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- 172.21.15.87:0/2670536742 >> 172.21.15.87:0/2670536742 conn(0x55bea2ebaba0 msgr2=0x55bea2ebaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- 172.21.15.87:0/2670536742 shutdown_connections 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- 172.21.15.87:0/2670536742 wait complete. 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 -- start start 2024-04-22T13:42:41.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 Processor -- start 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.741+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ecdf70 0x55bea2ecb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bea2ed0d60 con 0x55bea2ec0a00 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bea2dde500 con 0x55bea2ecdf70 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bea2ed0ee0 con 0x55bea2ec00e0 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 10 client.0 _ll_register_callbacks cb 0x55bea2dacb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43546/0 (socket says 172.21.15.87:43546) 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 -- 172.21.15.87:0/1753487201 learned_addr learned my addr 172.21.15.87:0/1753487201 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc041d9b640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ecdf70 0x55bea2ecb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc041d9b640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 -- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 msgr2=0x55bea2ecd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 -- 172.21.15.87:0/1753487201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ecdf70 msgr2=0x55bea2ecb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ecdf70 0x55bea2ecb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 -- 172.21.15.87:0/1753487201 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea2b373f0 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc041d9b640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04259c640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc034009900 tx=0x7fc034004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96df09d83e850862 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc029ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034005d50 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc038016070 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fc034005030 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc034005230 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 10 client.18831 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.742+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bea2c017d0 con 0x55bea2ec0a00 2024-04-22T13:42:41.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc038016070 con 0x55bea2ec0a00 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc034003680 con 0x55bea2ec0a00 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 10 client.18831 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 10 client.18831 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 20 client.18831 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2ede000 0x55bea2ee03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 4 client.18831 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bea2dc4930 con 0x55bea2ede000 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc041d9b640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2ede000 0x55bea2ee03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc041d9b640 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2ede000 0x55bea2ee03f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ada370a153e3f27c server_cookie=c6df11eea2ab96ec in_seq=0 out_seq=0 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.743+0000 7fc029ffb640 10 client.18831 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc029ffb640 1 -- 172.21.15.87:0/1753487201 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bea2dc4930 con 0x55bea2ede000 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc029ffb640 10 client.18831 handle_command_reply: tid=0 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 client.18831 shutdown 2024-04-22T13:42:41.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 msgr2=0x55bea2ecac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc034009900 tx=0x7fc034004270 comp rx=0 tx=0).stop 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 shutdown_connections 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2ede000 0x55bea2ee03f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ecdf70 0x55bea2ecb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc029ffb640 0 client.18831 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecac50 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/1753487201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 >> 172.21.15.87:0/1753487201 conn(0x55bea2ebaba0 msgr2=0x55bea2ebc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 shutdown_connections 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.744+0000 7fc04b3f8740 1 -- 172.21.15.87:0/1753487201 wait complete. 2024-04-22T13:42:41.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.745+0000 7fc04b3f8740 20 client.18831 trim_cache size 0 max 16384 2024-04-22T13:42:41.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.757+0000 7fc04b3f8740 1 Processor -- start 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.757+0000 7fc04b3f8740 1 -- start start 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec00e0 0x55bea2ed9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ecd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441ae1c0 con 0x55bea2ec0a00 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441aba90 con 0x55bea2ec00e0 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441a9900 con 0x55bea2ecdf70 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43554/0 (socket says 172.21.15.87:43554) 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 -- 172.21.15.87:0/4075918551 learned_addr learned my addr 172.21.15.87:0/4075918551 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 -- 172.21.15.87:0/4075918551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 msgr2=0x55bea2ecd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ecd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 -- 172.21.15.87:0/4075918551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec00e0 msgr2=0x55bea2ed9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec00e0 0x55bea2ed9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 -- 172.21.15.87:0/4075918551 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea2ecb520 con 0x55bea2ec0a00 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc04259c640 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc03402c170 tx=0x7fc03402c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d5ea54e4c30cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc02bfff640 1 -- 172.21.15.87:0/4075918551 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034005d50 con 0x55bea2ec0a00 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc02bfff640 1 -- 172.21.15.87:0/4075918551 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0340035f0 con 0x55bea2ec0a00 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.758+0000 7fc02bfff640 1 -- 172.21.15.87:0/4075918551 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03403fbe0 con 0x55bea2ec0a00 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- 172.21.15.87:0/4075918551 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 msgr2=0x55bea2ecd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc03402c170 tx=0x7fc03402c330 comp rx=0 tx=0).stop 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- 172.21.15.87:0/4075918551 shutdown_connections 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ecdf70 0x55bea2ecd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd380 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/4075918551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec00e0 0x55bea2ed9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- 172.21.15.87:0/4075918551 >> 172.21.15.87:0/4075918551 conn(0x55bea2ebaba0 msgr2=0x55bea2eb9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- 172.21.15.87:0/4075918551 shutdown_connections 2024-04-22T13:42:41.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- 172.21.15.87:0/4075918551 wait complete. 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- start start 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 Processor -- start 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441a9f50 con 0x55bea2ecdf70 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441ab3c0 con 0x55bea2ec0a00 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0441aa620 con 0x55bea2ec00e0 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc04b3f8740 10 client.0 _ll_register_callbacks cb 0x7fc03402cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc041d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.759+0000 7fc041d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43562/0 (socket says 172.21.15.87:43562) 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 -- 172.21.15.87:0/3655558353 learned_addr learned my addr 172.21.15.87:0/3655558353 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 -- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 msgr2=0x55bea2ecce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04259c640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 -- 172.21.15.87:0/3655558353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 msgr2=0x55bea2ecd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 -- 172.21.15.87:0/3655558353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea2f43a20 con 0x55bea2ecdf70 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04259c640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc041d9b640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc038012fd0 tx=0x7fc038005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b982fd5da78722a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038004220 con 0x55bea2ecdf70 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fc0380043c0 con 0x55bea2ecdf70 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0340035f0 con 0x55bea2ecdf70 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc0380035f0 con 0x55bea2ecdf70 2024-04-22T13:42:41.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 10 client.18837 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bea2f44bc0 con 0x55bea2ecdf70 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0340035f0 con 0x55bea2ecdf70 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc038024040 con 0x55bea2ecdf70 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 10 client.18837 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 10 client.18837 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 20 client.18837 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2f44fb0 0x55bea2f473d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 4 client.18837 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.760+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bea2e32c10 con 0x55bea2f44fb0 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc04259c640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2f44fb0 0x55bea2f473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc029ffb640 10 client.18837 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc04259c640 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2f44fb0 0x55bea2f473d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d97f764f98665111 server_cookie=6e7afe545fb47b7f in_seq=0 out_seq=0 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc029ffb640 1 -- 172.21.15.87:0/3655558353 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bea2e32c10 con 0x55bea2f44fb0 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc029ffb640 10 client.18837 handle_command_reply: tid=0 2024-04-22T13:42:41.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc04b3f8740 1 client.18837 shutdown 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 msgr2=0x55bea2f434e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.761+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc038012fd0 tx=0x7fc038005fb0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 shutdown_connections 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55bea2f44fb0 0x55bea2f473d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bea2ecdf70 0x55bea2f434e0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bea2ec0a00 0x55bea2ecd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 --2- 172.21.15.87:0/3655558353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bea2ec00e0 0x55bea2ecce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc029ffb640 0 client.18837 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 >> 172.21.15.87:0/3655558353 conn(0x55bea2ebaba0 msgr2=0x55bea2ece580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 shutdown_connections 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 1 -- 172.21.15.87:0/3655558353 wait complete. 2024-04-22T13:42:41.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc04b3f8740 20 client.18837 trim_cache size 0 max 16384 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 msgr2=0x7fc02c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 0x7fc02c0a61e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0987d0 tx=0x7fc03c06d840 comp rx=0 tx=0).stop 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 msgr2=0x7fc04419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00b970 tx=0x7fc03c00be30 comp rx=0 tx=0).stop 2024-04-22T13:42:41.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 shutdown_connections 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc02c0a3d20 0x7fc02c0a61e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc044105e20 0x7fc04419bf00 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc044102350 0x7fc044197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 --2- 172.21.15.87:0/371746670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc044101990 0x7fc044197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 >> 172.21.15.87:0/371746670 conn(0x7fc0440760e0 msgr2=0x7fc0440fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 shutdown_connections 2024-04-22T13:42:41.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.762+0000 7fc048806640 1 -- 172.21.15.87:0/371746670 wait complete. 2024-04-22T13:42:41.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 -- 172.21.15.87:0/477462647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 msgr2=0x7f18c4101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/477462647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c4101490 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009900 tx=0x7f18b802bc30 comp rx=0 tx=0).stop 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 -- 172.21.15.87:0/477462647 shutdown_connections 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/477462647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/477462647 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c4101490 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/477462647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 0x7f18c4106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 -- 172.21.15.87:0/477462647 >> 172.21.15.87:0/477462647 conn(0x7f18c4077bf0 msgr2=0x7f18c4078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 -- 172.21.15.87:0/477462647 shutdown_connections 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.872+0000 7f18c8b9f640 1 -- 172.21.15.87:0/477462647 wait complete. 2024-04-22T13:42:41.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 Processor -- start 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 -- start start 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 0x7f18c419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4112080 con 0x7f18c4101010 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4111f00 con 0x7f18c41019d0 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.873+0000 7f18c8b9f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4112200 con 0x7f18c4105d30 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43572/0 (socket says 172.21.15.87:43572) 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 -- 172.21.15.87:0/3675026943 learned_addr learned my addr 172.21.15.87:0/3675026943 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c3fff640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 0x7f18c419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 -- 172.21.15.87:0/3675026943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 msgr2=0x7f18c419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 0x7f18c419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 -- 172.21.15.87:0/3675026943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 msgr2=0x7f18c419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 -- 172.21.15.87:0/3675026943 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c419f640 con 0x7f18c4101010 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f18b400c920 tx=0x7f18b400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c0ff9640 1 -- 172.21.15.87:0/3675026943 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4007c70 con 0x7f18c4101010 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c0ff9640 1 -- 172.21.15.87:0/3675026943 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b4007e10 con 0x7f18c4101010 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c419f930 con 0x7f18c4101010 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c0ff9640 1 -- 172.21.15.87:0/3675026943 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4018820 con 0x7f18c4101010 2024-04-22T13:42:41.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.874+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c41a71c0 con 0x7f18c4101010 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.876+0000 7f18c0ff9640 1 -- 172.21.15.87:0/3675026943 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f18b400f450 con 0x7f18c4101010 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.876+0000 7f18cc9f0740 1 Processor -- start 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 -- start start 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 0x55d513b34ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b3d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d513a991f0 con 0x55d513b3dc00 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d513a364b0 con 0x55d513b35080 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d513a951b0 con 0x55d513b346d0 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43582/0 (socket says 172.21.15.87:43582) 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 0x55d513b34ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c2ffd640 1 -- 172.21.15.87:0/2466791331 learned_addr learned my addr 172.21.15.87:0/2466791331 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c37fe640 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b3d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c0ff9640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 0x7f18ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 -- 172.21.15.87:0/2466791331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 msgr2=0x55d513b3d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b3d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 -- 172.21.15.87:0/2466791331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 msgr2=0x55d513b40050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 -- 172.21.15.87:0/2466791331 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d513a281b0 con 0x55d513b346d0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 0x7f18ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c0ff9640 1 -- 172.21.15.87:0/3675026943 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f18b40d0990 con 0x7f18c4101010 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.877+0000 7f18c3fff640 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 0x55d513b34ab0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000b60 tx=0x7f18bc011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90aef038d780fb6e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18c3fff640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 0x7f18ac0a61e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18bc022000 tx=0x7f18bc01f000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18a9ffb640 1 -- 172.21.15.87:0/2466791331 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0142d0 con 0x55d513b346d0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18a9ffb640 1 -- 172.21.15.87:0/2466791331 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18bc00f990 con 0x55d513b346d0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18a9ffb640 1 -- 172.21.15.87:0/2466791331 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc012870 con 0x55d513b346d0 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 -- 172.21.15.87:0/2466791331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 msgr2=0x55d513b34ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 0x55d513b34ab0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000b60 tx=0x7f18bc011870 comp rx=0 tx=0).stop 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 -- 172.21.15.87:0/2466791331 shutdown_connections 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b3d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/2466791331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b346d0 0x55d513b34ab0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 -- 172.21.15.87:0/2466791331 >> 172.21.15.87:0/2466791331 conn(0x55d513b2eda0 msgr2=0x55d513b2f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 -- 172.21.15.87:0/2466791331 shutdown_connections 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.878+0000 7f18cc9f0740 1 -- 172.21.15.87:0/2466791331 wait complete. 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 -- start start 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 Processor -- start 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b3d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b3b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55d513b40b70 con 0x55d513b35080 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d513a52500 con 0x55d513b346d0 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d513b40cf0 con 0x55d513b3dc00 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18cc9f0740 10 client.0 _ll_register_callbacks cb 0x55d513a20b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18c2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b3d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18c2ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43588/0 (socket says 172.21.15.87:43588) 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18c2ffd640 1 -- 172.21.15.87:0/760267588 learned_addr learned my addr 172.21.15.87:0/760267588 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.879+0000 7f18c3fff640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b3b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 -- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 msgr2=0x55d513b3b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b3b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 -- 172.21.15.87:0/760267588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 msgr2=0x55d513b3d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b3d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 -- 172.21.15.87:0/760267588 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5137ab3f0 con 0x55d513b35080 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c3fff640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b3b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c37fe640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b3d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f18b8006010 tx=0x7f18b8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d20f13a66f86298 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8005230 con 0x55d513b35080 2024-04-22T13:42:41.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f18b80053d0 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc00f990 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f18b80055d0 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18cc9f0740 10 client.18849 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5138757d0 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.880+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18bc00f990 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f18b8003b90 con 0x55d513b35080 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 10 client.18849 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 10 client.18849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 20 client.18849 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513b4de80 0x55d513b50270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 4 client.18849 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d513a38930 con 0x55d513b4de80 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18c37fe640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513b4de80 0x55d513b50270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18c37fe640 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513b4de80 0x55d513b50270 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=961af7f230af291d server_cookie=8259c3159cc8b5f7 in_seq=0 out_seq=0 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.881+0000 7f18a2ffd640 10 client.18849 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18a2ffd640 1 -- 172.21.15.87:0/760267588 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d513a38930 con 0x55d513b4de80 2024-04-22T13:42:41.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18a2ffd640 10 client.18849 handle_command_reply: tid=0 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 client.18849 shutdown 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 msgr2=0x55d513b3a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f18b8006010 tx=0x7f18b8004270 comp rx=0 tx=0).stop 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 shutdown_connections 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513b4de80 0x55d513b50270 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b3b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b35080 0x55d513b3a930 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18a2ffd640 0 client.18849 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/760267588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b3d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 >> 172.21.15.87:0/760267588 conn(0x55d513b2eda0 msgr2=0x55d513b30af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 shutdown_connections 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 1 -- 172.21.15.87:0/760267588 wait complete. 2024-04-22T13:42:41.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.882+0000 7f18cc9f0740 20 client.18849 trim_cache size 0 max 16384 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 Processor -- start 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 -- start start 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b55a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b55fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41b2200 con 0x55d513b346d0 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41afad0 con 0x55d513b35080 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.895+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ad940 con 0x55d513b3dc00 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43592/0 (socket says 172.21.15.87:43592) 2024-04-22T13:42:41.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 -- 172.21.15.87:0/3774993044 learned_addr learned my addr 172.21.15.87:0/3774993044 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b55a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c3fff640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b55fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 -- 172.21.15.87:0/3774993044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 msgr2=0x55d513b55fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b55fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 -- 172.21.15.87:0/3774993044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 msgr2=0x55d513b55a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b55a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 -- 172.21.15.87:0/3774993044 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d513b50830 con 0x55d513b346d0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b55a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18c37fe640 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f18b4098dd0 tx=0x7f18b4087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94db243daae65682 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18a97fa640 1 -- 172.21.15.87:0/3774993044 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4098fe0 con 0x55d513b346d0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18a97fa640 1 -- 172.21.15.87:0/3774993044 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b409c970 con 0x55d513b346d0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.896+0000 7f18a97fa640 1 -- 172.21.15.87:0/3774993044 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b40a5150 con 0x55d513b346d0 2024-04-22T13:42:41.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- 172.21.15.87:0/3774993044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 msgr2=0x55d513b49a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f18b4098dd0 tx=0x7f18b4087040 comp rx=0 tx=0).stop 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- 172.21.15.87:0/3774993044 shutdown_connections 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b3dc00 0x55d513b55fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b35080 0x55d513b55a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/3774993044 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b346d0 0x55d513b49a30 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- 172.21.15.87:0/3774993044 >> 172.21.15.87:0/3774993044 conn(0x55d513b2eda0 msgr2=0x55d513b30980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- 172.21.15.87:0/3774993044 shutdown_connections 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- 172.21.15.87:0/3774993044 wait complete. 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 -- start start 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.897+0000 7f18cc9f0740 1 Processor -- start 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b41b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b35080 0x55d513b420a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41adf90 con 0x55d513b3dc00 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41af400 con 0x55d513b346d0 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ae660 con 0x55d513b35080 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 10 client.0 _ll_register_callbacks cb 0x7f18b802cc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43600/0 (socket says 172.21.15.87:43600) 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 -- 172.21.15.87:0/1636376769 learned_addr learned my addr 172.21.15.87:0/1636376769 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c37fe640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b41b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c2ffd640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b35080 0x55d513b420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 -- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b35080 msgr2=0x55d513b420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b35080 0x55d513b420a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 -- 172.21.15.87:0/1636376769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 msgr2=0x55d513b41b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b41b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 -- 172.21.15.87:0/1636376769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d513bb34d0 con 0x55d513b3dc00 2024-04-22T13:42:41.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18c3fff640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f18bc016fd0 tx=0x7f18bc012d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a99e6433152e004 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc012650 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f18bc0127f0 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b409c970 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f18bc041050 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 10 client.18855 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.898+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d513bb4770 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b409c970 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f18bc00fc90 con 0x55d513b3dc00 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 10 client.18855 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 10 client.18855 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 20 client.18855 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513bb4cb0 0x55d513bb70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:41.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 4 client.18855 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d513aa6c10 con 0x55d513bb4cb0 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18c37fe640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513bb4cb0 0x55d513bb70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18c37fe640 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513bb4cb0 0x55d513bb70d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41f21c8f749ab893 server_cookie=6888adec288482b in_seq=0 out_seq=0 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.899+0000 7f18a2ffd640 10 client.18855 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18a2ffd640 1 -- 172.21.15.87:0/1636376769 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d513aa6c10 con 0x55d513bb4cb0 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18a2ffd640 10 client.18855 handle_command_reply: tid=0 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 client.18855 shutdown 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 msgr2=0x55d513b425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f18bc016fd0 tx=0x7f18bc012d10 comp rx=0 tx=0).stop 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 shutdown_connections 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55d513bb4cb0 0x55d513bb70d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d513b3dc00 0x55d513b425e0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18a2ffd640 0 client.18855 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:42:41.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d513b35080 0x55d513b420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 --2- 172.21.15.87:0/1636376769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d513b346d0 0x55d513b41b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 >> 172.21.15.87:0/1636376769 conn(0x55d513b2eda0 msgr2=0x55d513b2d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 shutdown_connections 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 1 -- 172.21.15.87:0/1636376769 wait complete. 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18cc9f0740 20 client.18855 trim_cache size 0 max 16384 2024-04-22T13:42:41.903 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 msgr2=0x7f18ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 0x7f18ac0a61e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18bc022000 tx=0x7f18bc01f000 comp rx=0 tx=0).stop 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 msgr2=0x7f18c419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f18b400c920 tx=0x7f18b400cde0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 shutdown_connections 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f18ac0a3d20 0x7f18ac0a61e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18c4105d30 0x7f18c419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18c41019d0 0x7f18c419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.900+0000 7f18c8b9f640 1 --2- 172.21.15.87:0/3675026943 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c4101010 0x7f18c419a650 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:41.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.901+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 >> 172.21.15.87:0/3675026943 conn(0x7f18c4077bf0 msgr2=0x7f18c410c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:41.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.901+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 shutdown_connections 2024-04-22T13:42:41.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:41.901+0000 7f18c8b9f640 1 -- 172.21.15.87:0/3675026943 wait complete. 2024-04-22T13:42:41.906 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:41 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2255659158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:41.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:42:41.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:41 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2255659158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:41.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:41 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2255659158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 -- 172.21.15.87:0/2064140531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 msgr2=0x7f6f7010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 --2- 172.21.15.87:0/2064140531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f7010ee30 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6f68009ca0 tx=0x7f6f6802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 -- 172.21.15.87:0/2064140531 shutdown_connections 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 --2- 172.21.15.87:0/2064140531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f7010ee30 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 --2- 172.21.15.87:0/2064140531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f70102b40 0x7f6f70106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 --2- 172.21.15.87:0/2064140531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 -- 172.21.15.87:0/2064140531 >> 172.21.15.87:0/2064140531 conn(0x7f6f700793a0 msgr2=0x7f6f700797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 -- 172.21.15.87:0/2064140531 shutdown_connections 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.011+0000 7f6f7725c640 1 -- 172.21.15.87:0/2064140531 wait complete. 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 Processor -- start 2024-04-22T13:42:42.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 -- start start 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 0x7f6f7019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70112220 con 0x7f6f70107260 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701120a0 con 0x7f6f7019cfe0 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f7725c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701123a0 con 0x7f6f70102190 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43620/0 (socket says 172.21.15.87:43620) 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f6ffff640 1 -- 172.21.15.87:0/3539057654 learned_addr learned my addr 172.21.15.87:0/3539057654 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.012+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 0x7f6f7019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 -- 172.21.15.87:0/3539057654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 msgr2=0x7f6f70198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 -- 172.21.15.87:0/3539057654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 msgr2=0x7f6f7019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 0x7f6f7019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 -- 172.21.15.87:0/3539057654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7019dad0 con 0x7f6f70107260 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 0x7f6f7019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c970 tx=0x7f6f6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6dffb640 1 -- 172.21.15.87:0/3539057654 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64014070 con 0x7f6f70107260 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6dffb640 1 -- 172.21.15.87:0/3539057654 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f64004440 con 0x7f6f70107260 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f6dffb640 1 -- 172.21.15.87:0/3539057654 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64003e50 con 0x7f6f70107260 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7019ddc0 con 0x7f6f70107260 2024-04-22T13:42:42.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.013+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f701a55b0 con 0x7f6f70107260 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.015+0000 7f6f6dffb640 1 -- 172.21.15.87:0/3539057654 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6f64017070 con 0x7f6f70107260 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 Processor -- start 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 -- start start 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f6dffb640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 0x7f6f580a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d03430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d05e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5599b6c5f1f0 con 0x5599b6d03a00 2024-04-22T13:42:42.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5599b6bfc4b0 con 0x5599b6cfae80 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f79e52740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5599b6c5b1b0 con 0x5599b6cfa4d0 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 0x7f6f580a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f6dffb640 1 -- 172.21.15.87:0/3539057654 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f6f640d06c0 con 0x7f6f70107260 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f74ff8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d05e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f74ff8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:33604/0 (socket says 172.21.15.87:33604) 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f74ff8640 1 -- 172.21.15.87:0/807660070 learned_addr learned my addr 172.21.15.87:0/807660070 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d03430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 -- 172.21.15.87:0/807660070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 msgr2=0x5599b6cfa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 -- 172.21.15.87:0/807660070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 msgr2=0x5599b6d03430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d03430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 -- 172.21.15.87:0/807660070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b6bee1b0 con 0x5599b6d03a00 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:42.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 0x7f6f580a6380 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f68032460 tx=0x7f6f6802cd20 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.016+0000 7f6f757f9640 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d05e50 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802c700 tx=0x7f6f68046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb0ed2672810bda server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f56ffd640 1 -- 172.21.15.87:0/807660070 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68003970 con 0x5599b6d03a00 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f56ffd640 1 -- 172.21.15.87:0/807660070 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f68051070 con 0x5599b6d03a00 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f56ffd640 1 -- 172.21.15.87:0/807660070 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68004830 con 0x5599b6d03a00 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 -- 172.21.15.87:0/807660070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 msgr2=0x5599b6d05e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d05e50 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802c700 tx=0x7f6f68046000 comp rx=0 tx=0).stop 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 -- 172.21.15.87:0/807660070 shutdown_connections 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d05e50 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d03430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 --2- 172.21.15.87:0/807660070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6cfa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 -- 172.21.15.87:0/807660070 >> 172.21.15.87:0/807660070 conn(0x5599b6cf4ba0 msgr2=0x5599b6cf4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 -- 172.21.15.87:0/807660070 shutdown_connections 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.017+0000 7f6f79e52740 1 -- 172.21.15.87:0/807660070 wait complete. 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 -- start start 2024-04-22T13:42:42.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 Processor -- start 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d00650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5599b6d06910 con 0x5599b6d03a00 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5599b6c18500 con 0x5599b6cfae80 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5599b6d06a90 con 0x5599b6cfa4d0 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f79e52740 10 client.0 _ll_register_callbacks cb 0x5599b6be6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43642/0 (socket says 172.21.15.87:43642) 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 -- 172.21.15.87:0/721651424 learned_addr learned my addr 172.21.15.87:0/721651424 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d00650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 -- 172.21.15.87:0/721651424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 msgr2=0x5599b6d032c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d032c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 -- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 msgr2=0x5599b6d00650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d00650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.018+0000 7f6f757f9640 1 -- 172.21.15.87:0/721651424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b69713f0 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f757f9640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802c130 tx=0x7f6f68032aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9319bc70b0e06c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f680047b0 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f6f68003970 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f68051070 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6f68003b70 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 10 client.18867 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5599b6a3b7d0 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f68051070 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6f6804b070 con 0x5599b6d03a00 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 10 client.18867 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:42.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 10 client.18867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 20 client.18867 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d139c0 0x5599b6d15db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 4 client.18867 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.019+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599b6bfe930 con 0x5599b6d139c0 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.020+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d139c0 0x5599b6d15db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.020+0000 7f6f4ffff640 10 client.18867 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.020+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d139c0 0x5599b6d15db0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69c3c445c008016 server_cookie=9b3249af9840c79d in_seq=0 out_seq=0 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f4ffff640 1 -- 172.21.15.87:0/721651424 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599b6bfe930 con 0x5599b6d139c0 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f4ffff640 10 client.18867 handle_command_reply: tid=0 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 client.18867 shutdown 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 msgr2=0x5599b6d00bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802c130 tx=0x7f6f68032aa0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 shutdown_connections 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d139c0 0x5599b6d15db0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6d03a00 0x5599b6d00bb0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6cfae80 0x5599b6d00650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f4ffff640 0 client.18867 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 --2- 172.21.15.87:0/721651424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 >> 172.21.15.87:0/721651424 conn(0x5599b6cf4ba0 msgr2=0x5599b6cf68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 shutdown_connections 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 1 -- 172.21.15.87:0/721651424 wait complete. 2024-04-22T13:42:42.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.021+0000 7f6f79e52740 20 client.18867 trim_cache size 0 max 16384 2024-04-22T13:42:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 Processor -- start 2024-04-22T13:42:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 -- start start 2024-04-22T13:42:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d02d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d18dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701b03c0 con 0x5599b6cfae80 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701adc90 con 0x5599b6d03a00 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.034+0000 7f6f79e52740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701abb00 con 0x5599b6cfa4d0 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43648/0 (socket says 172.21.15.87:43648) 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 -- 172.21.15.87:0/4125649704 learned_addr learned my addr 172.21.15.87:0/4125649704 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f757f9640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d18dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d02d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 -- 172.21.15.87:0/4125649704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 msgr2=0x5599b6d02d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d02d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 -- 172.21.15.87:0/4125649704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 msgr2=0x5599b6d18dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d18dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 -- 172.21.15.87:0/4125649704 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b6cf0470 con 0x5599b6cfae80 2024-04-22T13:42:42.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d02d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f757f9640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000c00 tx=0x7f6f64079da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8a7ca407d2b29b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f567fc640 1 -- 172.21.15.87:0/4125649704 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640798b0 con 0x5599b6cfae80 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f567fc640 1 -- 172.21.15.87:0/4125649704 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f6409c6a0 con 0x5599b6cfae80 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.035+0000 7f6f567fc640 1 -- 172.21.15.87:0/4125649704 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640036c0 con 0x5599b6cfae80 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- 172.21.15.87:0/4125649704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 msgr2=0x5599b6d032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000c00 tx=0x7f6f64079da0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- 172.21.15.87:0/4125649704 shutdown_connections 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfae80 0x5599b6d032c0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 --2- 172.21.15.87:0/4125649704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfa4d0 0x5599b6d02d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- 172.21.15.87:0/4125649704 >> 172.21.15.87:0/4125649704 conn(0x5599b6cf4ba0 msgr2=0x5599b6cfbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- 172.21.15.87:0/4125649704 shutdown_connections 2024-04-22T13:42:42.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- 172.21.15.87:0/4125649704 wait complete. 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 -- start start 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 Processor -- start 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.036+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 0x5599b6d16c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d17190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701ac150 con 0x5599b6cfa4d0 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701ad5c0 con 0x5599b6d03a00 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701ac820 con 0x5599b6cfae80 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 10 client.0 _ll_register_callbacks cb 0x7f6f68009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43656/0 (socket says 172.21.15.87:43656) 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 -- 172.21.15.87:0/3790476811 learned_addr learned my addr 172.21.15.87:0/3790476811 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 0x5599b6d16c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d17190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 -- 172.21.15.87:0/3790476811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 msgr2=0x5599b6d16c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 0x5599b6d16c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 -- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 msgr2=0x5599b6d17190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d17190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 -- 172.21.15.87:0/3790476811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b6d78ba0 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 0x5599b6d16c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f757f9640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d17190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f74ff8640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6f60004810 tx=0x7f6f60014700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6ef251121220ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6001f030 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f6f6001bb50 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f6409c6a0 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6f6001bd50 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 10 client.18873 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.037+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5599b6d79d50 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f6409c6a0 con 0x5599b6cfa4d0 2024-04-22T13:42:42.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6f6002d6e0 con 0x5599b6cfa4d0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 10 client.18873 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 10 client.18873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 20 client.18873 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d7a4d0 0x5599b6d7c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 4 client.18873 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599b6c6cc10 con 0x5599b6d7a4d0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.038+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d7a4d0 0x5599b6d7c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f6ffff640 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d7a4d0 0x5599b6d7c8f0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55d6e3d5c6b5bc9 server_cookie=cbb9fde090b0a5db in_seq=0 out_seq=0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f4ffff640 10 client.18873 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f4ffff640 1 -- 172.21.15.87:0/3790476811 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599b6c6cc10 con 0x5599b6d7a4d0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f4ffff640 10 client.18873 handle_command_reply: tid=0 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 client.18873 shutdown 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 msgr2=0x5599b6d16710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6f60004810 tx=0x7f6f60014700 comp rx=0 tx=0).stop 2024-04-22T13:42:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 shutdown_connections 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5599b6d7a4d0 0x5599b6d7c8f0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599b6d03a00 0x5599b6d17190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f4ffff640 0 client.18873 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5599b6cfae80 0x5599b6d16c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 --2- 172.21.15.87:0/3790476811 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5599b6cfa4d0 0x5599b6d16710 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 >> 172.21.15.87:0/3790476811 conn(0x5599b6cf4ba0 msgr2=0x5599b6cfb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 shutdown_connections 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 1 -- 172.21.15.87:0/3790476811 wait complete. 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.039+0000 7f6f79e52740 20 client.18873 trim_cache size 0 max 16384 2024-04-22T13:42:42.042 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 msgr2=0x7f6f580a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 0x7f6f580a6380 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f68032460 tx=0x7f6f6802cd20 comp rx=0 tx=0).stop 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 msgr2=0x7f6f70198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c970 tx=0x7f6f6400ce30 comp rx=0 tx=0).stop 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 shutdown_connections 2024-04-22T13:42:42.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f6f580a3ec0 0x7f6f580a6380 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f7019cfe0 0x7f6f7019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6f70107260 0x7f6f70198c60 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 --2- 172.21.15.87:0/3539057654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f70102190 0x7f6f70198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 >> 172.21.15.87:0/3539057654 conn(0x7f6f700793a0 msgr2=0x7f6f700ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 shutdown_connections 2024-04-22T13:42:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:42.040+0000 7f6f7725c640 1 -- 172.21.15.87:0/3539057654 wait complete. 2024-04-22T13:42:42.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:42 smithi132 ceph-mon[31531]: pgmap v339: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:42.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:42 smithi087 ceph-mon[27999]: pgmap v339: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:43.047 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:42:43.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:42:43.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 -- 172.21.15.87:0/2308230555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 msgr2=0x7f58dc10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 --2- 172.21.15.87:0/2308230555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc10ee30 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ca0 tx=0x7f58d402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 -- 172.21.15.87:0/2308230555 shutdown_connections 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 --2- 172.21.15.87:0/2308230555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc10ee30 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 --2- 172.21.15.87:0/2308230555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc102b40 0x7f58dc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 --2- 172.21.15.87:0/2308230555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.166+0000 7f58e38c7640 1 -- 172.21.15.87:0/2308230555 >> 172.21.15.87:0/2308230555 conn(0x7f58dc0fd990 msgr2=0x7f58dc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- 172.21.15.87:0/2308230555 shutdown_connections 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- 172.21.15.87:0/2308230555 wait complete. 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 Processor -- start 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- start start 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc19cfd0 0x7f58dc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1121b0 con 0x7f58dc107260 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc112030 con 0x7f58dc102190 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.167+0000 7f58e38c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc112330 con 0x7f58dc19cfd0 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43676/0 (socket says 172.21.15.87:43676) 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 -- 172.21.15.87:0/3506241337 learned_addr learned my addr 172.21.15.87:0/3506241337 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e1e64640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc19cfd0 0x7f58dc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e1663640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 -- 172.21.15.87:0/3506241337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc19cfd0 msgr2=0x7f58dc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc19cfd0 0x7f58dc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 -- 172.21.15.87:0/3506241337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 msgr2=0x7f58dc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 -- 172.21.15.87:0/3506241337 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc19dac0 con 0x7f58dc107260 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e1663640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e0e62640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f58d800d770 tx=0x7f58d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58d27fc640 1 -- 172.21.15.87:0/3506241337 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d8017070 con 0x7f58dc107260 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58d27fc640 1 -- 172.21.15.87:0/3506241337 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d8009c30 con 0x7f58dc107260 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58dc19ddb0 con 0x7f58dc107260 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58d27fc640 1 -- 172.21.15.87:0/3506241337 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d800f620 con 0x7f58dc107260 2024-04-22T13:42:43.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.168+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58dc1a55a0 con 0x7f58dc107260 2024-04-22T13:42:43.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.170+0000 7f58d27fc640 1 -- 172.21.15.87:0/3506241337 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f58d801a070 con 0x7f58dc107260 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 Processor -- start 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 -- start start 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 0x55e6ef8a7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8aa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef802f00 con 0x55e6ef89e8e0 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef7a0590 con 0x55e6ef8a7e10 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e658b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef7feec0 con 0x55e6ef89f290 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43678/0 (socket says 172.21.15.87:43678) 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 -- 172.21.15.87:0/4168107062 learned_addr learned my addr 172.21.15.87:0/4168107062 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e0e62640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58d27fc640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 0x7f58c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1663640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 0x55e6ef8a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 -- 172.21.15.87:0/4168107062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 msgr2=0x55e6ef8a7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 0x55e6ef8a7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 -- 172.21.15.87:0/4168107062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef8a7e10 msgr2=0x55e6ef8aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.171+0000 7f58e1e64640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58e1e64640 1 -- 172.21.15.87:0/4168107062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6ef5dd7e0 con 0x55e6ef89e8e0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58e1663640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 0x55e6ef8a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58d27fc640 1 -- 172.21.15.87:0/3506241337 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f58d80d02a0 con 0x7f58dc107260 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58e1e64640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 0x7f58c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58e1e64640 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ca0 tx=0x7f58d4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2b26649920e1df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58c77fe640 1 -- 172.21.15.87:0/4168107062 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4044040 con 0x55e6ef89e8e0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58c77fe640 1 -- 172.21.15.87:0/4168107062 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d4003e20 con 0x55e6ef89e8e0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.172+0000 7f58c77fe640 1 -- 172.21.15.87:0/4168107062 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4039040 con 0x55e6ef89e8e0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- 172.21.15.87:0/4168107062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 msgr2=0x55e6ef89ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e1e64640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 0x7f58c80a61e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f58d404a000 tx=0x7f58d4047000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ca0 tx=0x7f58d4009900 comp rx=0 tx=0).stop 2024-04-22T13:42:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- 172.21.15.87:0/4168107062 shutdown_connections 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89f290 0x55e6ef8a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 --2- 172.21.15.87:0/4168107062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef89e8e0 0x55e6ef89ecc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- 172.21.15.87:0/4168107062 >> 172.21.15.87:0/4168107062 conn(0x55e6ef898fb0 msgr2=0x55e6ef899390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- 172.21.15.87:0/4168107062 shutdown_connections 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- 172.21.15.87:0/4168107062 wait complete. 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 -- start start 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.173+0000 7f58e658b740 1 Processor -- start 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8a5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef8aad80 con 0x55e6ef8a7e10 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef7bc580 con 0x55e6ef89f290 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6ef8aaf00 con 0x55e6ef89e8e0 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e658b740 10 client.0 _ll_register_callbacks cb 0x55e6ef7c2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43688/0 (socket says 172.21.15.87:43688) 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 -- 172.21.15.87:0/556385890 learned_addr learned my addr 172.21.15.87:0/556385890 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1663640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 -- 172.21.15.87:0/556385890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 msgr2=0x55e6ef8a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 -- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 msgr2=0x55e6ef8a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 -- 172.21.15.87:0/556385890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6ef63df70 con 0x55e6ef8a7e10 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58e1e64640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f58d4003db0 tx=0x7f58d4004650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258b83403e7e5a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.174+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4056020 con 0x55e6ef8a7e10 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f58d4032ca0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d4003e20 con 0x55e6ef8a7e10 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f58d4061050 con 0x55e6ef8a7e10 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 10 client.18885 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e6ef4d5170 con 0x55e6ef8a7e10 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d4003e20 con 0x55e6ef8a7e10 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f58d403f040 con 0x55e6ef8a7e10 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 10 client.18885 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 10 client.18885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 20 client.18885 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef8b81c0 0x55e6ef8ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 4 client.18885 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.175+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6ef810920 con 0x55e6ef8b81c0 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.176+0000 7f58e1663640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef8b81c0 0x55e6ef8ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.176+0000 7f58c4ff9640 10 client.18885 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.176+0000 7f58e1663640 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef8b81c0 0x55e6ef8ba5b0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b02f57f327d4dd8 server_cookie=903cbef64749b640 in_seq=0 out_seq=0 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58c4ff9640 1 -- 172.21.15.87:0/556385890 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6ef810920 con 0x55e6ef8b81c0 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58c4ff9640 10 client.18885 handle_command_reply: tid=0 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 client.18885 shutdown 2024-04-22T13:42:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 msgr2=0x55e6ef8a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f58d4003db0 tx=0x7f58d4004650 comp rx=0 tx=0).stop 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 shutdown_connections 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef8b81c0 0x55e6ef8ba5b0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a5840 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58c4ff9640 0 client.18885 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 --2- 172.21.15.87:0/556385890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 >> 172.21.15.87:0/556385890 conn(0x55e6ef898fb0 msgr2=0x55e6ef89ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 shutdown_connections 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 1 -- 172.21.15.87:0/556385890 wait complete. 2024-04-22T13:42:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.177+0000 7f58e658b740 20 client.18885 trim_cache size 0 max 16384 2024-04-22T13:42:43.179 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:42 smithi120 ceph-mon[31101]: pgmap v339: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 Processor -- start 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 -- start start 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1b0370 con 0x55e6ef8a7e10 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1adc40 con 0x55e6ef89f290 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.194+0000 7f58e658b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1abab0 con 0x55e6ef89e8e0 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43700/0 (socket says 172.21.15.87:43700) 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 -- 172.21.15.87:0/358650549 learned_addr learned my addr 172.21.15.87:0/358650549 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 -- 172.21.15.87:0/358650549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 msgr2=0x55e6ef8a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 -- 172.21.15.87:0/358650549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 msgr2=0x55e6ef8b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e0e62640 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 -- 172.21.15.87:0/358650549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6ef8aa6c0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58e1e64640 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f58d402c630 tx=0x7f58d4009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a826c33e7123fe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58c6ffd640 1 -- 172.21.15.87:0/358650549 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4056070 con 0x55e6ef8a7e10 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58c6ffd640 1 -- 172.21.15.87:0/358650549 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d40098e0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.195+0000 7f58c6ffd640 1 -- 172.21.15.87:0/358650549 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4044040 con 0x55e6ef8a7e10 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- 172.21.15.87:0/358650549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 msgr2=0x55e6ef8a3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f58d402c630 tx=0x7f58d4009540 comp rx=0 tx=0).stop 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- 172.21.15.87:0/358650549 shutdown_connections 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef8a3c70 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- 172.21.15.87:0/358650549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- 172.21.15.87:0/358650549 >> 172.21.15.87:0/358650549 conn(0x55e6ef898fb0 msgr2=0x55e6ef8a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- 172.21.15.87:0/358650549 shutdown_connections 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- 172.21.15.87:0/358650549 wait complete. 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- start start 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 Processor -- start 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ac100 con 0x55e6ef8a7e10 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ad570 con 0x55e6ef89f290 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.196+0000 7f58e658b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ac7d0 con 0x55e6ef89e8e0 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e658b740 10 client.0 _ll_register_callbacks cb 0x7f58d80985a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43712/0 (socket says 172.21.15.87:43712) 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 -- 172.21.15.87:0/689153291 learned_addr learned my addr 172.21.15.87:0/689153291 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1663640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e0e62640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 -- 172.21.15.87:0/689153291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 msgr2=0x55e6ef8ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 -- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 msgr2=0x55e6ef8accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 -- 172.21.15.87:0/689153291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6ef916c50 con 0x55e6ef8a7e10 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1663640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e1e64640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f58d402c130 tx=0x7f58d4003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3123115526399b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4072060 con 0x55e6ef8a7e10 2024-04-22T13:42:43.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f58d4004450 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d40098e0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f58d4044040 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e658b740 10 client.18891 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.197+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e6ef91eff0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d40098e0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f58d40659a0 con 0x55e6ef8a7e10 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 10 client.18891 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 10 client.18891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 20 client.18891 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef91f5a0 0x55e6ef921990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 4 client.18891 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e6ef80a790 con 0x55e6ef91f5a0 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e1663640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef91f5a0 0x55e6ef921990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58c4ff9640 10 client.18891 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.198+0000 7f58e1663640 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef91f5a0 0x55e6ef921990 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=670b069ced93f253 server_cookie=858062c149a7a345 in_seq=0 out_seq=0 2024-04-22T13:42:43.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.199+0000 7f58c4ff9640 1 -- 172.21.15.87:0/689153291 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e6ef80a790 con 0x55e6ef91f5a0 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.199+0000 7f58c4ff9640 10 client.18891 handle_command_reply: tid=0 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.199+0000 7f58e658b740 1 client.18891 shutdown 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 msgr2=0x55e6ef916470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f58d402c130 tx=0x7f58d4003eb0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 shutdown_connections 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6ef91f5a0 0x55e6ef921990 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6ef8a7e10 0x55e6ef916470 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6ef89f290 0x55e6ef8accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 --2- 172.21.15.87:0/689153291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6ef89e8e0 0x55e6ef8ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 >> 172.21.15.87:0/689153291 conn(0x55e6ef898fb0 msgr2=0x55e6ef8a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58c4ff9640 0 client.18891 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 shutdown_connections 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 1 -- 172.21.15.87:0/689153291 wait complete. 2024-04-22T13:42:43.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e658b740 20 client.18891 trim_cache size 0 max 16384 2024-04-22T13:42:43.202 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:43.202 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:42:43.202 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "f9a3d2d7-6d91-4f8e-8396-e7c0925d472d", 2024-04-22T13:42:43.202 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:42:43.202 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 msgr2=0x7f58c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 0x7f58c80a61e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f58d404a000 tx=0x7f58d4047000 comp rx=0 tx=0).stop 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 msgr2=0x7f58dc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f58d800d770 tx=0x7f58d800dc30 comp rx=0 tx=0).stop 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 shutdown_connections 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f58c80a3d20 0x7f58c80a61e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58dc19cfd0 0x7f58dc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f58dc107260 0x7f58dc198c50 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 --2- 172.21.15.87:0/3506241337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58dc102190 0x7f58dc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 >> 172.21.15.87:0/3506241337 conn(0x7f58dc0fd990 msgr2=0x7f58dc0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 shutdown_connections 2024-04-22T13:42:43.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.200+0000 7f58e38c7640 1 -- 172.21.15.87:0/3506241337 wait complete. 2024-04-22T13:42:43.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:42:43.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:42:43.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2065993480 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac106130 msgr2=0x7f1dac10dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2065993480 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac106130 0x7f1dac10dd00 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009ca0 tx=0x7f1da402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2065993480 shutdown_connections 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2065993480 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac106130 0x7f1dac10dd00 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2065993480 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac101a10 0x7f1dac105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2065993480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac101060 0x7f1dac101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2065993480 >> 172.21.15.87:0/2065993480 conn(0x7f1dac0771d0 msgr2=0x7f1dac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2065993480 shutdown_connections 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.312+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2065993480 wait complete. 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 Processor -- start 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 -- start start 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac106130 0x7f1dac198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac19cf50 0x7f1dac19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac111080 con 0x7f1dac101060 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac110f00 con 0x7f1dac19cf50 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db2ec7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac111200 con 0x7f1dac106130 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43730/0 (socket says 172.21.15.87:43730) 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 -- 172.21.15.87:0/2808457720 learned_addr learned my addr 172.21.15.87:0/2808457720 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db1464640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac19cf50 0x7f1dac19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1dabfff640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac106130 0x7f1dac198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 -- 172.21.15.87:0/2808457720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac106130 msgr2=0x7f1dac198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac106130 0x7f1dac198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 -- 172.21.15.87:0/2808457720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac19cf50 msgr2=0x7f1dac19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac19cf50 0x7f1dac19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 -- 172.21.15.87:0/2808457720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac19da40 con 0x7f1dac101060 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.313+0000 7f1db0c63640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f1da000d950 tx=0x7f1da000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.314+0000 7f1da9ffb640 1 -- 172.21.15.87:0/2808457720 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0017030 con 0x7f1dac101060 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.314+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac19dd30 con 0x7f1dac101060 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.314+0000 7f1da9ffb640 1 -- 172.21.15.87:0/2808457720 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da0004020 con 0x7f1dac101060 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.314+0000 7f1da9ffb640 1 -- 172.21.15.87:0/2808457720 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00047a0 con 0x7f1dac101060 2024-04-22T13:42:43.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.314+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac1a5570 con 0x7f1dac101060 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1da9ffb640 1 -- 172.21.15.87:0/2808457720 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1da0009a30 con 0x7f1dac101060 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 Processor -- start 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 -- start start 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1da9ffb640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 0x7f1d9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a57d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a58d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561b4a4e21e0 con 0x561b4a57db50 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b4a47f390 con 0x561b4a57d230 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.316+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561b4a4de1a0 con 0x561b4a58ae90 2024-04-22T13:42:43.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1db1464640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 0x7f1d9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1da9ffb640 1 -- 172.21.15.87:0/2808457720 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f1da00d0240 con 0x7f1dac101060 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43738/0 (socket says 172.21.15.87:43738) 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 -- 172.21.15.87:0/1656170759 learned_addr learned my addr 172.21.15.87:0/1656170759 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1db0c63640 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a57d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1db1464640 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a58d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 -- 172.21.15.87:0/1656170759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 msgr2=0x561b4a58d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a58d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 -- 172.21.15.87:0/1656170759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 msgr2=0x561b4a57d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a57d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 -- 172.21.15.87:0/1656170759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b4a2be7d0 con 0x561b4a57db50 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb90 tx=0x7f1d9800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6533812c9c2b6b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1db1464640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 0x7f1d9c0a61e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1da4004af0 tx=0x7f1da4032ca0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1d96ffd640 1 -- 172.21.15.87:0/1656170759 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98005220 con 0x561b4a57db50 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1d96ffd640 1 -- 172.21.15.87:0/1656170759 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d98004c50 con 0x561b4a57db50 2024-04-22T13:42:43.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.317+0000 7f1d96ffd640 1 -- 172.21.15.87:0/1656170759 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98010720 con 0x561b4a57db50 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1656170759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 msgr2=0x561b4a58a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb90 tx=0x7f1d9800e720 comp rx=0 tx=0).stop 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1656170759 shutdown_connections 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a58d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58a950 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1656170759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a57d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1656170759 >> 172.21.15.87:0/1656170759 conn(0x561b4a577cb0 msgr2=0x561b4a578090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1656170759 shutdown_connections 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1656170759 wait complete. 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 -- start start 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.318+0000 7f1db5b8a740 1 Processor -- start 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57db50 0x561b4a5883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561b4a58dcd0 con 0x561b4a57d230 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b4a49b2f0 con 0x561b4a58ae90 2024-04-22T13:42:43.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561b4a58de50 con 0x561b4a57db50 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 10 client.0 _ll_register_callbacks cb 0x561b4a4a8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43742/0 (socket says 172.21.15.87:43742) 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 -- 172.21.15.87:0/972760885 learned_addr learned my addr 172.21.15.87:0/972760885 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1dabfff640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57db50 0x561b4a5883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db1464640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 -- 172.21.15.87:0/972760885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57db50 msgr2=0x561b4a5883d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57db50 0x561b4a5883d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 -- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 msgr2=0x561b4a588910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 -- 172.21.15.87:0/972760885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b4a1f43f0 con 0x561b4a57d230 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db1464640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db0c63640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1da0098630 tx=0x7f1da00b5a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c323915d9bc2787 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1d8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00a05d0 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f1da00041d0 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.319+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d98004c50 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1da00b8020 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 10 client.18903 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561b4a1b40f0 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d98004c50 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1da00b5d60 con 0x561b4a57d230 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 10 client.18903 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 10 client.18903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 20 client.18903 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a59d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 4 client.18903 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b4a4efc00 con 0x561b4a59ae20 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.320+0000 7f1dabfff640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a59d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1d8bfff640 10 client.18903 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1dabfff640 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a59d290 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b3d5fc427c04e6e server_cookie=db871a7197996ca0 in_seq=0 out_seq=0 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1d8bfff640 1 -- 172.21.15.87:0/972760885 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b4a4efc00 con 0x561b4a59ae20 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1d8bfff640 10 client.18903 handle_command_reply: tid=0 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 client.18903 shutdown 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 msgr2=0x561b4a587e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1da0098630 tx=0x7f1da00b5a60 comp rx=0 tx=0).stop 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 shutdown_connections 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a59d290 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57db50 0x561b4a5883d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1d8bfff640 0 client.18903 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/972760885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57d230 0x561b4a587e90 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.321+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 >> 172.21.15.87:0/972760885 conn(0x561b4a577cb0 msgr2=0x561b4a579a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.322+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 shutdown_connections 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.322+0000 7f1db5b8a740 1 -- 172.21.15.87:0/972760885 wait complete. 2024-04-22T13:42:43.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.322+0000 7f1db5b8a740 20 client.18903 trim_cache size 0 max 16384 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 Processor -- start 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 -- start start 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57d230 0x561b4a596bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1adfe0 con 0x561b4a57db50 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1ab8b0 con 0x561b4a58ae90 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1a9720 con 0x561b4a57d230 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43748/0 (socket says 172.21.15.87:43748) 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 -- 172.21.15.87:0/2948704311 learned_addr learned my addr 172.21.15.87:0/2948704311 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db1464640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db0c63640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57d230 0x561b4a596bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 -- 172.21.15.87:0/2948704311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57d230 msgr2=0x561b4a596bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57d230 0x561b4a596bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 -- 172.21.15.87:0/2948704311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 msgr2=0x561b4a588560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1dabfff640 1 -- 172.21.15.87:0/2948704311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b4a58dd70 con 0x561b4a57db50 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.340+0000 7f1db1464640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.341+0000 7f1dabfff640 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb60 tx=0x7f1d9800c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7274da76e193c90c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.341+0000 7f1d967fc640 1 -- 172.21.15.87:0/2948704311 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9802a3b0 con 0x561b4a57db50 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.341+0000 7f1d967fc640 1 -- 172.21.15.87:0/2948704311 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d98030070 con 0x561b4a57db50 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.341+0000 7f1d967fc640 1 -- 172.21.15.87:0/2948704311 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98010bf0 con 0x561b4a57db50 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- 172.21.15.87:0/2948704311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 msgr2=0x561b4a587770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800cb60 tx=0x7f1d9800c480 comp rx=0 tx=0).stop 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- 172.21.15.87:0/2948704311 shutdown_connections 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a58ae90 0x561b4a588560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a587770 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/2948704311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a57d230 0x561b4a596bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- 172.21.15.87:0/2948704311 >> 172.21.15.87:0/2948704311 conn(0x561b4a577cb0 msgr2=0x561b4a587e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- 172.21.15.87:0/2948704311 shutdown_connections 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- 172.21.15.87:0/2948704311 wait complete. 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- start start 2024-04-22T13:42:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 Processor -- start 2024-04-22T13:42:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a58eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a600420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1a9d70 con 0x561b4a57db50 2024-04-22T13:42:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1ab1e0 con 0x561b4a57d230 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac1aa440 con 0x561b4a58ae90 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1db5b8a740 10 client.0 _ll_register_callbacks cb 0x7f1da009bd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.342+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43758/0 (socket says 172.21.15.87:43758) 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 -- 172.21.15.87:0/1464156939 learned_addr learned my addr 172.21.15.87:0/1464156939 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db0c63640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a58eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db1464640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a600420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 -- 172.21.15.87:0/1464156939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 msgr2=0x561b4a600420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a600420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 -- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 msgr2=0x561b4a58eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a58eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 -- 172.21.15.87:0/1464156939 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b4a600960 con 0x561b4a57db50 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db0c63640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a58eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1dabfff640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800e760 tx=0x7f1d9800c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3901dad3ed847b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1d8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980050c0 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7f1d98005260 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d98030070 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1d98005460 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db5b8a740 10 client.18909 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561b4a601b10 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.343+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d98030070 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1d9802a4b0 con 0x561b4a57db50 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 10 client.18909 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 10 client.18909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 20 client.18909 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a604080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 4 client.18909 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b4a4e9a70 con 0x561b4a59ae20 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db0c63640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a604080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1d8bfff640 10 client.18909 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1db0c63640 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a604080 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7323a6a306955277 server_cookie=cc3eba4ca37a619b in_seq=0 out_seq=0 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1d8bfff640 1 -- 172.21.15.87:0/1464156939 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561b4a4e9a70 con 0x561b4a59ae20 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.344+0000 7f1d8bfff640 10 client.18909 handle_command_reply: tid=0 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 client.18909 shutdown 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 msgr2=0x561b4a58f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800e760 tx=0x7f1d9800c750 comp rx=0 tx=0).stop 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 shutdown_connections 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b4a59ae20 0x561b4a604080 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b4a58ae90 0x561b4a600420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b4a57db50 0x561b4a58f420 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1d8bfff640 0 client.18909 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 --2- 172.21.15.87:0/1464156939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b4a57d230 0x561b4a58eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 >> 172.21.15.87:0/1464156939 conn(0x561b4a577cb0 msgr2=0x561b4a576d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 shutdown_connections 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 1 -- 172.21.15.87:0/1464156939 wait complete. 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db5b8a740 20 client.18909 trim_cache size 0 max 16384 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "f9a3d2d7-6d91-4f8e-8396-e7c0925d472d": { 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "f9a3d2d7-6d91-4f8e-8396-e7c0925d472d", 2024-04-22T13:42:43.347 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:42:43.348 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:43.348 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:42:43.348 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:42:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 msgr2=0x7f1d9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 0x7f1d9c0a61e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1da4004af0 tx=0x7f1da4032ca0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 msgr2=0x7f1dac198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f1da000d950 tx=0x7f1da000de10 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 shutdown_connections 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d9c0a3d20 0x7f1d9c0a61e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dac19cf50 0x7f1dac19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dac106130 0x7f1dac198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 --2- 172.21.15.87:0/2808457720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1dac101060 0x7f1dac198690 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.345+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 >> 172.21.15.87:0/2808457720 conn(0x7f1dac0771d0 msgr2=0x7f1dac104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.346+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 shutdown_connections 2024-04-22T13:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:42:43.346+0000 7f1db2ec7640 1 -- 172.21.15.87:0/2808457720 wait complete. 2024-04-22T13:42:43.353 INFO:tasks.cephfs.filesystem:scrub status for tag:f9a3d2d7-6d91-4f8e-8396-e7c0925d472d - {'path': '/', 'tag': 'f9a3d2d7-6d91-4f8e-8396-e7c0925d472d', 'options': 'recursive,force'} 2024-04-22T13:42:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 -- 172.21.15.87:0/1333266163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806b120 msgr2=0x7fbb1810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 --2- 172.21.15.87:0/1333266163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806b120 0x7fbb1810ee40 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009ca0 tx=0x7fbb1002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 -- 172.21.15.87:0/1333266163 shutdown_connections 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 --2- 172.21.15.87:0/1333266163 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806b120 0x7fbb1810ee40 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 --2- 172.21.15.87:0/1333266163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb1806a760 0x7fbb1806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 --2- 172.21.15.87:0/1333266163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb18106f70 0x7fbb18107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 -- 172.21.15.87:0/1333266163 >> 172.21.15.87:0/1333266163 conn(0x7fbb18077dc0 msgr2=0x7fbb180781d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 -- 172.21.15.87:0/1333266163 shutdown_connections 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.408+0000 7fbb1d352640 1 -- 172.21.15.87:0/1333266163 wait complete. 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 Processor -- start 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 -- start start 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb1806b120 0x7fbb1819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb18106f70 0x7fbb18198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181121d0 con 0x7fbb1806a760 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb18112050 con 0x7fbb1806b120 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb1d352640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb18112350 con 0x7fbb18106f70 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43786/0 (socket says 172.21.15.87:43786) 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb16ffd640 1 -- 172.21.15.87:0/234714532 learned_addr learned my addr 172.21.15.87:0/234714532 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.409+0000 7fbb167fc640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb1806b120 0x7fbb1819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb177fe640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb18106f70 0x7fbb18198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 -- 172.21.15.87:0/234714532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb18106f70 msgr2=0x7fbb18198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb18106f70 0x7fbb18198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 -- 172.21.15.87:0/234714532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb1806b120 msgr2=0x7fbb1819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb1806b120 0x7fbb1819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 -- 172.21.15.87:0/234714532 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb18198db0 con 0x7fbb1806a760 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800d950 tx=0x7fbb0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbafffff640 1 -- 172.21.15.87:0/234714532 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08017030 con 0x7fbb1806a760 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb181990a0 con 0x7fbb1806a760 2024-04-22T13:42:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbafffff640 1 -- 172.21.15.87:0/234714532 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb08004020 con 0x7fbb1806a760 2024-04-22T13:42:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbafffff640 1 -- 172.21.15.87:0/234714532 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb080047a0 con 0x7fbb1806a760 2024-04-22T13:42:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.410+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb181a52f0 con 0x7fbb1806a760 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.412+0000 7fbafffff640 1 -- 172.21.15.87:0/234714532 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb08009a30 con 0x7fbb1806a760 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.412+0000 7fbb20029740 1 Processor -- start 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.412+0000 7fbb20029740 1 -- start start 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418dc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a8418318d0 con 0x55a8418cd8d0 2024-04-22T13:42:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a8417cee80 con 0x55a8418da4f0 2024-04-22T13:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb20029740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a84182d890 con 0x55a8418ccb60 2024-04-22T13:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbafffff640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 0x7fbb000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43796/0 (socket says 172.21.15.87:43796) 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 -- 172.21.15.87:0/3048693644 learned_addr learned my addr 172.21.15.87:0/3048693644 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb167fc640 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418dc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 0x7fbb000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbafffff640 1 -- 172.21.15.87:0/234714532 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fbb080d00c0 con 0x7fbb1806a760 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 -- 172.21.15.87:0/3048693644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 msgr2=0x55a8418ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 -- 172.21.15.87:0/3048693644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 msgr2=0x55a8418dc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418dc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 -- 172.21.15.87:0/3048693644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8417f0320 con 0x55a8418cd8d0 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbb10032460 tx=0x7fbb10032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49ffe7fb57e2988 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbb177fe640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 0x7fbb000a6280 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009a60 tx=0x7fbb10041000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.413+0000 7fbafd7fa640 1 -- 172.21.15.87:0/3048693644 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10051070 con 0x55a8418cd8d0 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbafd7fa640 1 -- 172.21.15.87:0/3048693644 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb10048070 con 0x55a8418cd8d0 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbafd7fa640 1 -- 172.21.15.87:0/3048693644 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10039040 con 0x55a8418cd8d0 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- 172.21.15.87:0/3048693644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 msgr2=0x55a8418cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbb10032460 tx=0x7fbb10032f40 comp rx=0 tx=0).stop 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- 172.21.15.87:0/3048693644 shutdown_connections 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418dc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 --2- 172.21.15.87:0/3048693644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418cc620 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- 172.21.15.87:0/3048693644 >> 172.21.15.87:0/3048693644 conn(0x55a841893c90 msgr2=0x55a8418c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- 172.21.15.87:0/3048693644 shutdown_connections 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- 172.21.15.87:0/3048693644 wait complete. 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.414+0000 7fbb20029740 1 -- start start 2024-04-22T13:42:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 Processor -- start 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418d9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418da4f0 0x55a8418d7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a8418dd1c0 con 0x55a8418cd8d0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a8417b9eb0 con 0x55a8418ccb60 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8418dd340 con 0x55a8418da4f0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 10 client.0 _ll_register_callbacks cb 0x55a8417e6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43812/0 (socket says 172.21.15.87:43812) 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 -- 172.21.15.87:0/1797253513 learned_addr learned my addr 172.21.15.87:0/1797253513 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418d9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 -- 172.21.15.87:0/1797253513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418da4f0 msgr2=0x55a8418d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb177fe640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418da4f0 0x55a8418d7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418da4f0 0x55a8418d7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 -- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 msgr2=0x55a8418d9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418d9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 -- 172.21.15.87:0/1797253513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a841544210 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418d9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb167fc640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c012700 tx=0x7fbb0c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3e91fccccc74e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbafaff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c004020 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fbb0c0041c0 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.415+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb10048070 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb0c0043c0 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 10 client.18921 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a841610620 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb10048070 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb0c003ae0 con 0x55a8418cd8d0 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 10 client.18921 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.421 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 10 client.18921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 20 client.18921 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8418ea960 0x55a8418ece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 4 client.18921 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a841845880 con 0x55a8418ea960 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.416+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8418ea960 0x55a8418ece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbafaff5640 10 client.18921 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8418ea960 0x55a8418ece00 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e764965b5ee1dc server_cookie=e0f1524982e11e9e in_seq=0 out_seq=0 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbafaff5640 1 -- 172.21.15.87:0/1797253513 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a841845880 con 0x55a8418ea960 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbafaff5640 10 client.18921 handle_command_reply: tid=0 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb20029740 1 client.18921 shutdown 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 msgr2=0x55a8418d71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c012700 tx=0x7fbb0c012bc0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 shutdown_connections 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.417+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8418ea960 0x55a8418ece00 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418da4f0 0x55a8418d7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d71e0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbafaff5640 0 client.18921 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 --2- 172.21.15.87:0/1797253513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418d9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 >> 172.21.15.87:0/1797253513 conn(0x55a841893c90 msgr2=0x55a8418c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 shutdown_connections 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 1 -- 172.21.15.87:0/1797253513 wait complete. 2024-04-22T13:42:43.423 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.418+0000 7fbb20029740 20 client.18921 trim_cache size 0 max 16384 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 Processor -- start 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 -- start start 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418d9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181ae120 con 0x55a8418cd8d0 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181ab9f0 con 0x55a8418da4f0 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb20029740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181a9860 con 0x55a8418ccb60 2024-04-22T13:42:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43816/0 (socket says 172.21.15.87:43816) 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 -- 172.21.15.87:0/526798562 learned_addr learned my addr 172.21.15.87:0/526798562 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418d9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb177fe640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 -- 172.21.15.87:0/526798562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 msgr2=0x55a8418d9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418d9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 -- 172.21.15.87:0/526798562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 msgr2=0x55a8418f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.432+0000 7fbb167fc640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb167fc640 1 -- 172.21.15.87:0/526798562 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8418cbe30 con 0x55a8418cd8d0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418d9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb167fc640 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009710 tx=0x7fbb0c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0f22f131452fe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbafcff9640 1 -- 172.21.15.87:0/526798562 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c029070 con 0x55a8418cd8d0 2024-04-22T13:42:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbafcff9640 1 -- 172.21.15.87:0/526798562 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb0c00fdd0 con 0x55a8418cd8d0 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbafcff9640 1 -- 172.21.15.87:0/526798562 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c003a50 con 0x55a8418cd8d0 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 -- 172.21.15.87:0/526798562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 msgr2=0x55a8418d9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009710 tx=0x7fbb0c00f850 comp rx=0 tx=0).stop 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 -- 172.21.15.87:0/526798562 shutdown_connections 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418da4f0 0x55a8418f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418cd8d0 0x55a8418d9e50 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 --2- 172.21.15.87:0/526798562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418ccb60 0x55a8418d9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 -- 172.21.15.87:0/526798562 >> 172.21.15.87:0/526798562 conn(0x55a841893c90 msgr2=0x55a8418d5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 -- 172.21.15.87:0/526798562 shutdown_connections 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.433+0000 7fbb20029740 1 -- 172.21.15.87:0/526798562 wait complete. 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 -- start start 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 Processor -- start 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.435 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418cd8d0 0x55a8418efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181a9eb0 con 0x55a8418da4f0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181ab320 con 0x55a8418ccb60 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181aa580 con 0x55a8418cd8d0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb20029740 10 client.0 _ll_register_callbacks cb 0x7fbb0c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43818/0 (socket says 172.21.15.87:43818) 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 -- 172.21.15.87:0/4049310556 learned_addr learned my addr 172.21.15.87:0/4049310556 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 -- 172.21.15.87:0/4049310556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418cd8d0 msgr2=0x55a8418efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb167fc640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418cd8d0 0x55a8418efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418cd8d0 0x55a8418efdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 -- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 msgr2=0x55a8418f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.434+0000 7fbb177fe640 1 -- 172.21.15.87:0/4049310556 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8418f0c00 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb177fe640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fbb1002c700 tx=0x7fbb10005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568c0022a88cdc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1005a060 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fbb10003970 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb0c00fdd0 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbb10051070 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 10 client.18927 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a841950f40 con 0x55a8418da4f0 2024-04-22T13:42:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb0c00fdd0 con 0x55a8418da4f0 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbb100044f0 con 0x55a8418da4f0 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 10 client.18927 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 10 client.18927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 20 client.18927 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8419512d0 0x55a8419536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 4 client.18927 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8417d13d0 con 0x55a8419512d0 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.435+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8419512d0 0x55a8419536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.436+0000 7fbafaff5640 10 client.18927 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.436+0000 7fbb16ffd640 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8419512d0 0x55a8419536c0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c79baafa5b87626 server_cookie=304c9f77abc0f3e2 in_seq=0 out_seq=0 2024-04-22T13:42:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbafaff5640 1 -- 172.21.15.87:0/4049310556 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a8417d13d0 con 0x55a8419512d0 2024-04-22T13:42:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbafaff5640 10 client.18927 handle_command_reply: tid=0 2024-04-22T13:42:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 client.18927 shutdown 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 msgr2=0x55a8418f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fbb1002c700 tx=0x7fbb10005d80 comp rx=0 tx=0).stop 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 shutdown_connections 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a8419512d0 0x55a8419536c0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a8418da4f0 0x55a8418f0400 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8418cd8d0 0x55a8418efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbafaff5640 0 client.18927 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 --2- 172.21.15.87:0/4049310556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8418ccb60 0x55a8418f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 >> 172.21.15.87:0/4049310556 conn(0x55a841893c90 msgr2=0x55a8418e5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 shutdown_connections 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 1 -- 172.21.15.87:0/4049310556 wait complete. 2024-04-22T13:42:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.441+0000 7fbb20029740 20 client.18927 trim_cache size 0 max 16384 2024-04-22T13:42:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:42:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:42:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:42:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 msgr2=0x7fbb000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 0x7fbb000a6280 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009a60 tx=0x7fbb10041000 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 msgr2=0x7fbb1819e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800d950 tx=0x7fbb0800de10 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 shutdown_connections 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbb000a3dc0 0x7fbb000a6280 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb18106f70 0x7fbb18198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb1806b120 0x7fbb1819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 --2- 172.21.15.87:0/234714532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb1806a760 0x7fbb1819e430 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 >> 172.21.15.87:0/234714532 conn(0x7fbb18077dc0 msgr2=0x7fbb180ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 shutdown_connections 2024-04-22T13:42:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:43.442+0000 7fbb1d352640 1 -- 172.21.15.87:0/234714532 wait complete. 2024-04-22T13:42:44.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:44 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:42:44.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:44 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:44.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:44 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:42:44.421 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:44 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:44.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:44 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:42:44.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:44 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:44.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:44 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:42:44.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:44 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:44.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:44 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:42:44.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:44 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:44.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:44 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:42:44.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:44 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:42:45.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:45 smithi120 ceph-mon[31101]: pgmap v340: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:45.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:45 smithi132 ceph-mon[31531]: pgmap v340: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:45.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:45 smithi087 ceph-mon[27999]: pgmap v340: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:42:46.387 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:46 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:42:46.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:46 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:42:46.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:46 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:42:47.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:47 smithi120 ceph-mon[31101]: pgmap v341: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:47.463 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:47 smithi132 ceph-mon[31531]: pgmap v341: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:47.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:47 smithi087 ceph-mon[27999]: pgmap v341: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T13:42:48.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:48 smithi120 ceph-mon[31101]: pgmap v342: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:48.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:48 smithi132 ceph-mon[31531]: pgmap v342: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:48.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:48 smithi087 ceph-mon[27999]: pgmap v342: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:50.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:50 smithi120 ceph-mon[31101]: pgmap v343: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:50.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:50 smithi132 ceph-mon[31531]: pgmap v343: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:50.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:50 smithi087 ceph-mon[27999]: pgmap v343: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:52.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:52 smithi120 ceph-mon[31101]: pgmap v344: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:52.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:52 smithi132 ceph-mon[31531]: pgmap v344: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:52.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:52 smithi087 ceph-mon[27999]: pgmap v344: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:55.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:55 smithi120 ceph-mon[31101]: pgmap v345: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:55.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:55 smithi132 ceph-mon[31531]: pgmap v345: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:55.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:55 smithi087 ceph-mon[27999]: pgmap v345: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:42:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- 172.21.15.87:0/51986990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb010e8a0 msgr2=0x7febb0110c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 --2- 172.21.15.87:0/51986990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb010e8a0 0x7febb0110c90 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7feba8009ca0 tx=0x7feba802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- 172.21.15.87:0/51986990 shutdown_connections 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 --2- 172.21.15.87:0/51986990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb010e8a0 0x7febb0110c90 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 --2- 172.21.15.87:0/51986990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb006acc0 0x7febb006b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 --2- 172.21.15.87:0/51986990 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb0102f00 0x7febb006a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- 172.21.15.87:0/51986990 >> 172.21.15.87:0/51986990 conn(0x7febb0076c50 msgr2=0x7febb0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- 172.21.15.87:0/51986990 shutdown_connections 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- 172.21.15.87:0/51986990 wait complete. 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 Processor -- start 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.538+0000 7febb5eb3640 1 -- start start 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb006acc0 0x7febb01a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb010e8a0 0x7febb019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7febb0114460 con 0x7febb0102f00 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7febb01142e0 con 0x7febb010e8a0 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7febb01145e0 con 0x7febb006acc0 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48928/0 (socket says 172.21.15.87:48928) 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 -- 172.21.15.87:0/3244647712 learned_addr learned my addr 172.21.15.87:0/3244647712 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaf7fe640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb006acc0 0x7febb01a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaffff640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb010e8a0 0x7febb019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 -- 172.21.15.87:0/3244647712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb006acc0 msgr2=0x7febb01a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb006acc0 0x7febb01a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 -- 172.21.15.87:0/3244647712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb010e8a0 msgr2=0x7febb019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb010e8a0 0x7febb019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 -- 172.21.15.87:0/3244647712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb019ae40 con 0x7febb0102f00 2024-04-22T13:42:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febaeffd640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7feba400e9f0 tx=0x7feba400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febacff9640 1 -- 172.21.15.87:0/3244647712 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba400ccc0 con 0x7febb0102f00 2024-04-22T13:42:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febacff9640 1 -- 172.21.15.87:0/3244647712 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba40108e0 con 0x7febb0102f00 2024-04-22T13:42:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb019b130 con 0x7febb0102f00 2024-04-22T13:42:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.539+0000 7febacff9640 1 -- 172.21.15.87:0/3244647712 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4018880 con 0x7febb0102f00 2024-04-22T13:42:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.540+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb01a7330 con 0x7febb0102f00 2024-04-22T13:42:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febacff9640 1 -- 172.21.15.87:0/3244647712 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feba4010430 con 0x7febb0102f00 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 Processor -- start 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 -- start start 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad51fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562c7acb68d0 con 0x562c7ad528d0 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562c7ac53e80 con 0x562c7ad51b60 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.542+0000 7febb8b80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562c7acb2890 con 0x562c7ad5b0e0 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febacff9640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 0x7feb980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48932/0 (socket says 172.21.15.87:48932) 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 -- 172.21.15.87:0/2198051024 learned_addr learned my addr 172.21.15.87:0/2198051024 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaf7fe640 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad51fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 0x7feb980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febacff9640 1 -- 172.21.15.87:0/3244647712 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7feba40d0b70 con 0x7febb0102f00 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaeffd640 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 -- 172.21.15.87:0/2198051024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 msgr2=0x562c7ad5d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 -- 172.21.15.87:0/2198051024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 msgr2=0x562c7ad51fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad51fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 -- 172.21.15.87:0/2198051024 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c7ac75320 con 0x562c7ad528d0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7feba8037040 tx=0x7feba8032540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77fd2bb73ccacb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7febaffff640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 0x7feb980a6280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feba8009ca0 tx=0x7feba8042000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7feb95ffb640 1 -- 172.21.15.87:0/2198051024 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8051070 con 0x562c7ad528d0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7feb95ffb640 1 -- 172.21.15.87:0/2198051024 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba8032970 con 0x562c7ad528d0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.543+0000 7feb95ffb640 1 -- 172.21.15.87:0/2198051024 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8039040 con 0x562c7ad528d0 2024-04-22T13:42:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- 172.21.15.87:0/2198051024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 msgr2=0x562c7ad51620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7feba8037040 tx=0x7feba8032540 comp rx=0 tx=0).stop 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- 172.21.15.87:0/2198051024 shutdown_connections 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad51fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 --2- 172.21.15.87:0/2198051024 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad51620 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- 172.21.15.87:0/2198051024 >> 172.21.15.87:0/2198051024 conn(0x562c7ad18c90 msgr2=0x562c7ad4c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- 172.21.15.87:0/2198051024 shutdown_connections 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- 172.21.15.87:0/2198051024 wait complete. 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.544+0000 7febb8b80740 1 -- start start 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 Processor -- start 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad5aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 0x562c7ad57e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x562c7ad5ddb0 con 0x562c7ad5b0e0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562c7ac3eeb0 con 0x562c7ad528d0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562c7ad5df30 con 0x562c7ad51b60 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febb8b80740 10 client.0 _ll_register_callbacks cb 0x562c7ac6bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48938/0 (socket says 172.21.15.87:48938) 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 -- 172.21.15.87:0/3986938427 learned_addr learned my addr 172.21.15.87:0/3986938427 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaf7fe640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad5aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaeffd640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 0x562c7ad57e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 -- 172.21.15.87:0/3986938427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 msgr2=0x562c7ad5aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad5aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 -- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 msgr2=0x562c7ad57e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 0x562c7ad57e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaffff640 1 -- 172.21.15.87:0/3986938427 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c7a9c9210 con 0x562c7ad5b0e0 2024-04-22T13:42:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.545+0000 7febaeffd640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 0x562c7ad57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febaffff640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7feba802c630 tx=0x7feba8004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80c37e174245c41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8039030 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7feba8004450 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feba8032970 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7feba8051040 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 10 client.18939 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562c7aa95620 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba8032970 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7feba804b030 con 0x562c7ad5b0e0 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 10 client.18939 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 10 client.18939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 20 client.18939 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7ad6af60 0x562c7ad6d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 4 client.18939 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.546+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c7acca880 con 0x562c7ad6af60 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.547+0000 7febaf7fe640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7ad6af60 0x562c7ad6d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.547+0000 7feb8effd640 10 client.18939 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.547+0000 7febaf7fe640 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7ad6af60 0x562c7ad6d350 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c590b9062ad562 server_cookie=ffba697c4fe10bf0 in_seq=0 out_seq=0 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7feb8effd640 1 -- 172.21.15.87:0/3986938427 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c7acca880 con 0x562c7ad6af60 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7feb8effd640 10 client.18939 handle_command_reply: tid=0 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 client.18939 shutdown 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 msgr2=0x562c7ad58480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7feba802c630 tx=0x7feba8004eb0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 shutdown_connections 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7ad6af60 0x562c7ad6d350 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad5b0e0 0x562c7ad58480 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad528d0 0x562c7ad57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 --2- 172.21.15.87:0/3986938427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad5aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7feb8effd640 0 client.18939 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 >> 172.21.15.87:0/3986938427 conn(0x562c7ad18c90 msgr2=0x562c7ad4e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 shutdown_connections 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 1 -- 172.21.15.87:0/3986938427 wait complete. 2024-04-22T13:42:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.548+0000 7febb8b80740 20 client.18939 trim_cache size 0 max 16384 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.563+0000 7febb8b80740 1 Processor -- start 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.563+0000 7febb8b80740 1 -- start start 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.563+0000 7febb8b80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad66cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.563+0000 7febb8b80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febb8b80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febb8b80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7febb01b2310 con 0x562c7ad528d0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febb8b80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7febb01afbe0 con 0x562c7ad5b0e0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febb8b80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7febb01adaa0 con 0x562c7ad51b60 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48950/0 (socket says 172.21.15.87:48950) 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 -- 172.21.15.87:0/941194834 learned_addr learned my addr 172.21.15.87:0/941194834 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaf7fe640 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad66cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaffff640 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 -- 172.21.15.87:0/941194834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 msgr2=0x562c7ad66cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad66cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 -- 172.21.15.87:0/941194834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad5b0e0 msgr2=0x562c7ad5a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 -- 172.21.15.87:0/941194834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c7ad50e30 con 0x562c7ad528d0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7febaeffd640 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7feba409cae0 tx=0x7feba40bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08a822e68aa2f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7feb957fa640 1 -- 172.21.15.87:0/941194834 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba40b6070 con 0x562c7ad528d0 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7feb957fa640 1 -- 172.21.15.87:0/941194834 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba409d250 con 0x562c7ad528d0 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.564+0000 7feb957fa640 1 -- 172.21.15.87:0/941194834 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4010a90 con 0x562c7ad528d0 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- 172.21.15.87:0/941194834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 msgr2=0x562c7ad4b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7feba409cae0 tx=0x7feba40bd040 comp rx=0 tx=0).stop 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- 172.21.15.87:0/941194834 shutdown_connections 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad5b0e0 0x562c7ad5a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad4b750 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 --2- 172.21.15.87:0/941194834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad51b60 0x562c7ad66cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- 172.21.15.87:0/941194834 >> 172.21.15.87:0/941194834 conn(0x562c7ad18c90 msgr2=0x562c7ad4b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- 172.21.15.87:0/941194834 shutdown_connections 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- 172.21.15.87:0/941194834 wait complete. 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.565+0000 7febb8b80740 1 -- start start 2024-04-22T13:42:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 Processor -- start 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad54b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad55580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7febb01ae0a0 con 0x562c7ad528d0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7febb01af510 con 0x562c7ad51b60 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7febb01ae770 con 0x562c7ad5b0e0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febb8b80740 10 client.0 _ll_register_callbacks cb 0x7feba804f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48954/0 (socket says 172.21.15.87:48954) 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 -- 172.21.15.87:0/2704194916 learned_addr learned my addr 172.21.15.87:0/2704194916 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaffff640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad55580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaf7fe640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad54b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 -- 172.21.15.87:0/2704194916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 msgr2=0x562c7ad55580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad55580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 -- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 msgr2=0x562c7ad54b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad54b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 -- 172.21.15.87:0/2704194916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c7ad47f30 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7febaeffd640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7feba409c850 tx=0x7feba40d0e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98eb79283313279c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7feb8effd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.566+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba40b6040 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7feba4010a90 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feba409d250 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7feba40a0f00 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 10 client.18945 fetch_fsmap learned FSMap version 23 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562c7add12e0 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba409d250 con 0x562c7ad528d0 2024-04-22T13:42:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7feba409ba10 con 0x562c7ad528d0 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 10 client.18945 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 10 client.18945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 20 client.18945 populate_metadata read hostname 'smithi087' 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7add1980 0x562c7add3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 4 client.18945 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c7ac563d0 con 0x562c7add1980 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.567+0000 7febaf7fe640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7add1980 0x562c7add3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.568+0000 7feb8effd640 10 client.18945 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.568+0000 7febaf7fe640 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7add1980 0x562c7add3da0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=397deb958df457fc server_cookie=f10f8c24a338f974 in_seq=0 out_seq=0 2024-04-22T13:42:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7feb8effd640 1 -- 172.21.15.87:0/2704194916 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c7ac563d0 con 0x562c7add1980 2024-04-22T13:42:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7feb8effd640 10 client.18945 handle_command_reply: tid=0 2024-04-22T13:42:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 client.18945 shutdown 2024-04-22T13:42:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 msgr2=0x562c7ad55040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7feba409c850 tx=0x7feba40d0e70 comp rx=0 tx=0).stop 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 shutdown_connections 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x562c7add1980 0x562c7add3da0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c7ad5b0e0 0x562c7ad55580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562c7ad528d0 0x562c7ad55040 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 --2- 172.21.15.87:0/2704194916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562c7ad51b60 0x562c7ad54b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7feb8effd640 0 client.18945 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 >> 172.21.15.87:0/2704194916 conn(0x562c7ad18c90 msgr2=0x562c7ad4dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 shutdown_connections 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 1 -- 172.21.15.87:0/2704194916 wait complete. 2024-04-22T13:42:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.573+0000 7febb8b80740 20 client.18945 trim_cache size 0 max 16384 2024-04-22T13:42:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:42:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:42:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:42:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 msgr2=0x7feb980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 0x7feb980a6280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feba8009ca0 tx=0x7feba8042000 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 msgr2=0x7febb01a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7feba400e9f0 tx=0x7feba400edf0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 shutdown_connections 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7feb980a3dc0 0x7feb980a6280 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febb010e8a0 0x7febb019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febb0102f00 0x7febb01a09b0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 --2- 172.21.15.87:0/3244647712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7febb006acc0 0x7febb01a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 >> 172.21.15.87:0/3244647712 conn(0x7febb0076c50 msgr2=0x7febb0100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 shutdown_connections 2024-04-22T13:42:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:42:56.574+0000 7febb5eb3640 1 -- 172.21.15.87:0/3244647712 wait complete. 2024-04-22T13:42:57.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:57 smithi120 ceph-mon[31101]: pgmap v346: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:57.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:57 smithi132 ceph-mon[31531]: pgmap v346: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:57.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:57 smithi087 ceph-mon[27999]: pgmap v346: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T13:42:59.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:42:59 smithi120 ceph-mon[31101]: pgmap v347: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:59.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:42:59 smithi132 ceph-mon[31531]: pgmap v347: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:42:59.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:42:59 smithi087 ceph-mon[27999]: pgmap v347: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T13:43:01.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:01 smithi120 ceph-mon[31101]: pgmap v348: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:01.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:01 smithi132 ceph-mon[31531]: pgmap v348: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:01.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:01 smithi087 ceph-mon[27999]: pgmap v348: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:01.762 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.7 2024-04-22T13:43:01.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 -- 172.21.15.87:0/1466277493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e810e810 msgr2=0x7fd7e8110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 --2- 172.21.15.87:0/1466277493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e810e810 0x7fd7e8110c00 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009ca0 tx=0x7fd7d802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 -- 172.21.15.87:0/1466277493 shutdown_connections 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 --2- 172.21.15.87:0/1466277493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e810e810 0x7fd7e8110c00 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 --2- 172.21.15.87:0/1466277493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8069620 0x7fd7e8069a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 --2- 172.21.15.87:0/1466277493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e8102e80 0x7fd7e80690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 -- 172.21.15.87:0/1466277493 >> 172.21.15.87:0/1466277493 conn(0x7fd7e8077e30 msgr2=0x7fd7e8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 -- 172.21.15.87:0/1466277493 shutdown_connections 2024-04-22T13:43:01.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 -- 172.21.15.87:0/1466277493 wait complete. 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.877+0000 7fd7ed140640 1 Processor -- start 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 -- start start 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 0x7fd7e81a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 0x7fd7e819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8114390 con 0x7fd7e8069620 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8114210 con 0x7fd7e810e810 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7ed140640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8114510 con 0x7fd7e8102e80 2024-04-22T13:43:01.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39116/0 (socket says 172.21.15.87:39116) 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 -- 172.21.15.87:0/2209004486 learned_addr learned my addr 172.21.15.87:0/2209004486 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e659c640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 0x7fd7e81a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e759e640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 0x7fd7e819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 -- 172.21.15.87:0/2209004486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 msgr2=0x7fd7e81a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 0x7fd7e81a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 -- 172.21.15.87:0/2209004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 msgr2=0x7fd7e819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 0x7fd7e819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e6d9d640 1 -- 172.21.15.87:0/2209004486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e819af30 con 0x7fd7e8069620 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e659c640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 0x7fd7e81a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.878+0000 7fd7e759e640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 0x7fd7e819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7e6d9d640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c890 tx=0x7fd7e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0004220 con 0x7fd7e8069620 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7e00043c0 con 0x7fd7e8069620 2024-04-22T13:43:01.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e819b220 con 0x7fd7e8069620 2024-04-22T13:43:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0004a00 con 0x7fd7e8069620 2024-04-22T13:43:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.879+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e81a7530 con 0x7fd7e8069620 2024-04-22T13:43:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.880+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e8101430 con 0x7fd7e8069620 2024-04-22T13:43:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.881+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd7e00079b0 con 0x7fd7e8069620 2024-04-22T13:43:01.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.882+0000 7fd7dffff640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 0x7fd7d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.882+0000 7fd7e659c640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 0x7fd7d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.882+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fd7e00d09b0 con 0x7fd7e8069620 2024-04-22T13:43:01.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.882+0000 7fd7e659c640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 0x7fd7d40a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd7d00058c0 tx=0x7fd7d0005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:01.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:01.885+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7e0014030 con 0x7fd7e8069620 2024-04-22T13:43:02.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.103+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7e81a7700 con 0x7fd7e8069620 2024-04-22T13:43:02.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.103+0000 7fd7dffff640 1 -- 172.21.15.87:0/2209004486 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fd7e009d090 con 0x7fd7e8069620 2024-04-22T13:43:02.099 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:43:02.100 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:43:02.100 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:43:02.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 msgr2=0x7fd7d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 0x7fd7d40a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd7d00058c0 tx=0x7fd7d0005850 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 msgr2=0x7fd7e81a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c890 tx=0x7fd7e000cd50 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 shutdown_connections 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd7d40a3d20 0x7fd7d40a61e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7e810e810 0x7fd7e819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7e8102e80 0x7fd7e81a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 --2- 172.21.15.87:0/2209004486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd7e8069620 0x7fd7e81a05b0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.105+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 >> 172.21.15.87:0/2209004486 conn(0x7fd7e8077e30 msgr2=0x7fd7e810f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.106+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 shutdown_connections 2024-04-22T13:43:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.106+0000 7fd7ed140640 1 -- 172.21.15.87:0/2209004486 wait complete. 2024-04-22T13:43:02.112 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db / --timeout 60 --expiration 79.7 --await' 2024-04-22T13:43:02.113 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db / --timeout 60 --expiration 79.7 --await 2024-04-22T13:43:02.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 -- 172.21.15.87:0/3514198287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 msgr2=0x7fada010bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 --2- 172.21.15.87:0/3514198287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada010bfb0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fad94009900 tx=0x7fad9402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 -- 172.21.15.87:0/3514198287 shutdown_connections 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 --2- 172.21.15.87:0/3514198287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada010c580 0x7fada010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 --2- 172.21.15.87:0/3514198287 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada010bfb0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 --2- 172.21.15.87:0/3514198287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada01031b0 0x7fada0103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 -- 172.21.15.87:0/3514198287 >> 172.21.15.87:0/3514198287 conn(0x7fada00771d0 msgr2=0x7fada00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 -- 172.21.15.87:0/3514198287 shutdown_connections 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.223+0000 7fada485c640 1 -- 172.21.15.87:0/3514198287 wait complete. 2024-04-22T13:43:02.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 Processor -- start 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 -- start start 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada01031b0 0x7fada019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada010c580 0x7fada01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fada01121a0 con 0x7fada0103ad0 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fada0112020 con 0x7fada01031b0 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fada485c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fada0112320 con 0x7fada010c580 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39128/0 (socket says 172.21.15.87:39128) 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 -- 172.21.15.87:0/1800890492 learned_addr learned my addr 172.21.15.87:0/1800890492 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9e59c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada01031b0 0x7fada019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada010c580 0x7fada01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 -- 172.21.15.87:0/1800890492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada010c580 msgr2=0x7fada01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada010c580 0x7fada01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 -- 172.21.15.87:0/1800890492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada01031b0 msgr2=0x7fada019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada01031b0 0x7fada019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.224+0000 7fad9dd9b640 1 -- 172.21.15.87:0/1800890492 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada0198d30 con 0x7fada0103ad0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fad94005940 tx=0x7fad9402d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fad8f7fe640 1 -- 172.21.15.87:0/1800890492 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94043020 con 0x7fada0103ad0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada0198fc0 con 0x7fada0103ad0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fad8f7fe640 1 -- 172.21.15.87:0/1800890492 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad9402dc90 con 0x7fada0103ad0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fad8f7fe640 1 -- 172.21.15.87:0/1800890492 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94003ab0 con 0x7fada0103ad0 2024-04-22T13:43:02.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.225+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada01a52c0 con 0x7fada0103ad0 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.227+0000 7fad8f7fe640 1 -- 172.21.15.87:0/1800890492 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad94046020 con 0x7fada0103ad0 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 Processor -- start 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 -- start start 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a660ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a6613980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a66162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5612a656b0b0 con 0x5612a6613ec0 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612a651c680 con 0x5612a660b2e0 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fada7450740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5612a6528c60 con 0x5612a660a9c0 2024-04-22T13:43:02.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad8f7fe640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 0x7fad880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a66162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a66162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39130/0 (socket says 172.21.15.87:39130) 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9dd9b640 1 -- 172.21.15.87:0/753288245 learned_addr learned my addr 172.21.15.87:0/753288245 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9e59c640 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a6613980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 0x7fad880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a660ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.228+0000 7fad8f7fe640 1 -- 172.21.15.87:0/1800890492 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fad940f2810 con 0x7fada0103ad0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 -- 172.21.15.87:0/753288245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 msgr2=0x5612a660ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a660ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 -- 172.21.15.87:0/753288245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 msgr2=0x5612a66162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a66162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 -- 172.21.15.87:0/753288245 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a634dc90 con 0x5612a660b2e0 2024-04-22T13:43:02.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 0x7fad880a61e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad98005c40 tx=0x7fad98009c90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad9e59c640 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a6613980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad9000bbf0 tx=0x7fad900103d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c35d9ff96201720 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad8cff9640 1 -- 172.21.15.87:0/753288245 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad90004030 con 0x5612a660b2e0 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad8cff9640 1 -- 172.21.15.87:0/753288245 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad900035f0 con 0x5612a660b2e0 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.229+0000 7fad8cff9640 1 -- 172.21.15.87:0/753288245 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad900116a0 con 0x5612a660b2e0 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- 172.21.15.87:0/753288245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 msgr2=0x5612a6613980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a6613980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad9000bbf0 tx=0x7fad900103d0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- 172.21.15.87:0/753288245 shutdown_connections 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a66162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a6613980 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- 172.21.15.87:0/753288245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a660ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- 172.21.15.87:0/753288245 >> 172.21.15.87:0/753288245 conn(0x5612a66050f0 msgr2=0x5612a66054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- 172.21.15.87:0/753288245 shutdown_connections 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- 172.21.15.87:0/753288245 wait complete. 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- start start 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 Processor -- start 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a6610e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a66118f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5612a656e400 con 0x5612a660b2e0 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612a652a840 con 0x5612a6613ec0 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.230+0000 7fada7450740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5612a6573720 con 0x5612a660a9c0 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fada7450740 10 client.0 _ll_register_callbacks cb 0x5612a657bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:02.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39132/0 (socket says 172.21.15.87:39132) 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2724100690 learned_addr learned my addr 172.21.15.87:0/2724100690 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a66118f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9e59c640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a6610e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2724100690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 msgr2=0x5612a6610e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a6610e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 msgr2=0x5612a66118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a66118f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2724100690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a63ae420 con 0x5612a660b2e0 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9e59c640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a6610e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fad94006010 tx=0x7fad9402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2ee3d72e25e4da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940d3e60 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fad94003c50 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad900035f0 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fad94003e50 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fada7450740 10 client.18963 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5612a6241580 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.231+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad900035f0 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fad940c70e0 con 0x5612a660b2e0 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 10 client.18963 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 10 client.18963 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:02.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 20 client.18963 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a6623ab0 0x5612a6625f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 4 client.18963 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a657cb10 con 0x5612a6623ab0 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fad9e59c640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a6623ab0 0x5612a6625f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fad85ffb640 10 client.18963 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.232+0000 7fad9e59c640 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a6623ab0 0x5612a6625f50 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c70cab2506de26 server_cookie=b0859a7564ab88b9 in_seq=0 out_seq=0 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fad85ffb640 1 -- 172.21.15.87:0/2724100690 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612a657cb10 con 0x5612a6623ab0 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fad85ffb640 10 client.18963 handle_command_reply: tid=0 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 client.18963 shutdown 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 msgr2=0x5612a66113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fad94006010 tx=0x7fad9402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 shutdown_connections 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a6623ab0 0x5612a6625f50 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a66118f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fad85ffb640 0 client.18963 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a66113b0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 --2- 172.21.15.87:0/2724100690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a6610e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.233+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 >> 172.21.15.87:0/2724100690 conn(0x5612a66050f0 msgr2=0x5612a6606e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.234+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 shutdown_connections 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.234+0000 7fada7450740 1 -- 172.21.15.87:0/2724100690 wait complete. 2024-04-22T13:43:02.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.234+0000 7fada7450740 20 client.18963 trim_cache size 0 max 16384 2024-04-22T13:43:02.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 Processor -- start 2024-04-22T13:43:02.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 -- start start 2024-04-22T13:43:02.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a66135a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a662b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fada01adc30 con 0x5612a660b2e0 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fada01abaa0 con 0x5612a6613ec0 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fada7450740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fada01ad560 con 0x5612a660a9c0 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39136/0 (socket says 172.21.15.87:39136) 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2630844637 learned_addr learned my addr 172.21.15.87:0/2630844637 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9e59c640 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a66135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2630844637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 msgr2=0x5612a66135a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a66135a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2630844637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 msgr2=0x5612a662b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a662b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 -- 172.21.15.87:0/2630844637 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a6604f20 con 0x5612a660b2e0 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fad940be600 tx=0x7fad94000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360be812b733893b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad87fff640 1 -- 172.21.15.87:0/2630844637 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940be440 con 0x5612a660b2e0 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad87fff640 1 -- 172.21.15.87:0/2630844637 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad94088070 con 0x5612a660b2e0 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.251+0000 7fad87fff640 1 -- 172.21.15.87:0/2630844637 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940be900 con 0x5612a660b2e0 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- 172.21.15.87:0/2630844637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 msgr2=0x5612a661f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fad940be600 tx=0x7fad94000bc0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- 172.21.15.87:0/2630844637 shutdown_connections 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a6613ec0 0x5612a662b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a660b2e0 0x5612a661f8a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 --2- 172.21.15.87:0/2630844637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a66135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- 172.21.15.87:0/2630844637 >> 172.21.15.87:0/2630844637 conn(0x5612a66050f0 msgr2=0x5612a6606dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- 172.21.15.87:0/2630844637 shutdown_connections 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- 172.21.15.87:0/2630844637 wait complete. 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 -- start start 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.252+0000 7fada7450740 1 Processor -- start 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a661bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a661c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fada01ac7c0 con 0x5612a6613ec0 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fada01ac0f0 con 0x5612a660b2e0 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fada01ace90 con 0x5612a660a9c0 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 10 client.0 _ll_register_callbacks cb 0x7fad94009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39148/0 (socket says 172.21.15.87:39148) 2024-04-22T13:43:02.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 -- 172.21.15.87:0/587984652 learned_addr learned my addr 172.21.15.87:0/587984652 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9e59c640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a661bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9dd9b640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a661c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 -- 172.21.15.87:0/587984652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 msgr2=0x5612a661bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a661bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 -- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 msgr2=0x5612a661c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a661c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 -- 172.21.15.87:0/587984652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a6619710 con 0x5612a6613ec0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad9ed9d640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fad98010c60 tx=0x7fad98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d561a71126bb6947 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad98010e10 con 0x5612a6613ec0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8455+0+0 (secure 0 0 0) 0x7fad9801eb00 con 0x5612a6613ec0 2024-04-22T13:43:02.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad94088070 con 0x5612a6613ec0 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fad9801ed00 con 0x5612a6613ec0 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 10 client.18969 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.253+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5612a668acf0 con 0x5612a6613ec0 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad94088070 con 0x5612a6613ec0 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fad9801f070 con 0x5612a6613ec0 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 10 client.18969 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 10 client.18969 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 20 client.18969 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a668b510 0x5612a668d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 4 client.18969 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await": true}] 2024-04-22T13:43:02.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.7, "await": true}) -- 0x5612a6576980 con 0x5612a668b510 2024-04-22T13:43:02.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fad9e59c640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a668b510 0x5612a668d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:02.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fad9e59c640 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a668b510 0x5612a668d900 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868634580f967ef8 server_cookie=27459439f4f73259 in_seq=0 out_seq=0 2024-04-22T13:43:02.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.254+0000 7fad85ffb640 10 client.18969 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:02.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.265+0000 7fad85ffb640 1 -- 172.21.15.87:0/587984652 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5612a6576980 con 0x5612a668b510 2024-04-22T13:43:02.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.265+0000 7fad85ffb640 10 client.18969 handle_command_reply: tid=0 2024-04-22T13:43:02.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 client.18969 shutdown 2024-04-22T13:43:02.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 msgr2=0x5612a6618ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fad98010c60 tx=0x7fad98004a40 comp rx=0 tx=0).stop 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 shutdown_connections 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5612a668b510 0x5612a668d900 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5612a6613ec0 0x5612a6618ee0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fad85ffb640 0 client.18969 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612a660b2e0 0x5612a661c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 --2- 172.21.15.87:0/587984652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a660a9c0 0x5612a661bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 >> 172.21.15.87:0/587984652 conn(0x5612a66050f0 msgr2=0x5612a6606540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 shutdown_connections 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 1 -- 172.21.15.87:0/587984652 wait complete. 2024-04-22T13:43:02.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada7450740 20 client.18969 trim_cache size 0 max 16384 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 42, 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "b582c357": { 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 42, 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:43:02.264 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 79.7, 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "QUIESCED", 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:02.265 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:02.266 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:02.266 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 msgr2=0x7fad880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 0x7fad880a61e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad98005c40 tx=0x7fad98009c90 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 msgr2=0x7fada019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fad94005940 tx=0x7fad9402d890 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 shutdown_connections 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fad880a3d20 0x7fad880a61e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fada010c580 0x7fada01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fada0103ad0 0x7fada019e940 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 --2- 172.21.15.87:0/1800890492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada01031b0 0x7fada019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.266+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 >> 172.21.15.87:0/1800890492 conn(0x7fada00771d0 msgr2=0x7fada00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.267+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 shutdown_connections 2024-04-22T13:43:02.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:02.267+0000 7fada485c640 1 -- 172.21.15.87:0/1800890492 wait complete. 2024-04-22T13:43:02.273 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: b582c357, quiesce took 0.5 seconds. Will release after: 19.7 2024-04-22T13:43:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:02 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2209004486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:02.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:02 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2209004486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:02.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:02 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2209004486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:03.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:03 smithi120 ceph-mon[31101]: pgmap v349: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:03.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:03 smithi132 ceph-mon[31531]: pgmap v349: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:03.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:03 smithi087 ceph-mon[27999]: pgmap v349: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:05.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:05 smithi120 ceph-mon[31101]: pgmap v350: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:43:05.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:05 smithi132 ceph-mon[31531]: pgmap v350: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:43:05.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:05 smithi087 ceph-mon[27999]: pgmap v350: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T13:43:07.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:07 smithi120 ceph-mon[31101]: pgmap v351: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:43:07.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:07 smithi120 ceph-mon[31101]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T13:43:07.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:07 smithi132 ceph-mon[31531]: pgmap v351: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:43:07.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:07 smithi132 ceph-mon[31531]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T13:43:07.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:07 smithi087 ceph-mon[27999]: pgmap v351: 97 pgs: 97 active+clean; 22 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T13:43:07.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:07 smithi087 ceph-mon[27999]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T13:43:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/242728415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 msgr2=0x7f0c741014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/242728415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741014d0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009ca0 tx=0x7f0c6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/242728415 shutdown_connections 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/242728415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74106170 0x7f0c7410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/242728415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74101aa0 0x7f0c74105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/242728415 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741014d0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/242728415 >> 172.21.15.87:0/242728415 conn(0x7f0c74077fd0 msgr2=0x7f0c740783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/242728415 shutdown_connections 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.669+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/242728415 wait complete. 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 Processor -- start 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 -- start start 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 0x7f0c74199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 0x7f0c741a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741110b0 con 0x7f0c741010f0 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74110f30 con 0x7f0c74106170 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7bd9f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74111230 con 0x7f0c74199fc0 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53794/0 (socket says 172.21.15.87:53794) 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3529368398 learned_addr learned my addr 172.21.15.87:0/3529368398 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 0x7f0c74199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3529368398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 msgr2=0x7f0c741a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 0x7f0c741a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 0x7f0c741a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3529368398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 msgr2=0x7f0c74199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 0x7f0c74199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3529368398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c741a4d40 con 0x7f0c741010f0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 0x7f0c74199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.670+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 0x7f0c741a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c004af0 tx=0x7f0c6c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c6affd640 1 -- 172.21.15.87:0/3529368398 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c041020 con 0x7f0c741010f0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c6affd640 1 -- 172.21.15.87:0/3529368398 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6c032970 con 0x7f0c741010f0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c6affd640 1 -- 172.21.15.87:0/3529368398 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c039040 con 0x7f0c741010f0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c741a4fd0 con 0x7f0c741010f0 2024-04-22T13:43:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.671+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c741a5430 con 0x7f0c741010f0 2024-04-22T13:43:09.671 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:09 smithi120 ceph-mon[31101]: pgmap v353: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.673+0000 7f0c6affd640 1 -- 172.21.15.87:0/3529368398 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c6c044020 con 0x7f0c741010f0 2024-04-22T13:43:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 Processor -- start 2024-04-22T13:43:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 -- start start 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4a7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4a7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561b7e40c8d0 con 0x561b7e4a7b60 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b7e3a9e80 con 0x561b7e4b10e0 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561b7e408890 con 0x561b7e4a88d0 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c6affd640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 0x7f0c600a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4a7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4a7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53796/0 (socket says 172.21.15.87:53796) 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 0x7f0c600a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c79b3b640 1 -- 172.21.15.87:0/10950564 learned_addr learned my addr 172.21.15.87:0/10950564 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7933a640 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.674+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4a7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c6affd640 1 -- 172.21.15.87:0/3529368398 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f0c6c0f2460 con 0x7f0c741010f0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 -- 172.21.15.87:0/10950564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 msgr2=0x561b7e4a7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4a7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 -- 172.21.15.87:0/10950564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 msgr2=0x561b7e4a7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4a7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 -- 172.21.15.87:0/10950564 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7e3cb320 con 0x561b7e4b10e0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 0x7f0c600a6280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009460 tx=0x7f0c70009a10 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c7933a640 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4b34d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c7406b730 tx=0x7f0c64010370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e29618024dbb3b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c5ffff640 1 -- 172.21.15.87:0/10950564 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64010da0 con 0x561b7e4b10e0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c5ffff640 1 -- 172.21.15.87:0/10950564 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c640035f0 con 0x561b7e4b10e0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.675+0000 7f0c5ffff640 1 -- 172.21.15.87:0/10950564 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c640126b0 con 0x561b7e4b10e0 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- 172.21.15.87:0/10950564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 msgr2=0x561b7e4b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4b34d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0c7406b730 tx=0x7f0c64010370 comp rx=0 tx=0).stop 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- 172.21.15.87:0/10950564 shutdown_connections 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4b34d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4a7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/10950564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- 172.21.15.87:0/10950564 >> 172.21.15.87:0/10950564 conn(0x561b7e46ec90 msgr2=0x561b7e4a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- 172.21.15.87:0/10950564 shutdown_connections 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- 172.21.15.87:0/10950564 wait complete. 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.676+0000 7f0c7e98a740 1 -- start start 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 Processor -- start 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561b7e4b3e10 con 0x561b7e4a88d0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b7e394eb0 con 0x561b7e4b10e0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561b7e4b3f90 con 0x561b7e4a7b60 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7e98a740 10 client.0 _ll_register_callbacks cb 0x561b7e3c1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53810/0 (socket says 172.21.15.87:53810) 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7a33c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 -- 172.21.15.87:0/3519703010 learned_addr learned my addr 172.21.15.87:0/3519703010 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 -- 172.21.15.87:0/3519703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 msgr2=0x561b7e4b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 -- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 msgr2=0x561b7e4ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7933a640 1 -- 172.21.15.87:0/3519703010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7e11f210 con 0x561b7e4a88d0 2024-04-22T13:43:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.677+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f0c64000c00 tx=0x7f0c640039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e01d5adfce124e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64029060 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f0c640041c0 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c640035f0 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0c64009850 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 10 client.18981 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561b7e1eb620 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c640035f0 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0c6400f040 con 0x561b7e4a88d0 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 10 client.18981 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 10 client.18981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 20 client.18981 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e4c1010 0x561b7e4c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 4 client.18981 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.678+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b7e420880 con 0x561b7e4c1010 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.679+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e4c1010 0x561b7e4c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.679+0000 7f0c5d7fa640 10 client.18981 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.679+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e4c1010 0x561b7e4c3400 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eba58949d32efc7 server_cookie=bd6bd3abee7f0335 in_seq=0 out_seq=0 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3519703010 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b7e420880 con 0x561b7e4c1010 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c5d7fa640 10 client.18981 handle_command_reply: tid=0 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 client.18981 shutdown 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 msgr2=0x561b7e4ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f0c64000c00 tx=0x7f0c640039d0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 shutdown_connections 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e4c1010 0x561b7e4c3400 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a88d0 0x561b7e4ade10 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3519703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c5d7fa640 0 client.18981 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 >> 172.21.15.87:0/3519703010 conn(0x561b7e46ec90 msgr2=0x561b7e4a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 shutdown_connections 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3519703010 wait complete. 2024-04-22T13:43:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.680+0000 7f0c7e98a740 20 client.18981 trim_cache size 0 max 16384 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 Processor -- start 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 -- start start 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4a88d0 0x561b7e4b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741b03c0 con 0x561b7e4b10e0 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741adc90 con 0x561b7e4a88d0 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741abb00 con 0x561b7e4a7b60 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53812/0 (socket says 172.21.15.87:53812) 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 -- 172.21.15.87:0/1315636543 learned_addr learned my addr 172.21.15.87:0/1315636543 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7933a640 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4a88d0 0x561b7e4b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 -- 172.21.15.87:0/1315636543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 msgr2=0x561b7e4bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 -- 172.21.15.87:0/1315636543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4a88d0 msgr2=0x561b7e4b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4a88d0 0x561b7e4b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.695+0000 7f0c7a33c640 1 -- 172.21.15.87:0/1315636543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7e4a6e30 con 0x561b7e4b10e0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0c70014440 tx=0x7f0c7001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60a71de774d9454 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c5f7fe640 1 -- 172.21.15.87:0/1315636543 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70004230 con 0x561b7e4b10e0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c5f7fe640 1 -- 172.21.15.87:0/1315636543 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c70011e70 con 0x561b7e4b10e0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c5f7fe640 1 -- 172.21.15.87:0/1315636543 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70003b90 con 0x561b7e4b10e0 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 -- 172.21.15.87:0/1315636543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 msgr2=0x561b7e4b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f0c70014440 tx=0x7f0c7001e000 comp rx=0 tx=0).stop 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 -- 172.21.15.87:0/1315636543 shutdown_connections 2024-04-22T13:43:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4b10e0 0x561b7e4b0a80 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4a88d0 0x561b7e4b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/1315636543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a7b60 0x561b7e4bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 -- 172.21.15.87:0/1315636543 >> 172.21.15.87:0/1315636543 conn(0x561b7e46ec90 msgr2=0x561b7e4a2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 -- 172.21.15.87:0/1315636543 shutdown_connections 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.696+0000 7f0c7e98a740 1 -- 172.21.15.87:0/1315636543 wait complete. 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 -- start start 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 Processor -- start 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4b9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4aa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ac150 con 0x561b7e4a7b60 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ad5c0 con 0x561b7e4b10e0 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741ac820 con 0x561b7e4a88d0 2024-04-22T13:43:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c7e98a740 10 client.0 _ll_register_callbacks cb 0x7f0c70009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c79b3b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53824/0 (socket says 172.21.15.87:53824) 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.697+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3550684144 learned_addr learned my addr 172.21.15.87:0/3550684144 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c7a33c640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3550684144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 msgr2=0x561b7e4b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 msgr2=0x561b7e4aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4aa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 -- 172.21.15.87:0/3550684144 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7e4aaa50 con 0x561b7e4a7b60 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c79b3b640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0f01c0 tx=0x7f0c6c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc95940f0e05d88 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0dee60 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f0c6c0efe40 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c70011e70 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0c6c0bec20 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c7e98a740 10 client.18987 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561b7e4abdc0 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.698+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c70011e70 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0c6c03c030 con 0x561b7e4a7b60 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 10 client.18987 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 10 client.18987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 20 client.18987 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e528610 0x561b7e52aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 4 client.18987 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b7e3ac3d0 con 0x561b7e528610 2024-04-22T13:43:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e528610 0x561b7e52aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c5d7fa640 10 client.18987 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.699+0000 7f0c7933a640 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e528610 0x561b7e52aa00 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ba058c909f439b server_cookie=64915ee475960af4 in_seq=0 out_seq=0 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c5d7fa640 1 -- 172.21.15.87:0/3550684144 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b7e3ac3d0 con 0x561b7e528610 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c5d7fa640 10 client.18987 handle_command_reply: tid=0 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 client.18987 shutdown 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 msgr2=0x561b7e4b9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0f01c0 tx=0x7f0c6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 shutdown_connections 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561b7e528610 0x561b7e52aa00 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b7e4b10e0 0x561b7e4aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561b7e4a88d0 0x561b7e4b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c5d7fa640 0 client.18987 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 --2- 172.21.15.87:0/3550684144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561b7e4a7b60 0x561b7e4b9250 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 >> 172.21.15.87:0/3550684144 conn(0x561b7e46ec90 msgr2=0x561b7e4a1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 shutdown_connections 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.707+0000 7f0c7e98a740 1 -- 172.21.15.87:0/3550684144 wait complete. 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7e98a740 20 client.18987 trim_cache size 0 max 16384 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:43:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 msgr2=0x7f0c600a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 0x7f0c600a6280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009460 tx=0x7f0c70009a10 comp rx=0 tx=0).stop 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 msgr2=0x7f0c741994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c004af0 tx=0x7f0c6c0346d0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 shutdown_connections 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f0c600a3dc0 0x7f0c600a6280 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c74199fc0 0x7f0c741a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c74106170 0x7f0c74199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 --2- 172.21.15.87:0/3529368398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c741010f0 0x7f0c741994f0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 >> 172.21.15.87:0/3529368398 conn(0x7f0c74077fd0 msgr2=0x7f0c740fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 shutdown_connections 2024-04-22T13:43:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:09.708+0000 7f0c7bd9f640 1 -- 172.21.15.87:0/3529368398 wait complete. 2024-04-22T13:43:09.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:09 smithi132 ceph-mon[31531]: pgmap v353: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:09.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:09 smithi087 ceph-mon[27999]: pgmap v353: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 15 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:11.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:11 smithi120 ceph-mon[31101]: pgmap v354: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:11.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:11 smithi132 ceph-mon[31531]: pgmap v354: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:11.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:11 smithi087 ceph-mon[27999]: pgmap v354: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:13.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:43:13.354 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.471+0000 7f28526a5640 1 -- 172.21.15.87:0/1223561660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c06b1d0 msgr2=0x7f284c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.471+0000 7f28526a5640 1 --2- 172.21.15.87:0/1223561660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c06b1d0 0x7f284c106b90 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f2840009900 tx=0x7f284002bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.471+0000 7f28526a5640 1 -- 172.21.15.87:0/1223561660 shutdown_connections 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.471+0000 7f28526a5640 1 --2- 172.21.15.87:0/1223561660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c107270 0x7f284c10ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.471+0000 7f28526a5640 1 --2- 172.21.15.87:0/1223561660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c06b1d0 0x7f284c106b90 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 --2- 172.21.15.87:0/1223561660 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c06a820 0x7f284c06ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 -- 172.21.15.87:0/1223561660 >> 172.21.15.87:0/1223561660 conn(0x7f284c077c40 msgr2=0x7f284c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 -- 172.21.15.87:0/1223561660 shutdown_connections 2024-04-22T13:43:13.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 -- 172.21.15.87:0/1223561660 wait complete. 2024-04-22T13:43:13.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 Processor -- start 2024-04-22T13:43:13.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.472+0000 7f28526a5640 1 -- start start 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 0x7f284c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 0x7f284c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f284c112190 con 0x7f284c107270 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f284c112010 con 0x7f284c06a820 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f284c112310 con 0x7f284c19cf50 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53844/0 (socket says 172.21.15.87:53844) 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 -- 172.21.15.87:0/1233051567 learned_addr learned my addr 172.21.15.87:0/1233051567 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f2850c42640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 0x7f284c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284bfff640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 0x7f284c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 -- 172.21.15.87:0/1233051567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 msgr2=0x7f284c19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 0x7f284c19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 -- 172.21.15.87:0/1233051567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 msgr2=0x7f284c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 0x7f284c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 -- 172.21.15.87:0/1233051567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c19da40 con 0x7f284c107270 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f2850c42640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 0x7f284c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284bfff640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 0x7f284c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f284b7fe640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f2840036040 tx=0x7f284002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28497fa640 1 -- 172.21.15.87:0/1233051567 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840043020 con 0x7f284c107270 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284c19dcd0 con 0x7f284c107270 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28497fa640 1 -- 172.21.15.87:0/1233051567 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284002cbe0 con 0x7f284c107270 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28497fa640 1 -- 172.21.15.87:0/1233051567 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840003c30 con 0x7f284c107270 2024-04-22T13:43:13.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.473+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284c1a54f0 con 0x7f284c107270 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f28497fa640 1 -- 172.21.15.87:0/1233051567 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2840046020 con 0x7f284c107270 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 Processor -- start 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 -- start start 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38eea0 0x55eecd397540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd399e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55eecd2f31e0 con 0x55eecd38e580 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eecd290390 con 0x55eecd397a80 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2855374740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eecd2ef1a0 con 0x55eecd38eea0 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f28497fa640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 0x7f28340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2850c42640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2850c42640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53854/0 (socket says 172.21.15.87:53854) 2024-04-22T13:43:13.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2850c42640 1 -- 172.21.15.87:0/3076058320 learned_addr learned my addr 172.21.15.87:0/3076058320 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f2850c42640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 0x7f28340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f284bfff640 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38eea0 0x55eecd397540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f28497fa640 1 -- 172.21.15.87:0/1233051567 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f28400f29f0 con 0x7f284c107270 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.476+0000 7f284b7fe640 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd399e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 -- 172.21.15.87:0/3076058320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38eea0 msgr2=0x55eecd397540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38eea0 0x55eecd397540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 -- 172.21.15.87:0/3076058320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 msgr2=0x55eecd399e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd399e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 -- 172.21.15.87:0/3076058320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eecd0cf7d0 con 0x55eecd38e580 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f2844014f30 tx=0x7f2844012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff9d3fbf5b4a0aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f2850c42640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 0x7f28340a61e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2844012d20 tx=0x7f284401e000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f28327fc640 1 -- 172.21.15.87:0/3076058320 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844023070 con 0x55eecd38e580 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f28327fc640 1 -- 172.21.15.87:0/3076058320 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2844014510 con 0x55eecd38e580 2024-04-22T13:43:13.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.477+0000 7f28327fc640 1 -- 172.21.15.87:0/3076058320 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844005600 con 0x55eecd38e580 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- 172.21.15.87:0/3076058320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 msgr2=0x55eecd38e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f2844014f30 tx=0x7f2844012820 comp rx=0 tx=0).stop 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- 172.21.15.87:0/3076058320 shutdown_connections 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd399e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38eea0 0x55eecd397540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 --2- 172.21.15.87:0/3076058320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38e580 0x55eecd38e960 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- 172.21.15.87:0/3076058320 >> 172.21.15.87:0/3076058320 conn(0x55eecd388cb0 msgr2=0x55eecd389090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- 172.21.15.87:0/3076058320 shutdown_connections 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- 172.21.15.87:0/3076058320 wait complete. 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 -- start start 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.478+0000 7f2855374740 1 Processor -- start 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd394b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd3955d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55eecd39a810 con 0x55eecd38eea0 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eecd2ac2f0 con 0x55eecd397a80 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eecd39a990 con 0x55eecd38e580 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 10 client.0 _ll_register_callbacks cb 0x55eecd2b9580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53860/0 (socket says 172.21.15.87:53860) 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 -- 172.21.15.87:0/2013150017 learned_addr learned my addr 172.21.15.87:0/2013150017 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284bfff640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd394b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2850c42640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd3955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 -- 172.21.15.87:0/2013150017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 msgr2=0x55eecd394b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd394b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 -- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 msgr2=0x55eecd3955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd3955d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 -- 172.21.15.87:0/2013150017 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eecd0053f0 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f284b7fe640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f28400f0750 tx=0x7f2840003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788acb5ade301d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f282b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400be410 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f28400bcdb0 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.479+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2844014510 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f28400bf1b0 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 10 client.18999 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eeccfc50f0 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2844014510 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f28400bb120 con 0x55eecd38eea0 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 10 client.18999 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 10 client.18999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 20 client.18999 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:13.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd3a7a30 0x55eecd3a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 4 client.18999 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eecd300c00 con 0x55eecd3a7a30 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.480+0000 7f284bfff640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd3a7a30 0x55eecd3a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.481+0000 7f284bfff640 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd3a7a30 0x55eecd3a9e70 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e23486aa44490a server_cookie=ee1afc9923131599 in_seq=0 out_seq=0 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.481+0000 7f282b7fe640 10 client.18999 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.481+0000 7f282b7fe640 1 -- 172.21.15.87:0/2013150017 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eecd300c00 con 0x55eecd3a7a30 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.481+0000 7f282b7fe640 10 client.18999 handle_command_reply: tid=0 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.481+0000 7f2855374740 1 client.18999 shutdown 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 msgr2=0x55eecd395090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f28400f0750 tx=0x7f2840003760 comp rx=0 tx=0).stop 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 shutdown_connections 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd3a7a30 0x55eecd3a9e70 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd397a80 0x55eecd3955d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd395090 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f282b7fe640 0 client.18999 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:13.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 --2- 172.21.15.87:0/2013150017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd394b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 >> 172.21.15.87:0/2013150017 conn(0x55eecd388cb0 msgr2=0x55eecd38aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 shutdown_connections 2024-04-22T13:43:13.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 1 -- 172.21.15.87:0/2013150017 wait complete. 2024-04-22T13:43:13.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.482+0000 7f2855374740 20 client.18999 trim_cache size 0 max 16384 2024-04-22T13:43:13.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 Processor -- start 2024-04-22T13:43:13.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 -- start start 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 0x55eecd3a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd393a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1b0270 con 0x55eecd38eea0 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1adb40 con 0x55eecd388cb0 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2855374740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1ab9b0 con 0x55eecd38e580 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53868/0 (socket says 172.21.15.87:53868) 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 -- 172.21.15.87:0/3626218740 learned_addr learned my addr 172.21.15.87:0/3626218740 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f284b7fe640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd393a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f284bfff640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 0x55eecd3a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 -- 172.21.15.87:0/3626218740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 msgr2=0x55eecd393a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd393a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 -- 172.21.15.87:0/3626218740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 msgr2=0x55eecd3a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 0x55eecd3a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f2850c42640 1 -- 172.21.15.87:0/3626218740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eecd38ecd0 con 0x55eecd38eea0 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.499+0000 7f284bfff640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 0x55eecd3a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2850c42640 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2844009cb0 tx=0x7f2844009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6961a02f6bc70d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2831ffb640 1 -- 172.21.15.87:0/3626218740 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844034070 con 0x55eecd38eea0 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2831ffb640 1 -- 172.21.15.87:0/3626218740 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2844005870 con 0x55eecd38eea0 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2831ffb640 1 -- 172.21.15.87:0/3626218740 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844023040 con 0x55eecd38eea0 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 -- 172.21.15.87:0/3626218740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 msgr2=0x55eecd3953f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2844009cb0 tx=0x7f2844009550 comp rx=0 tx=0).stop 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 -- 172.21.15.87:0/3626218740 shutdown_connections 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd38eea0 0x55eecd3953f0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd393a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 --2- 172.21.15.87:0/3626218740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd388cb0 0x55eecd3a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 -- 172.21.15.87:0/3626218740 >> 172.21.15.87:0/3626218740 conn(0x55eecd390520 msgr2=0x55eecd38a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 -- 172.21.15.87:0/3626218740 shutdown_connections 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.500+0000 7f2855374740 1 -- 172.21.15.87:0/3626218740 wait complete. 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 -- start start 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 Processor -- start 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd39be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd38eea0 0x55eecd40cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1ac000 con 0x55eecd388cb0 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1ad470 con 0x55eecd38eea0 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1ac6d0 con 0x55eecd38e580 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2855374740 10 client.0 _ll_register_callbacks cb 0x7f2840009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53874/0 (socket says 172.21.15.87:53874) 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 -- 172.21.15.87:0/783715862 learned_addr learned my addr 172.21.15.87:0/783715862 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284b7fe640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd39be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f2850c42640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd38eea0 0x55eecd40cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 -- 172.21.15.87:0/783715862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 msgr2=0x55eecd39be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd39be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 -- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd38eea0 msgr2=0x55eecd40cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd38eea0 0x55eecd40cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284bfff640 1 -- 172.21.15.87:0/783715862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eecd39c6d0 con 0x55eecd388cb0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.501+0000 7f284b7fe640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd39be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f284bfff640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f283c00bdd0 tx=0x7f283c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1afbde34afb11b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c01d940 con 0x55eecd388cb0 2024-04-22T13:43:13.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f283c0105f0 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2844005870 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f283c02c050 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 10 client.19005 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eecd40e580 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2844005870 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f283c01c960 con 0x55eecd388cb0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 10 client.19005 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 10 client.19005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 20 client.19005 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd397a80 0x55eecd410d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 4 client.19005 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.502+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eecd2faa70 con 0x55eecd397a80 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f284b7fe640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd397a80 0x55eecd410d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f284b7fe640 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd397a80 0x55eecd410d60 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f632510b6a081e7f server_cookie=277ae7e24e30591f in_seq=0 out_seq=0 2024-04-22T13:43:13.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f282b7fe640 10 client.19005 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f282b7fe640 1 -- 172.21.15.87:0/783715862 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55eecd2faa70 con 0x55eecd397a80 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f282b7fe640 10 client.19005 handle_command_reply: tid=0 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 client.19005 shutdown 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 msgr2=0x55eecd39b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f283c00bdd0 tx=0x7f283c011b00 comp rx=0 tx=0).stop 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 shutdown_connections 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55eecd397a80 0x55eecd410d60 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eecd38eea0 0x55eecd40cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eecd38e580 0x55eecd39be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 --2- 172.21.15.87:0/783715862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55eecd388cb0 0x55eecd39b900 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f282b7fe640 0 client.19005 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 >> 172.21.15.87:0/783715862 conn(0x55eecd390520 msgr2=0x55eecd3874e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 shutdown_connections 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 1 -- 172.21.15.87:0/783715862 wait complete. 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.503+0000 7f2855374740 20 client.19005 trim_cache size 0 max 16384 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:43:13.507 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 msgr2=0x7f28340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 0x7f28340a61e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2844012d20 tx=0x7f284401e000 comp rx=0 tx=0).stop 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 msgr2=0x7f284c198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f2840036040 tx=0x7f284002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 shutdown_connections 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28340a3d20 0x7f28340a61e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f284c19cf50 0x7f284c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f284c107270 0x7f284c198bd0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 --2- 172.21.15.87:0/1233051567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f284c06a820 0x7f284c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 >> 172.21.15.87:0/1233051567 conn(0x7f284c077c40 msgr2=0x7f284c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 shutdown_connections 2024-04-22T13:43:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.504+0000 7f28526a5640 1 -- 172.21.15.87:0/1233051567 wait complete. 2024-04-22T13:43:13.511 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:43:13.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:43:13.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- 172.21.15.87:0/1437467829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f381021a0 msgr2=0x7f3f38102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/1437467829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f381021a0 0x7f3f38102580 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009900 tx=0x7f3f2c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:13.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- 172.21.15.87:0/1437467829 shutdown_connections 2024-04-22T13:43:13.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/1437467829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f38107270 0x7f3f3810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/1437467829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f38102b50 0x7f3f38106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/1437467829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f381021a0 0x7f3f38102580 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- 172.21.15.87:0/1437467829 >> 172.21.15.87:0/1437467829 conn(0x7f3f380793a0 msgr2=0x7f3f380797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- 172.21.15.87:0/1437467829 shutdown_connections 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- 172.21.15.87:0/1437467829 wait complete. 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 Processor -- start 2024-04-22T13:43:13.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.615+0000 7f3f3e50e640 1 -- start start 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f381021a0 0x7f3f381986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f3819cf90 0x7f3f3819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381121d0 con 0x7f3f38107270 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38112050 con 0x7f3f3819cf90 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38112350 con 0x7f3f381021a0 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53896/0 (socket says 172.21.15.87:53896) 2024-04-22T13:43:13.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 -- 172.21.15.87:0/3666975680 learned_addr learned my addr 172.21.15.87:0/3666975680 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3caab640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f3819cf90 0x7f3f3819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f37fff640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f381021a0 0x7f3f381986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 -- 172.21.15.87:0/3666975680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f381021a0 msgr2=0x7f3f381986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f381021a0 0x7f3f381986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 -- 172.21.15.87:0/3666975680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f3819cf90 msgr2=0x7f3f3819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f3819cf90 0x7f3f3819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 -- 172.21.15.87:0/3666975680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3819da80 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f377fe640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800d740 tx=0x7f3f2800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28017070 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f2800de50 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3819dd70 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28004ce0 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.616+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f381a5510 con 0x7f3f38107270 2024-04-22T13:43:13.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.617+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f38069230 con 0x7f3f38107270 2024-04-22T13:43:13.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.618+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3f28004e80 con 0x7f3f38107270 2024-04-22T13:43:13.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.619+0000 7f3f357fa640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 0x7f3f200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.619+0000 7f3f37fff640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 0x7f3f200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.619+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f3f280d00f0 con 0x7f3f38107270 2024-04-22T13:43:13.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.620+0000 7f3f37fff640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 0x7f3f200a61e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c006010 tx=0x7f3f2c005e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.624+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f2809c150 con 0x7f3f38107270 2024-04-22T13:43:13.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:13 smithi120 ceph-mon[31101]: pgmap v355: 97 pgs: 97 active+clean; 12 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:13.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:13 smithi132 ceph-mon[31531]: pgmap v355: 97 pgs: 97 active+clean; 12 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:13.736 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:13 smithi087 ceph-mon[27999]: pgmap v355: 97 pgs: 97 active+clean; 12 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T13:43:13.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.839+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f38102b50 con 0x7f3f38107270 2024-04-22T13:43:13.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.840+0000 7f3f357fa640 1 -- 172.21.15.87:0/3666975680 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f3f280a0450 con 0x7f3f38107270 2024-04-22T13:43:13.836 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:43:13.836 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:43:13.836 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:43:13.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 msgr2=0x7f3f200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 0x7f3f200a61e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c006010 tx=0x7f3f2c005e40 comp rx=0 tx=0).stop 2024-04-22T13:43:13.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 msgr2=0x7f3f38198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800d740 tx=0x7f3f2800dc00 comp rx=0 tx=0).stop 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 shutdown_connections 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3f200a3d20 0x7f3f200a61e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f3819cf90 0x7f3f3819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f38107270 0x7f3f38198c10 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 --2- 172.21.15.87:0/3666975680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3f381021a0 0x7f3f381986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 >> 172.21.15.87:0/3666975680 conn(0x7f3f380793a0 msgr2=0x7f3f380ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 shutdown_connections 2024-04-22T13:43:13.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.842+0000 7f3f3e50e640 1 -- 172.21.15.87:0/3666975680 wait complete. 2024-04-22T13:43:13.849 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.953+0000 7f84de587640 1 -- 172.21.15.87:0/2337499341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d8101090 msgr2=0x7f84d8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.953+0000 7f84de587640 1 --2- 172.21.15.87:0/2337499341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d8101090 0x7f84d8101470 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009900 tx=0x7f84c802bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 -- 172.21.15.87:0/2337499341 shutdown_connections 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 --2- 172.21.15.87:0/2337499341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8106160 0x7f84d810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 --2- 172.21.15.87:0/2337499341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8101a40 0x7f84d8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 --2- 172.21.15.87:0/2337499341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d8101090 0x7f84d8101470 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 -- 172.21.15.87:0/2337499341 >> 172.21.15.87:0/2337499341 conn(0x7f84d8077f50 msgr2=0x7f84d8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 -- 172.21.15.87:0/2337499341 shutdown_connections 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 -- 172.21.15.87:0/2337499341 wait complete. 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 Processor -- start 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.954+0000 7f84de587640 1 -- start start 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8101090 0x7f84d819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 0x7f84d819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81110c0 con 0x7f84d81985d0 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8110f40 con 0x7f84d8101090 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8111240 con 0x7f84d8106160 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53918/0 (socket says 172.21.15.87:53918) 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 -- 172.21.15.87:0/3001635805 learned_addr learned my addr 172.21.15.87:0/3001635805 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d7fff640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8101090 0x7f84d819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d77fe640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 0x7f84d819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 -- 172.21.15.87:0/3001635805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 msgr2=0x7f84d819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 0x7f84d819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 -- 172.21.15.87:0/3001635805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8101090 msgr2=0x7f84d819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8101090 0x7f84d819e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 -- 172.21.15.87:0/3001635805 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d81990f0 con 0x7f84d81985d0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d77fe640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 0x7f84d819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f84d000eaa0 tx=0x7f84d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d57fa640 1 -- 172.21.15.87:0/3001635805 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d000cdb0 con 0x7f84d81985d0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d57fa640 1 -- 172.21.15.87:0/3001635805 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d0005020 con 0x7f84d81985d0 2024-04-22T13:43:13.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d81993e0 con 0x7f84d81985d0 2024-04-22T13:43:13.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84d57fa640 1 -- 172.21.15.87:0/3001635805 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00116d0 con 0x7f84d81985d0 2024-04-22T13:43:13.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.955+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d81a55e0 con 0x7f84d81985d0 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.957+0000 7f84d57fa640 1 -- 172.21.15.87:0/3001635805 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f84d00051d0 con 0x7f84d81985d0 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 Processor -- start 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 -- start start 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84d57fa640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56090dbdb1f0 con 0x56090dc7fa00 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56090db784b0 con 0x56090dc76e80 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84e1170740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56090dbd71b0 con 0x56090dc764d0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84d77fe640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84d57fa640 1 -- 172.21.15.87:0/3001635805 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f84d00d0990 con 0x7f84d81985d0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53932/0 (socket says 172.21.15.87:53932) 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84dcb24640 1 -- 172.21.15.87:0/247541491 learned_addr learned my addr 172.21.15.87:0/247541491 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84d77fe640 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.958+0000 7f84d7fff640 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 -- 172.21.15.87:0/247541491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 msgr2=0x56090dc768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc768b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 -- 172.21.15.87:0/247541491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 msgr2=0x56090dc7f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 -- 172.21.15.87:0/247541491 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56090db6a1b0 con 0x56090dc7fa00 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84d77fe640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 0x7f84c00a61e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f84d80fe9d0 tx=0x7f84cc0138e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84dcb24640 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f84d00ce6f0 tx=0x7f84d0080620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4af0f316f64130c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84be7fc640 1 -- 172.21.15.87:0/247541491 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d001f040 con 0x56090dc7fa00 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84be7fc640 1 -- 172.21.15.87:0/247541491 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00a0cf0 con 0x56090dc7fa00 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84be7fc640 1 -- 172.21.15.87:0/247541491 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00ce500 con 0x56090dc7fa00 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84e1170740 1 -- 172.21.15.87:0/247541491 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 msgr2=0x56090dc81e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.959+0000 7f84e1170740 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f84d00ce6f0 tx=0x7f84d0080620 comp rx=0 tx=0).stop 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- 172.21.15.87:0/247541491 shutdown_connections 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc81e50 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- 172.21.15.87:0/247541491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- 172.21.15.87:0/247541491 >> 172.21.15.87:0/247541491 conn(0x56090dc70ba0 msgr2=0x56090dc70f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- 172.21.15.87:0/247541491 shutdown_connections 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- 172.21.15.87:0/247541491 wait complete. 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- start start 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 Processor -- start 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc7f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x56090dc828b0 con 0x56090dc7fa00 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56090db94500 con 0x56090dc76e80 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56090dc82a30 con 0x56090dc764d0 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84e1170740 10 client.0 _ll_register_callbacks cb 0x56090db62b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.960+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53934/0 (socket says 172.21.15.87:53934) 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 -- 172.21.15.87:0/3964723031 learned_addr learned my addr 172.21.15.87:0/3964723031 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84d77fe640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84d7fff640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc7f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 -- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 msgr2=0x56090dc7f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc7f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 -- 172.21.15.87:0/3964723031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 msgr2=0x56090dc7c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 -- 172.21.15.87:0/3964723031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56090d8ed3f0 con 0x56090dc7fa00 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84d7fff640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc7f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84dcb24640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f84d00992c0 tx=0x7f84d009cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78101f46f4ac22ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d001f070 con 0x56090dc7fa00 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f84d00054f0 con 0x56090dc7fa00 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d00a0cf0 con 0x56090dc7fa00 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f84d00ce4c0 con 0x56090dc7fa00 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84e1170740 10 client.19023 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:13.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56090d9b77d0 con 0x56090dc7fa00 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00a0cf0 con 0x56090dc7fa00 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.961+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f84d000f040 con 0x56090dc7fa00 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 10 client.19023 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 10 client.19023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 20 client.19023 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dc91e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 4 client.19023 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56090db7a930 con 0x56090dc8fa40 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84d7fff640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dc91e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84d7fff640 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dc91e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecd09a21332f3596 server_cookie=8bc899c18eacd15c in_seq=0 out_seq=0 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.962+0000 7f84b77fe640 10 client.19023 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84b77fe640 1 -- 172.21.15.87:0/3964723031 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56090db7a930 con 0x56090dc8fa40 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84b77fe640 10 client.19023 handle_command_reply: tid=0 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 client.19023 shutdown 2024-04-22T13:43:13.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 msgr2=0x56090dc7cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f84d00992c0 tx=0x7f84d009cac0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 shutdown_connections 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dc91e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84b77fe640 0 client.19023 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc7cd40 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 --2- 172.21.15.87:0/3964723031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc764d0 0x56090dc7f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 >> 172.21.15.87:0/3964723031 conn(0x56090dc70ba0 msgr2=0x56090dc728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 shutdown_connections 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 1 -- 172.21.15.87:0/3964723031 wait complete. 2024-04-22T13:43:13.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.963+0000 7f84e1170740 20 client.19023 trim_cache size 0 max 16384 2024-04-22T13:43:13.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.976+0000 7f84e1170740 1 Processor -- start 2024-04-22T13:43:13.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 -- start start 2024-04-22T13:43:13.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 0x56090dc7e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81b04b0 con 0x56090dc764d0 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81add80 con 0x56090dc76e80 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84e1170740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81abbf0 con 0x56090dc7fa00 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53942/0 (socket says 172.21.15.87:53942) 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 -- 172.21.15.87:0/1496303234 learned_addr learned my addr 172.21.15.87:0/1496303234 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d77fe640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84dcb24640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 0x56090dc7e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 -- 172.21.15.87:0/1496303234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 msgr2=0x56090dc7e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 0x56090dc7e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 -- 172.21.15.87:0/1496303234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 msgr2=0x56090dc7bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 -- 172.21.15.87:0/1496303234 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56090dc76c20 con 0x56090dc764d0 2024-04-22T13:43:13.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d77fe640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84dcb24640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 0x56090dc7e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84d7fff640 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f84c802c170 tx=0x7f84c8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfe875440d5d8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84bdffb640 1 -- 172.21.15.87:0/1496303234 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8048050 con 0x56090dc764d0 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84bdffb640 1 -- 172.21.15.87:0/1496303234 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84c802cc90 con 0x56090dc764d0 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.977+0000 7f84bdffb640 1 -- 172.21.15.87:0/1496303234 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8005230 con 0x56090dc764d0 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- 172.21.15.87:0/1496303234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 msgr2=0x56090dc8b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f84c802c170 tx=0x7f84c8005e40 comp rx=0 tx=0).stop 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- 172.21.15.87:0/1496303234 shutdown_connections 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc7fa00 0x56090dc7e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc76e80 0x56090dc7bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 --2- 172.21.15.87:0/1496303234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc764d0 0x56090dc8b510 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- 172.21.15.87:0/1496303234 >> 172.21.15.87:0/1496303234 conn(0x56090dc70ba0 msgr2=0x56090dc7de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- 172.21.15.87:0/1496303234 shutdown_connections 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- 172.21.15.87:0/1496303234 wait complete. 2024-04-22T13:43:13.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.978+0000 7f84e1170740 1 -- start start 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 Processor -- start 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc764d0 0x56090dc928d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc76e80 0x56090dc92e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ac240 con 0x56090dc7fa00 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ad6b0 con 0x56090dc764d0 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ac910 con 0x56090dc76e80 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84e1170740 10 client.0 _ll_register_callbacks cb 0x7f84d00ce370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53954/0 (socket says 172.21.15.87:53954) 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 -- 172.21.15.87:0/241530002 learned_addr learned my addr 172.21.15.87:0/241530002 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84d7fff640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc764d0 0x56090dc928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84d77fe640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc76e80 0x56090dc92e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 -- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc76e80 msgr2=0x56090dc92e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc76e80 0x56090dc92e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 -- 172.21.15.87:0/241530002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc764d0 msgr2=0x56090dc928d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc764d0 0x56090dc928d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 -- 172.21.15.87:0/241530002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56090dcf9060 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84dcb24640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f84d0098da0 tx=0x7f84d007bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b059cba4ed2d0ec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.979+0000 7f84b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d001f070 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f84d009bd90 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c802cc90 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f84d0098be0 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 10 client.19029 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56090dcfa2b0 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84c802cc90 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f84d0098690 con 0x56090dc7fa00 2024-04-22T13:43:13.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 10 client.19029 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 10 client.19029 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 20 client.19029 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dcfc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 4 client.19029 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56090dbe8c10 con 0x56090dc8fa40 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.980+0000 7f84d7fff640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dcfc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84b77fe640 10 client.19029 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84d7fff640 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dcfc7a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d80b9df2e5326f57 server_cookie=eef7c5e55de76c19 in_seq=0 out_seq=0 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84b77fe640 1 -- 172.21.15.87:0/241530002 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56090dbe8c10 con 0x56090dc8fa40 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84b77fe640 10 client.19029 handle_command_reply: tid=0 2024-04-22T13:43:13.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 client.19029 shutdown 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 msgr2=0x56090dc93350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f84d0098da0 tx=0x7f84d007bd20 comp rx=0 tx=0).stop 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 shutdown_connections 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x56090dc8fa40 0x56090dcfc7a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x56090dc7fa00 0x56090dc93350 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56090dc76e80 0x56090dc92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84b77fe640 0 client.19029 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 --2- 172.21.15.87:0/241530002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56090dc764d0 0x56090dc928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 >> 172.21.15.87:0/241530002 conn(0x56090dc70ba0 msgr2=0x56090dc725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 shutdown_connections 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 1 -- 172.21.15.87:0/241530002 wait complete. 2024-04-22T13:43:13.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.981+0000 7f84e1170740 20 client.19029 trim_cache size 0 max 16384 2024-04-22T13:43:13.985 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:13.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 msgr2=0x7f84c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 0x7f84c00a61e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f84d80fe9d0 tx=0x7f84cc0138e0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 msgr2=0x7f84d8198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f84d000eaa0 tx=0x7f84d000ef60 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 shutdown_connections 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f84d81985d0 0x7f84d8198a80 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84d8106160 0x7f84d819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 --2- 172.21.15.87:0/3001635805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84d8101090 0x7f84d819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 >> 172.21.15.87:0/3001635805 conn(0x7f84d8077f50 msgr2=0x7f84d80694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 shutdown_connections 2024-04-22T13:43:13.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:13.982+0000 7f84de587640 1 -- 172.21.15.87:0/3001635805 wait complete. 2024-04-22T13:43:13.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 -- 172.21.15.87:0/2399106982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b8107280 msgr2=0x7f76b810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 --2- 172.21.15.87:0/2399106982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b8107280 0x7f76b810ee30 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009ca0 tx=0x7f76b002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 -- 172.21.15.87:0/2399106982 shutdown_connections 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 --2- 172.21.15.87:0/2399106982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b8107280 0x7f76b810ee30 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 --2- 172.21.15.87:0/2399106982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b806b170 0x7f76b8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 --2- 172.21.15.87:0/2399106982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b806a7c0 0x7f76b806aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 -- 172.21.15.87:0/2399106982 >> 172.21.15.87:0/2399106982 conn(0x7f76b8076c50 msgr2=0x7f76b8077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 -- 172.21.15.87:0/2399106982 shutdown_connections 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 -- 172.21.15.87:0/2399106982 wait complete. 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.094+0000 7f76be9ef640 1 Processor -- start 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 -- start start 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b8107280 0x7f76b8198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 0x7f76b819d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81121c0 con 0x7f76b806a7c0 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76b8112040 con 0x7f76b819cec0 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76be9ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f76b8112340 con 0x7f76b8107280 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53958/0 (socket says 172.21.15.87:53958) 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 -- 172.21.15.87:0/1675861781 learned_addr learned my addr 172.21.15.87:0/1675861781 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 0x7f76b819d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b77fe640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b8107280 0x7f76b8198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 -- 172.21.15.87:0/1675861781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b8107280 msgr2=0x7f76b8198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b8107280 0x7f76b8198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 -- 172.21.15.87:0/1675861781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 msgr2=0x7f76b819d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 0x7f76b819d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 -- 172.21.15.87:0/1675861781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b819d9b0 con 0x7f76b806a7c0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 0x7f76b819d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.095+0000 7f76b7fff640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f76a800c920 tx=0x7f76a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.096+0000 7f76b57fa640 1 -- 172.21.15.87:0/1675861781 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8007c70 con 0x7f76b806a7c0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.096+0000 7f76b57fa640 1 -- 172.21.15.87:0/1675861781 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76a8007e10 con 0x7f76b806a7c0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.096+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b819dca0 con 0x7f76b806a7c0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.096+0000 7f76b57fa640 1 -- 172.21.15.87:0/1675861781 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8018820 con 0x7f76b806a7c0 2024-04-22T13:43:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.096+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b81a5490 con 0x7f76b806a7c0 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76b57fa640 1 -- 172.21.15.87:0/1675861781 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f76a800f450 con 0x7f76b806a7c0 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 Processor -- start 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 -- start start 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee248e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2497430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fee23f31f0 con 0x55fee2497a00 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fee23904b0 con 0x55fee248ee80 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76c15d6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fee23ef1b0 con 0x55fee248e4d0 2024-04-22T13:43:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.098+0000 7f76b57fa640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 0x7f76a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53974/0 (socket says 172.21.15.87:53974) 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 -- 172.21.15.87:0/639337541 learned_addr learned my addr 172.21.15.87:0/639337541 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b7fff640 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2497430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 0x7f76a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee248e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 -- 172.21.15.87:0/639337541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 msgr2=0x55fee248e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee248e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b57fa640 1 -- 172.21.15.87:0/1675861781 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f76a80d0a20 con 0x7f76b806a7c0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 -- 172.21.15.87:0/639337541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 msgr2=0x55fee2497430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2497430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 -- 172.21.15.87:0/639337541 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee23821b0 con 0x55fee2497a00 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76b77fe640 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f76ac005fd0 tx=0x7f76ac00e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f454888fe257454 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 0x7f76a00a61e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f76b0004af0 tx=0x7f76b0005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f769e7fc640 1 -- 172.21.15.87:0/639337541 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac016070 con 0x55fee2497a00 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f769e7fc640 1 -- 172.21.15.87:0/639337541 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76ac0054d0 con 0x55fee2497a00 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.099+0000 7f769e7fc640 1 -- 172.21.15.87:0/639337541 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac004db0 con 0x55fee2497a00 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- 172.21.15.87:0/639337541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 msgr2=0x55fee2499e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f76ac005fd0 tx=0x7f76ac00e2f0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- 172.21.15.87:0/639337541 shutdown_connections 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee2499e50 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2497430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 --2- 172.21.15.87:0/639337541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee248e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- 172.21.15.87:0/639337541 >> 172.21.15.87:0/639337541 conn(0x55fee2488ba0 msgr2=0x55fee2488f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- 172.21.15.87:0/639337541 shutdown_connections 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- 172.21.15.87:0/639337541 wait complete. 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 -- start start 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.100+0000 7f76c15d6740 1 Processor -- start 2024-04-22T13:43:14.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248e4d0 0x55fee24972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee2497a00 0x55fee2494c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55fee249a970 con 0x55fee248ee80 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fee23ac500 con 0x55fee248e4d0 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fee249aaf0 con 0x55fee2497a00 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 10 client.0 _ll_register_callbacks cb 0x55fee237ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53982/0 (socket says 172.21.15.87:53982) 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 -- 172.21.15.87:0/316699339 learned_addr learned my addr 172.21.15.87:0/316699339 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b7fff640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248e4d0 0x55fee24972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee2497a00 0x55fee2494c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 -- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee2497a00 msgr2=0x55fee2494c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee2497a00 0x55fee2494c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 -- 172.21.15.87:0/316699339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248e4d0 msgr2=0x55fee24972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248e4d0 0x55fee24972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 -- 172.21.15.87:0/316699339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee21053f0 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76b77fe640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0049f0 tx=0x7f76ac0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f796f723d4cc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76977fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac028060 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f76ac004d60 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.101+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ac0054d0 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f76ac016040 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 10 client.19041 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fee21cf7d0 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76ac0054d0 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f76ac00f040 con 0x55fee248ee80 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 10 client.19041 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:14.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 10 client.19041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 20 client.19041 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee24a7b30 0x55fee24a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 4 client.19041 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fee2392930 con 0x55fee24a7b30 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.102+0000 7f76b7fff640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee24a7b30 0x55fee24a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76977fe640 10 client.19041 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76b7fff640 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee24a7b30 0x55fee24a9f20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f22733bd0ed27628 server_cookie=b16453321d95659a in_seq=0 out_seq=0 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76977fe640 1 -- 172.21.15.87:0/316699339 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fee2392930 con 0x55fee24a7b30 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76977fe640 10 client.19041 handle_command_reply: tid=0 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 client.19041 shutdown 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 msgr2=0x55fee2494680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0049f0 tx=0x7f76ac0048f0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 shutdown_connections 2024-04-22T13:43:14.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee24a7b30 0x55fee24a9f20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee2497a00 0x55fee2494c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee2494680 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76977fe640 0 client.19041 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 --2- 172.21.15.87:0/316699339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248e4d0 0x55fee24972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.103+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 >> 172.21.15.87:0/316699339 conn(0x55fee2488ba0 msgr2=0x55fee248a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.104+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 shutdown_connections 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.104+0000 7f76c15d6740 1 -- 172.21.15.87:0/316699339 wait complete. 2024-04-22T13:43:14.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.104+0000 7f76c15d6740 20 client.19041 trim_cache size 0 max 16384 2024-04-22T13:43:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 Processor -- start 2024-04-22T13:43:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 -- start start 2024-04-22T13:43:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee24a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 0x55fee2496db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81b0300 con 0x55fee248ee80 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81adbd0 con 0x55fee2497a00 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76c15d6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81aba40 con 0x55fee248e4d0 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53990/0 (socket says 172.21.15.87:53990) 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 -- 172.21.15.87:0/705327541 learned_addr learned my addr 172.21.15.87:0/705327541 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 0x55fee2496db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b7fff640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee24a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 -- 172.21.15.87:0/705327541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 msgr2=0x55fee24a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee24a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 -- 172.21.15.87:0/705327541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 msgr2=0x55fee2496db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 0x55fee2496db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 -- 172.21.15.87:0/705327541 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee24a43a0 con 0x55fee248ee80 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 0x55fee2496db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b7fff640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee24a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f76b77fe640 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f76ac011da0 tx=0x7f76ac009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d962e7a4745aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f769dffb640 1 -- 172.21.15.87:0/705327541 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac028070 con 0x55fee248ee80 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f769dffb640 1 -- 172.21.15.87:0/705327541 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76ac0092a0 con 0x55fee248ee80 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.117+0000 7f769dffb640 1 -- 172.21.15.87:0/705327541 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac01a950 con 0x55fee248ee80 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- 172.21.15.87:0/705327541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 msgr2=0x55fee248fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f76ac011da0 tx=0x7f76ac009fc0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- 172.21.15.87:0/705327541 shutdown_connections 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee2497a00 0x55fee2496db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee248ee80 0x55fee248fae0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 --2- 172.21.15.87:0/705327541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee24a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- 172.21.15.87:0/705327541 >> 172.21.15.87:0/705327541 conn(0x55fee2488ba0 msgr2=0x55fee248f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- 172.21.15.87:0/705327541 shutdown_connections 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- 172.21.15.87:0/705327541 wait complete. 2024-04-22T13:43:14.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.118+0000 7f76c15d6740 1 -- start start 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 Processor -- start 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee249ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2490e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81ac090 con 0x55fee2497a00 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81ad500 con 0x55fee248ee80 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81ac760 con 0x55fee248e4d0 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76c15d6740 10 client.0 _ll_register_callbacks cb 0x7f76ac00eea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53994/0 (socket says 172.21.15.87:53994) 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 -- 172.21.15.87:0/3057488536 learned_addr learned my addr 172.21.15.87:0/3057488536 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76b77fe640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2490e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76b7fff640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee249ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 -- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 msgr2=0x55fee249ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee249ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 -- 172.21.15.87:0/3057488536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 msgr2=0x55fee2490e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2490e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 -- 172.21.15.87:0/3057488536 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee2491a20 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76bcf8c640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009ca0 tx=0x7f76b00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1ab895382e3825 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76977fe640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.119+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0052070 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7f76b0004510 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ac0092a0 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f76b0039040 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 10 client.19047 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fee250dbe0 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76ac0092a0 con 0x55fee2497a00 2024-04-22T13:43:14.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f76b004b040 con 0x55fee2497a00 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 10 client.19047 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 10 client.19047 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 20 client.19047 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee250e390 0x55fee25107b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 4 client.19047 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fee2400c10 con 0x55fee250e390 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.120+0000 7f76b7fff640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee250e390 0x55fee25107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76b7fff640 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee250e390 0x55fee25107b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8103efb8ced104b6 server_cookie=7969c29c372a9ee in_seq=0 out_seq=0 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76977fe640 10 client.19047 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76977fe640 1 -- 172.21.15.87:0/3057488536 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fee2400c10 con 0x55fee250e390 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76977fe640 10 client.19047 handle_command_reply: tid=0 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 client.19047 shutdown 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 msgr2=0x55fee24913b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009ca0 tx=0x7f76b00095c0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 shutdown_connections 2024-04-22T13:43:14.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55fee250e390 0x55fee25107b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55fee2497a00 0x55fee24913b0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fee248ee80 0x55fee2490e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 --2- 172.21.15.87:0/3057488536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee248e4d0 0x55fee249ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76977fe640 0 client.19047 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 >> 172.21.15.87:0/3057488536 conn(0x55fee2488ba0 msgr2=0x55fee24936b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 shutdown_connections 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 1 -- 172.21.15.87:0/3057488536 wait complete. 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.121+0000 7f76c15d6740 20 client.19047 trim_cache size 0 max 16384 2024-04-22T13:43:14.124 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 msgr2=0x7f76a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 0x7f76a00a61e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f76b0004af0 tx=0x7f76b0005c50 comp rx=0 tx=0).stop 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 msgr2=0x7f76b8198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f76a800c920 tx=0x7f76a800cde0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 shutdown_connections 2024-04-22T13:43:14.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f76a00a3d20 0x7f76a00a61e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b819cec0 0x7f76b819d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76b8107280 0x7f76b8198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 --2- 172.21.15.87:0/1675861781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76b806a7c0 0x7f76b8198600 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 >> 172.21.15.87:0/1675861781 conn(0x7f76b8076c50 msgr2=0x7f76b80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 shutdown_connections 2024-04-22T13:43:14.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.122+0000 7f76be9ef640 1 -- 172.21.15.87:0/1675861781 wait complete. 2024-04-22T13:43:14.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- 172.21.15.87:0/2655496846 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 msgr2=0x7fdbb4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/2655496846 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb4101130 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009900 tx=0x7fdba802bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- 172.21.15.87:0/2655496846 shutdown_connections 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/2655496846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 0x7fdbb4110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/2655496846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 0x7fdbb410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/2655496846 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb4101130 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- 172.21.15.87:0/2655496846 >> 172.21.15.87:0/2655496846 conn(0x7fdbb4077f50 msgr2=0x7fdbb4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- 172.21.15.87:0/2655496846 shutdown_connections 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- 172.21.15.87:0/2655496846 wait complete. 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 Processor -- start 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.233+0000 7fdbb8c81640 1 -- start start 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 0x7fdbb41a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 0x7fdbb419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41143c0 con 0x7fdbb4100d50 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb4114240 con 0x7fdbb410e900 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb4114540 con 0x7fdbb4101670 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54022/0 (socket says 172.21.15.87:54022) 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 -- 172.21.15.87:0/1869981322 learned_addr learned my addr 172.21.15.87:0/1869981322 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 0x7fdbb41a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 0x7fdbb419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 -- 172.21.15.87:0/1869981322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 msgr2=0x7fdbb41a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 0x7fdbb41a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 -- 172.21.15.87:0/1869981322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 msgr2=0x7fdbb419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 0x7fdbb419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 -- 172.21.15.87:0/1869981322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb419b000 con 0x7fdbb4100d50 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb259c640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdba8005940 tx=0x7fdba802cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdba37fe640 1 -- 172.21.15.87:0/1869981322 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8043020 con 0x7fdbb4100d50 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb419b290 con 0x7fdbb4100d50 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdba37fe640 1 -- 172.21.15.87:0/1869981322 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba8004350 con 0x7fdbb4100d50 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.234+0000 7fdba37fe640 1 -- 172.21.15.87:0/1869981322 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8003af0 con 0x7fdbb4100d50 2024-04-22T13:43:14.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.235+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb41a7600 con 0x7fdbb4100d50 2024-04-22T13:43:14.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdba37fe640 1 -- 172.21.15.87:0/1869981322 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdba8046020 con 0x7fdbb4100d50 2024-04-22T13:43:14.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 Processor -- start 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 -- start start 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 0x558b78a4bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b7899f1f0 con 0x558b78a43690 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b7893c4b0 con 0x558b78a4c210 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.237+0000 7fdbbb876740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b7899b1b0 con 0x558b78a42ce0 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 0x558b78a4bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.87:50110/0 (socket says 172.21.15.87:50110) 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 0x558b78a4bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54030/0 (socket says 172.21.15.87:54030) 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/1119373495 learned_addr learned my addr 172.21.15.87:0/1119373495 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdba37fe640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 0x7fdb9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 -- 172.21.15.87:0/1119373495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 msgr2=0x558b78a430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 0x7fdb9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 -- 172.21.15.87:0/1119373495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 msgr2=0x558b78a4bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 0x558b78a4bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 -- 172.21.15.87:0/1119373495 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b7892e1b0 con 0x558b78a4c210 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdba37fe640 1 -- 172.21.15.87:0/1869981322 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7fdba80f2820 con 0x7fdbb4100d50 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4e660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdba4005d00 tx=0x7fdba40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daca2e6034303a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 0x7fdb9c0a61e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdbac005c40 tx=0x7fdbac004ab0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.238+0000 7fdba0ff9640 1 -- 172.21.15.87:0/1119373495 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba401c480 con 0x558b78a4c210 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdba0ff9640 1 -- 172.21.15.87:0/1119373495 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba401cad0 con 0x558b78a4c210 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdba0ff9640 1 -- 172.21.15.87:0/1119373495 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4011ab0 con 0x558b78a4c210 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 -- 172.21.15.87:0/1119373495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 msgr2=0x558b78a4e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4e660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdba4005d00 tx=0x7fdba40107a0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 -- 172.21.15.87:0/1119373495 shutdown_connections 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4e660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a43690 0x558b78a4bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 --2- 172.21.15.87:0/1119373495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 -- 172.21.15.87:0/1119373495 >> 172.21.15.87:0/1119373495 conn(0x558b78a3d3b0 msgr2=0x558b78a3d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 -- 172.21.15.87:0/1119373495 shutdown_connections 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.239+0000 7fdbbb876740 1 -- 172.21.15.87:0/1119373495 wait complete. 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 -- start start 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 Processor -- start 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a4bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a43690 0x558b78a48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x558b78a4f180 con 0x558b78a4c210 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558b78958500 con 0x558b78a43690 2024-04-22T13:43:14.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b78a4f300 con 0x558b78a42ce0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbbb876740 10 client.0 _ll_register_callbacks cb 0x558b78926b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54034/0 (socket says 172.21.15.87:54034) 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/122016084 learned_addr learned my addr 172.21.15.87:0/122016084 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a43690 0x558b78a48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb259c640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a4bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/122016084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 msgr2=0x558b78a4bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a4bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a43690 msgr2=0x558b78a48ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a43690 0x558b78a48ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/122016084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b786b13f0 con 0x558b78a4c210 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.240+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdbac003d90 tx=0x7fdbac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bed9225c47d4de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:14.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac025030 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7fdbac0045d0 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdba401cad0 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbac003e80 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdbbb876740 10 client.19059 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558b7877b7d0 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba401cad0 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdbac01f040 con 0x558b78a4c210 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdbbb876740 10 client.19059 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.241+0000 7fdbbb876740 10 client.19059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbbb876740 20 client.19059 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78a5c400 0x558b78a5e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbbb876740 4 client.19059 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b7893e930 con 0x558b78a5c400 2024-04-22T13:43:14.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbb259c640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78a5c400 0x558b78a5e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdb99ffb640 10 client.19059 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.242+0000 7fdbb259c640 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78a5c400 0x558b78a5e880 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d934dad2d544a948 server_cookie=4c14e85ab0f2ce4c in_seq=0 out_seq=0 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdb99ffb640 1 -- 172.21.15.87:0/122016084 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b7893e930 con 0x558b78a5c400 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdb99ffb640 10 client.19059 handle_command_reply: tid=0 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 client.19059 shutdown 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 msgr2=0x558b78a49420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fdbac003d90 tx=0x7fdbac004a40 comp rx=0 tx=0).stop 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 shutdown_connections 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78a5c400 0x558b78a5e880 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a49420 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a43690 0x558b78a48ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 --2- 172.21.15.87:0/122016084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a42ce0 0x558b78a4bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 >> 172.21.15.87:0/122016084 conn(0x558b78a3d3b0 msgr2=0x558b78a3f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdb99ffb640 0 client.19059 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.243+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 shutdown_connections 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.244+0000 7fdbbb876740 1 -- 172.21.15.87:0/122016084 wait complete. 2024-04-22T13:43:14.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.244+0000 7fdbbb876740 20 client.19059 trim_cache size 0 max 16384 2024-04-22T13:43:14.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 Processor -- start 2024-04-22T13:43:14.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 -- start start 2024-04-22T13:43:14.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41ae250 con 0x558b78a42ce0 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41abb20 con 0x558b78a4c210 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbbb876740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41a9990 con 0x558b78a43690 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54048/0 (socket says 172.21.15.87:54048) 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 -- 172.21.15.87:0/2149679724 learned_addr learned my addr 172.21.15.87:0/2149679724 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 -- 172.21.15.87:0/2149679724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 msgr2=0x558b78a4b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 -- 172.21.15.87:0/2149679724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 msgr2=0x558b78a4bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.259+0000 7fdbb259c640 1 -- 172.21.15.87:0/2149679724 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b78a44a80 con 0x558b78a42ce0 2024-04-22T13:43:14.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdbb259c640 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdba80bb2b0 tx=0x7fdba80df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24461a5a5bc05485 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdb9bfff640 1 -- 172.21.15.87:0/2149679724 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80befe0 con 0x558b78a42ce0 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdb9bfff640 1 -- 172.21.15.87:0/2149679724 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba80f9040 con 0x558b78a42ce0 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdb9bfff640 1 -- 172.21.15.87:0/2149679724 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80f0200 con 0x558b78a42ce0 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdbbb876740 1 -- 172.21.15.87:0/2149679724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 msgr2=0x558b78a58040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdbbb876740 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdba80bb2b0 tx=0x7fdba80df040 comp rx=0 tx=0).stop 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.260+0000 7fdbbb876740 1 -- 172.21.15.87:0/2149679724 shutdown_connections 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a4c210 0x558b78a4bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- 172.21.15.87:0/2149679724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a42ce0 0x558b78a58040 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- 172.21.15.87:0/2149679724 >> 172.21.15.87:0/2149679724 conn(0x558b78a3d3b0 msgr2=0x558b78a48e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- 172.21.15.87:0/2149679724 shutdown_connections 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- 172.21.15.87:0/2149679724 wait complete. 2024-04-22T13:43:14.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- start start 2024-04-22T13:43:14.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 Processor -- start 2024-04-22T13:43:14.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a42ce0 0x558b78a54700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41a9fe0 con 0x558b78a4c210 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41ab450 con 0x558b78a42ce0 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41aa6b0 con 0x558b78a43690 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbbb876740 10 client.0 _ll_register_callbacks cb 0x7fdbac003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54052/0 (socket says 172.21.15.87:54052) 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/290061195 learned_addr learned my addr 172.21.15.87:0/290061195 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbb1d9b640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.261+0000 7fdbb259c640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a42ce0 0x558b78a54700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/290061195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 msgr2=0x558b78a4fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a42ce0 msgr2=0x558b78a54700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a42ce0 0x558b78a54700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 -- 172.21.15.87:0/290061195 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b78a50640 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbb2d9d640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00f790 tx=0x7fdbac0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1b9addca3e33ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdb99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac025040 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8499+0+0 (secure 0 0 0) 0x7fdbac021670 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdba80f9040 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbac021870 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbbb876740 10 client.19065 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558b78ac27a0 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.262+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba80f9040 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdbac010a00 con 0x558b78a4c210 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 10 client.19065 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 10 client.19065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 20 client.19065 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78ac2e60 0x558b78ac5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:14.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 4 client.19065 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b789acc10 con 0x558b78ac2e60 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbb259c640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78ac2e60 0x558b78ac5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdb99ffb640 10 client.19065 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbb259c640 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78ac2e60 0x558b78ac5250 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63495c223d5bb5ea server_cookie=cd737748f96e3955 in_seq=0 out_seq=0 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdb99ffb640 1 -- 172.21.15.87:0/290061195 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b789acc10 con 0x558b78ac2e60 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdb99ffb640 10 client.19065 handle_command_reply: tid=0 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.263+0000 7fdbbb876740 1 client.19065 shutdown 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 msgr2=0x558b78a50050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00f790 tx=0x7fdbac0046d0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 shutdown_connections 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x558b78ac2e60 0x558b78ac5250 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x558b78a4c210 0x558b78a50050 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdb99ffb640 0 client.19065 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b78a43690 0x558b78a4fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 --2- 172.21.15.87:0/290061195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558b78a42ce0 0x558b78a54700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 >> 172.21.15.87:0/290061195 conn(0x558b78a3d3b0 msgr2=0x558b78a43a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 shutdown_connections 2024-04-22T13:43:14.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 1 -- 172.21.15.87:0/290061195 wait complete. 2024-04-22T13:43:14.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbbb876740 20 client.19065 trim_cache size 0 max 16384 2024-04-22T13:43:14.266 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 msgr2=0x7fdb9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 0x7fdb9c0a61e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdbac005c40 tx=0x7fdbac004ab0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 msgr2=0x7fdbb41a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fdba8005940 tx=0x7fdba802cd80 comp rx=0 tx=0).stop 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 shutdown_connections 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fdb9c0a3d20 0x7fdb9c0a61e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb410e900 0x7fdbb419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdbb4101670 0x7fdbb41a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 --2- 172.21.15.87:0/1869981322 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdbb4100d50 0x7fdbb41a0680 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:14.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.264+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 >> 172.21.15.87:0/1869981322 conn(0x7fdbb4077f50 msgr2=0x7fdbb4110040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:14.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.265+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 shutdown_connections 2024-04-22T13:43:14.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:14.265+0000 7fdbb8c81640 1 -- 172.21.15.87:0/1869981322 wait complete. 2024-04-22T13:43:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:14 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3666975680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:14.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:14 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:43:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:14 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3666975680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:14 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:43:14.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:14 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3666975680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:14.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:14 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:43:15.273 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:43:15.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:43:15.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:43:15.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 -- 172.21.15.87:0/3830242922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101a40 msgr2=0x7f74d4105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 --2- 172.21.15.87:0/3830242922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101a40 0x7f74d4105a80 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009900 tx=0x7f74c402bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 -- 172.21.15.87:0/3830242922 shutdown_connections 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 --2- 172.21.15.87:0/3830242922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 --2- 172.21.15.87:0/3830242922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101a40 0x7f74d4105a80 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 --2- 172.21.15.87:0/3830242922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4101090 0x7f74d4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 -- 172.21.15.87:0/3830242922 >> 172.21.15.87:0/3830242922 conn(0x7f74d4077f50 msgr2=0x7f74d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 -- 172.21.15.87:0/3830242922 shutdown_connections 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.385+0000 7f74dc014640 1 -- 172.21.15.87:0/3830242922 wait complete. 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 Processor -- start 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 -- start start 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4198590 0x7f74d4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f74d4111080 con 0x7f74d4101090 2024-04-22T13:43:15.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f74d4110f00 con 0x7f74d4106160 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74dc014640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74d4111200 con 0x7f74d4198590 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54066/0 (socket says 172.21.15.87:54066) 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 -- 172.21.15.87:0/157561669 learned_addr learned my addr 172.21.15.87:0/157561669 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d95af640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 -- 172.21.15.87:0/157561669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4198590 msgr2=0x7f74d4198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74da5b1640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4198590 0x7f74d4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4198590 0x7f74d4198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.386+0000 7f74d9db0640 1 -- 172.21.15.87:0/157561669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 msgr2=0x7f74d419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74d9db0640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74d9db0640 1 -- 172.21.15.87:0/157561669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d41990b0 con 0x7f74d4101090 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74d95af640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74d9db0640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f74d000d950 tx=0x7f74d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74caffd640 1 -- 172.21.15.87:0/157561669 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0017030 con 0x7f74d4101090 2024-04-22T13:43:15.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74caffd640 1 -- 172.21.15.87:0/157561669 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d0004020 con 0x7f74d4101090 2024-04-22T13:43:15.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74caffd640 1 -- 172.21.15.87:0/157561669 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d00047a0 con 0x7f74d4101090 2024-04-22T13:43:15.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d41993a0 con 0x7f74d4101090 2024-04-22T13:43:15.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.387+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d41a55f0 con 0x7f74d4101090 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.389+0000 7f74caffd640 1 -- 172.21.15.87:0/157561669 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f74d0009a30 con 0x7f74d4101090 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.389+0000 7f74dec06740 1 Processor -- start 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.389+0000 7f74dec06740 1 -- start start 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f46cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564a63eaaf00 con 0x564a63f4fe10 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a63e48590 con 0x564a63f47290 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74dec06740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564a63ea6ec0 con 0x564a63f468e0 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74caffd640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 0x7f74c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54080/0 (socket says 172.21.15.87:54080) 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 -- 172.21.15.87:0/642347400 learned_addr learned my addr 172.21.15.87:0/642347400 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74da5b1640 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f46cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74da5b1640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 0x7f74c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74caffd640 1 -- 172.21.15.87:0/157561669 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f74d00d0310 con 0x7f74d4101090 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d9db0640 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 -- 172.21.15.87:0/642347400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 msgr2=0x564a63f46cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f46cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 -- 172.21.15.87:0/642347400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 msgr2=0x564a63f4f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 -- 172.21.15.87:0/642347400 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a63c857e0 con 0x564a63f4fe10 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74d95af640 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8869300b41ffdba5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.390+0000 7f74da5b1640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 0x7f74c00a61e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009fd0 tx=0x7f74cc009240 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74bffff640 1 -- 172.21.15.87:0/642347400 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4043020 con 0x564a63f4fe10 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74bffff640 1 -- 172.21.15.87:0/642347400 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74c402c7e0 con 0x564a63f4fe10 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74bffff640 1 -- 172.21.15.87:0/642347400 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4005570 con 0x564a63f4fe10 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 -- 172.21.15.87:0/642347400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 msgr2=0x564a63f52260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c402c650 comp rx=0 tx=0).stop 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 -- 172.21.15.87:0/642347400 shutdown_connections 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f52260 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 --2- 172.21.15.87:0/642347400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f46cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 -- 172.21.15.87:0/642347400 >> 172.21.15.87:0/642347400 conn(0x564a63f40fb0 msgr2=0x564a63f41390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 -- 172.21.15.87:0/642347400 shutdown_connections 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.391+0000 7f74dec06740 1 -- 172.21.15.87:0/642347400 wait complete. 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 -- start start 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 Processor -- start 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f47290 0x564a63f4d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f4fe10 0x564a63f4d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x564a63f52d20 con 0x564a63f468e0 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a63e64580 con 0x564a63f4fe10 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564a63f52ea0 con 0x564a63f47290 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74dec06740 10 client.0 _ll_register_callbacks cb 0x564a63e6a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74d9db0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74d9db0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54086/0 (socket says 172.21.15.87:54086) 2024-04-22T13:43:15.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74d9db0640 1 -- 172.21.15.87:0/3922343124 learned_addr learned my addr 172.21.15.87:0/3922343124 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74d95af640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f47290 0x564a63f4d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.392+0000 7f74da5b1640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f4fe10 0x564a63f4d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 -- 172.21.15.87:0/3922343124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f47290 msgr2=0x564a63f4d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f47290 0x564a63f4d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 -- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f4fe10 msgr2=0x564a63f4d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f4fe10 0x564a63f4d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 -- 172.21.15.87:0/3922343124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a63ce5f70 con 0x564a63f468e0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74d9db0640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f74d009c220 tx=0x7f74d009ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f1181855063e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d00a4f00 con 0x564a63f468e0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f74d0009d20 con 0x564a63f468e0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74c402c7e0 con 0x564a63f468e0 2024-04-22T13:43:15.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f74d0098900 con 0x564a63f468e0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 10 client.19077 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a63b7d170 con 0x564a63f468e0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74c402c7e0 con 0x564a63f468e0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f74d00a01f0 con 0x564a63f468e0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 10 client.19077 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 10 client.19077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 20 client.19077 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63f60330 0x564a63f62760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 4 client.19077 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.393+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a63eb8920 con 0x564a63f60330 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.394+0000 7f74d95af640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63f60330 0x564a63f62760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.394+0000 7f74bd7fa640 10 client.19077 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.394+0000 7f74d95af640 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63f60330 0x564a63f62760 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=394f34ec98fb0ff2 server_cookie=1f07f31c89b47ede in_seq=0 out_seq=0 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74bd7fa640 1 -- 172.21.15.87:0/3922343124 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a63eb8920 con 0x564a63f60330 2024-04-22T13:43:15.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74bd7fa640 10 client.19077 handle_command_reply: tid=0 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 client.19077 shutdown 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 msgr2=0x564a63f4cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f74d009c220 tx=0x7f74d009ca80 comp rx=0 tx=0).stop 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 shutdown_connections 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63f60330 0x564a63f62760 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f4fe10 0x564a63f4d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f47290 0x564a63f4d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 --2- 172.21.15.87:0/3922343124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f468e0 0x564a63f4cdd0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74bd7fa640 0 client.19077 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 >> 172.21.15.87:0/3922343124 conn(0x564a63f40fb0 msgr2=0x564a63f42d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 shutdown_connections 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 1 -- 172.21.15.87:0/3922343124 wait complete. 2024-04-22T13:43:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.395+0000 7f74dec06740 20 client.19077 trim_cache size 0 max 16384 2024-04-22T13:43:15.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.412+0000 7f74dec06740 1 Processor -- start 2024-04-22T13:43:15.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 -- start start 2024-04-22T13:43:15.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f4f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41b0440 con 0x564a63f4fe10 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41add10 con 0x564a63f47290 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74dec06740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41abb80 con 0x564a63f468e0 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54102/0 (socket says 172.21.15.87:54102) 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 -- 172.21.15.87:0/4143643308 learned_addr learned my addr 172.21.15.87:0/4143643308 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74d9db0640 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f4f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74d95af640 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 -- 172.21.15.87:0/4143643308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 msgr2=0x564a63f4f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f4f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 -- 172.21.15.87:0/4143643308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 msgr2=0x564a63f4bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 -- 172.21.15.87:0/4143643308 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a63f62d40 con 0x564a63f4fe10 2024-04-22T13:43:15.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74da5b1640 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f74cc013300 tx=0x7f74cc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4708a095f2ac7ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74bf7fe640 1 -- 172.21.15.87:0/4143643308 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc004b20 con 0x564a63f4fe10 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74bf7fe640 1 -- 172.21.15.87:0/4143643308 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74cc004cc0 con 0x564a63f4fe10 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.413+0000 7f74bf7fe640 1 -- 172.21.15.87:0/4143643308 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc01e710 con 0x564a63f4fe10 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- 172.21.15.87:0/4143643308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 msgr2=0x564a63f5bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f74cc013300 tx=0x7f74cc0095c0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- 172.21.15.87:0/4143643308 shutdown_connections 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63f5bbe0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 --2- 172.21.15.87:0/4143643308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f4f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- 172.21.15.87:0/4143643308 >> 172.21.15.87:0/4143643308 conn(0x564a63f40fb0 msgr2=0x564a63f40010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- 172.21.15.87:0/4143643308 shutdown_connections 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- 172.21.15.87:0/4143643308 wait complete. 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 -- start start 2024-04-22T13:43:15.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.414+0000 7f74dec06740 1 Processor -- start 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f49ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41ac1d0 con 0x564a63f4fe10 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41ad640 con 0x564a63f47290 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74d41ac8a0 con 0x564a63f468e0 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74dec06740 10 client.0 _ll_register_callbacks cb 0x7f74d00985c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54110/0 (socket says 172.21.15.87:54110) 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 -- 172.21.15.87:0/1054360279 learned_addr learned my addr 172.21.15.87:0/1054360279 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74d95af640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74d9db0640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f49ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 -- 172.21.15.87:0/1054360279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 msgr2=0x564a63f49ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f49ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 -- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 msgr2=0x564a63f4a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 -- 172.21.15.87:0/1054360279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a63fc5d90 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74da5b1640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0132d0 tx=0x7f74cc005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a9c9745b2e43ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.415+0000 7f74bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc005970 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f74cc004a40 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74cc004cc0 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f74cc01e4b0 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 10 client.19083 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a63fc6f90 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74cc004cc0 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f74cc026070 con 0x564a63f4fe10 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 10 client.19083 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 10 client.19083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 20 client.19083 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63fc7700 0x564a63fc9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 4 client.19083 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564a63eb2790 con 0x564a63fc7700 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.416+0000 7f74d9db0640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63fc7700 0x564a63fc9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.417+0000 7f74d9db0640 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63fc7700 0x564a63fc9b20 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf862ebf86b89526 server_cookie=969de397d51cf7fa in_seq=0 out_seq=0 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.417+0000 7f74bd7fa640 10 client.19083 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74bd7fa640 1 -- 172.21.15.87:0/1054360279 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564a63eb2790 con 0x564a63fc7700 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74bd7fa640 10 client.19083 handle_command_reply: tid=0 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 client.19083 shutdown 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 msgr2=0x564a63fc5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0132d0 tx=0x7f74cc005480 comp rx=0 tx=0).stop 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 shutdown_connections 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x564a63fc7700 0x564a63fc9b20 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564a63f4fe10 0x564a63fc5850 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74bd7fa640 0 client.19083 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a63f47290 0x564a63f4a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 --2- 172.21.15.87:0/1054360279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a63f468e0 0x564a63f49ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 >> 172.21.15.87:0/1054360279 conn(0x564a63f40fb0 msgr2=0x564a63f4cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 shutdown_connections 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 1 -- 172.21.15.87:0/1054360279 wait complete. 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.418+0000 7f74dec06740 20 client.19083 trim_cache size 0 max 16384 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:43:15.420 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "064d089d-c2a8-491a-b34f-2946629df0ad", 2024-04-22T13:43:15.421 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:43:15.421 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 msgr2=0x7f74c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 0x7f74c00a61e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009fd0 tx=0x7f74cc009240 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 msgr2=0x7f74d419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f74d000d950 tx=0x7f74d000de10 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 shutdown_connections 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f74c00a3d20 0x7f74c00a61e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74d4198590 0x7f74d4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74d4106160 0x7f74d419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 --2- 172.21.15.87:0/157561669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74d4101090 0x7f74d419e3e0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 >> 172.21.15.87:0/157561669 conn(0x7f74d4077f50 msgr2=0x7f74d4069520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 shutdown_connections 2024-04-22T13:43:15.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.419+0000 7f74dc014640 1 -- 172.21.15.87:0/157561669 wait complete. 2024-04-22T13:43:15.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:43:15.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:43:15.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 -- 172.21.15.87:0/1041236441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 msgr2=0x7f32601035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 --2- 172.21.15.87:0/1041236441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f32601035d0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3258009900 tx=0x7f325802bd60 comp rx=0 tx=0).stop 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 -- 172.21.15.87:0/1041236441 shutdown_connections 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 --2- 172.21.15.87:0/1041236441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326010e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 --2- 172.21.15.87:0/1041236441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f326010bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 --2- 172.21.15.87:0/1041236441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f32601035d0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 -- 172.21.15.87:0/1041236441 >> 172.21.15.87:0/1041236441 conn(0x7f32600771d0 msgr2=0x7f32600775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 -- 172.21.15.87:0/1041236441 shutdown_connections 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.537+0000 7f3264e69640 1 -- 172.21.15.87:0/1041236441 wait complete. 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 Processor -- start 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 -- start start 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f3260198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32601121c0 con 0x7f32601031f0 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3260112040 con 0x7f3260103b10 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f3264e69640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3260112340 con 0x7f326010c530 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54130/0 (socket says 172.21.15.87:54130) 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 -- 172.21.15.87:0/2038783983 learned_addr learned my addr 172.21.15.87:0/2038783983 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325dd9b640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f3260198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 -- 172.21.15.87:0/2038783983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 msgr2=0x7f326019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 -- 172.21.15.87:0/2038783983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 msgr2=0x7f3260198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f3260198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325e59c640 1 -- 172.21.15.87:0/2038783983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326019d700 con 0x7f32601031f0 2024-04-22T13:43:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325dd9b640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f3260198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.538+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f325e59c640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f3258005940 tx=0x7f325802d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f324f7fe640 1 -- 172.21.15.87:0/2038783983 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258043020 con 0x7f32601031f0 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326019d990 con 0x7f32601031f0 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f324f7fe640 1 -- 172.21.15.87:0/2038783983 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325802dd20 con 0x7f32601031f0 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f324f7fe640 1 -- 172.21.15.87:0/2038783983 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258003b80 con 0x7f32601031f0 2024-04-22T13:43:15.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.539+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601a5280 con 0x7f32601031f0 2024-04-22T13:43:15.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.541+0000 7f324f7fe640 1 -- 172.21.15.87:0/2038783983 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3258046020 con 0x7f32601031f0 2024-04-22T13:43:15.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.541+0000 7f3267a56740 1 Processor -- start 2024-04-22T13:43:15.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.541+0000 7f3267a56740 1 -- start start 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.541+0000 7f3267a56740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.541+0000 7f3267a56740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f3267a56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f3267a56740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5581468221e0 con 0x5581468bd580 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f3267a56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5581467bf390 con 0x5581468c6a80 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f3267a56740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55814681e1a0 con 0x5581468bdea0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f324f7fe640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 0x7f32480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54138/0 (socket says 172.21.15.87:54138) 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 -- 172.21.15.87:0/3829018104 learned_addr learned my addr 172.21.15.87:0/3829018104 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325e59c640 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 0x7f32480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f324f7fe640 1 -- 172.21.15.87:0/2038783983 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f32580f2910 con 0x7f32601031f0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325dd9b640 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 -- 172.21.15.87:0/3829018104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 msgr2=0x5581468c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 -- 172.21.15.87:0/3829018104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 msgr2=0x5581468c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 -- 172.21.15.87:0/3829018104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581465fe7d0 con 0x5581468bd580 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f3250013ab0 tx=0x7f3250013f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5def54e71fa2685f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 0x7f32480a61e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3250010800 tx=0x7f325001b000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f324cff9640 1 -- 172.21.15.87:0/3829018104 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250021070 con 0x5581468bd580 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f324cff9640 1 -- 172.21.15.87:0/3829018104 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325000b8b0 con 0x5581468bd580 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.542+0000 7f324cff9640 1 -- 172.21.15.87:0/3829018104 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325001d650 con 0x5581468bd580 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 -- 172.21.15.87:0/3829018104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 msgr2=0x5581468bd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f3250013ab0 tx=0x7f3250013f70 comp rx=0 tx=0).stop 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 -- 172.21.15.87:0/3829018104 shutdown_connections 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 --2- 172.21.15.87:0/3829018104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468bd960 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 -- 172.21.15.87:0/3829018104 >> 172.21.15.87:0/3829018104 conn(0x5581468b7cb0 msgr2=0x5581468b8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 -- 172.21.15.87:0/3829018104 shutdown_connections 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.543+0000 7f3267a56740 1 -- 172.21.15.87:0/3829018104 wait complete. 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 -- start start 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 Processor -- start 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bd580 0x5581468c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5581468c9810 con 0x5581468bdea0 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5581467db2f0 con 0x5581468c6a80 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5581468c9990 con 0x5581468bd580 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3267a56740 10 client.0 _ll_register_callbacks cb 0x5581467e8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54152/0 (socket says 172.21.15.87:54152) 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 -- 172.21.15.87:0/1877540652 learned_addr learned my addr 172.21.15.87:0/1877540652 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325e59c640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bd580 0x5581468c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325ed9d640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 -- 172.21.15.87:0/1877540652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bd580 msgr2=0x5581468c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bd580 0x5581468c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 -- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 msgr2=0x5581468c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 -- 172.21.15.87:0/1877540652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581465343f0 con 0x5581468bdea0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f325dd9b640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3254012700 tx=0x7f3254012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d4085fe596a47d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.544+0000 7f3245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254010dc0 con 0x5581468bdea0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f325401a3e0 con 0x5581468bdea0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55814682fc00 con 0x5581468bdea0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f325401a5e0 con 0x5581468bdea0 2024-04-22T13:43:15.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 10 client.19095 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5581464f40f0 con 0x5581468bdea0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55814682fc00 con 0x5581468bdea0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3254019070 con 0x5581468bdea0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 10 client.19095 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 10 client.19095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 20 client.19095 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5581468d6b60 0x5581468d8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 4 client.19095 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.545+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558146829a70 con 0x5581468d6b60 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.546+0000 7f325e59c640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5581468d6b60 0x5581468d8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.546+0000 7f3245ffb640 10 client.19095 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.546+0000 7f325e59c640 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5581468d6b60 0x5581468d8fa0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab3b3ad44e3c3a1c server_cookie=b6b9749e384b99f0 in_seq=0 out_seq=0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3245ffb640 1 -- 172.21.15.87:0/1877540652 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558146829a70 con 0x5581468d6b60 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3245ffb640 10 client.19095 handle_command_reply: tid=0 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 client.19095 shutdown 2024-04-22T13:43:15.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 msgr2=0x5581468c4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3254012700 tx=0x7f3254012bc0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 shutdown_connections 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5581468d6b60 0x5581468d8fa0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468c6a80 0x5581468c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bdea0 0x5581468c4050 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 --2- 172.21.15.87:0/1877540652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bd580 0x5581468c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3245ffb640 0 client.19095 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 >> 172.21.15.87:0/1877540652 conn(0x5581468b7cb0 msgr2=0x5581468b9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 shutdown_connections 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 1 -- 172.21.15.87:0/1877540652 wait complete. 2024-04-22T13:43:15.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.547+0000 7f3267a56740 20 client.19095 trim_cache size 0 max 16384 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 Processor -- start 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 -- start start 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bdea0 0x5581468c6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468c6a80 0x5581468d26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32601b03a0 con 0x5581468bd580 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32601adc70 con 0x5581468bdea0 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f3267a56740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32601abae0 con 0x5581468c6a80 2024-04-22T13:43:15.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f325e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f325e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54166/0 (socket says 172.21.15.87:54166) 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f325e59c640 1 -- 172.21.15.87:0/4197585205 learned_addr learned my addr 172.21.15.87:0/4197585205 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.564+0000 7f325dd9b640 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bdea0 0x5581468c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325ed9d640 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468c6a80 0x5581468d26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 -- 172.21.15.87:0/4197585205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468c6a80 msgr2=0x5581468d26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468c6a80 0x5581468d26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 -- 172.21.15.87:0/4197585205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bdea0 msgr2=0x5581468c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bdea0 0x5581468c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 -- 172.21.15.87:0/4197585205 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581468b77e0 con 0x5581468bd580 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f325e59c640 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f32580f06a0 tx=0x7f325802bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf00911b207348a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f3247fff640 1 -- 172.21.15.87:0/4197585205 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32580d7070 con 0x5581468bd580 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f3247fff640 1 -- 172.21.15.87:0/4197585205 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32580d9020 con 0x5581468bd580 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.565+0000 7f3247fff640 1 -- 172.21.15.87:0/4197585205 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32580bcd00 con 0x5581468bd580 2024-04-22T13:43:15.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- 172.21.15.87:0/4197585205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 msgr2=0x5581468c2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f32580f06a0 tx=0x7f325802bd60 comp rx=0 tx=0).stop 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- 172.21.15.87:0/4197585205 shutdown_connections 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468c6a80 0x5581468d26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bdea0 0x5581468c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- 172.21.15.87:0/4197585205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468bd580 0x5581468c2a50 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- 172.21.15.87:0/4197585205 >> 172.21.15.87:0/4197585205 conn(0x5581468b7cb0 msgr2=0x5581468b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- 172.21.15.87:0/4197585205 shutdown_connections 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- 172.21.15.87:0/4197585205 wait complete. 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- start start 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 Processor -- start 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bd580 0x5581468c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32601ac130 con 0x5581468c6a80 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32601ad5a0 con 0x5581468bd580 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32601ac800 con 0x5581468bdea0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f3267a56740 10 client.0 _ll_register_callbacks cb 0x7f32540099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f325ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f325ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54182/0 (socket says 172.21.15.87:54182) 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f325ed9d640 1 -- 172.21.15.87:0/2524247736 learned_addr learned my addr 172.21.15.87:0/2524247736 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f325e59c640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bd580 0x5581468c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.566+0000 7f325dd9b640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 -- 172.21.15.87:0/2524247736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 msgr2=0x5581468c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 -- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bd580 msgr2=0x5581468c0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bd580 0x5581468c0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 -- 172.21.15.87:0/2524247736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581468c19c0 con 0x5581468c6a80 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f325ed9d640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f3250013ab0 tx=0x7f3250013950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10addbb349aeaf26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:15.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250021040 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f3250005040 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32580d9020 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3250005e50 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 10 client.19101 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55814693d730 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32580d9020 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f325002e070 con 0x5581468c6a80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 10 client.19101 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 10 client.19101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 20 client.19101 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55814693dc80 0x5581469400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 4 client.19101 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.567+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581467a5590 con 0x55814693dc80 2024-04-22T13:43:15.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f325e59c640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55814693dc80 0x5581469400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3245ffb640 10 client.19101 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f325e59c640 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55814693dc80 0x5581469400a0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceac349f5cb83917 server_cookie=17cbd71c30001dcd in_seq=0 out_seq=0 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3245ffb640 1 -- 172.21.15.87:0/2524247736 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5581467a5590 con 0x55814693dc80 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3245ffb640 10 client.19101 handle_command_reply: tid=0 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 client.19101 shutdown 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 msgr2=0x55814693c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f3250013ab0 tx=0x7f3250013950 comp rx=0 tx=0).stop 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 shutdown_connections 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55814693dc80 0x5581469400a0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5581468c6a80 0x55814693c0e0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581468bdea0 0x5581468c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3245ffb640 0 client.19101 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:15.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 --2- 172.21.15.87:0/2524247736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581468bd580 0x5581468c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 >> 172.21.15.87:0/2524247736 conn(0x5581468b7cb0 msgr2=0x5581468c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 shutdown_connections 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 1 -- 172.21.15.87:0/2524247736 wait complete. 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.568+0000 7f3267a56740 20 client.19101 trim_cache size 0 max 16384 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "064d089d-c2a8-491a-b34f-2946629df0ad": { 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "064d089d-c2a8-491a-b34f-2946629df0ad", 2024-04-22T13:43:15.572 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:43:15.573 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:15.573 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:15.573 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:15.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 msgr2=0x7f32480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 0x7f32480a61e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3250010800 tx=0x7f325001b000 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 msgr2=0x7f3260198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f3258005940 tx=0x7f325802d920 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 shutdown_connections 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32480a3d20 0x7f32480a61e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f326010c530 0x7f326019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3260103b10 0x7f3260198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 --2- 172.21.15.87:0/2038783983 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32601031f0 0x7f3260198760 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 >> 172.21.15.87:0/2038783983 conn(0x7f32600771d0 msgr2=0x7f32600fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 shutdown_connections 2024-04-22T13:43:15.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:15.569+0000 7f3264e69640 1 -- 172.21.15.87:0/2038783983 wait complete. 2024-04-22T13:43:15.577 INFO:tasks.cephfs.filesystem:scrub status for tag:064d089d-c2a8-491a-b34f-2946629df0ad - {'path': '/', 'tag': '064d089d-c2a8-491a-b34f-2946629df0ad', 'options': 'recursive,force'} 2024-04-22T13:43:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: pgmap v356: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 6 op/s 2024-04-22T13:43:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T13:43:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:43:15.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:43:15.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:15 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: pgmap v356: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 6 op/s 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:43:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:15 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: pgmap v356: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 6 op/s 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:43:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:15 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:43:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:16 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:43:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:16 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:16 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:43:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:16 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.889 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:16 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:43:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:16 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:43:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:16 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:16 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:43:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:16 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:16 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:43:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:16 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:43:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:16 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:16 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:43:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:16 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:16 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:43:17.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:17 smithi120 ceph-mon[31101]: pgmap v358: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 825 B/s wr, 7 op/s 2024-04-22T13:43:17.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:17 smithi132 ceph-mon[31531]: pgmap v358: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 825 B/s wr, 7 op/s 2024-04-22T13:43:17.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:17 smithi087 ceph-mon[27999]: pgmap v358: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 825 B/s wr, 7 op/s 2024-04-22T13:43:19.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:19 smithi120 ceph-mon[31101]: pgmap v359: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 716 B/s wr, 9 op/s 2024-04-22T13:43:19.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:19 smithi132 ceph-mon[31531]: pgmap v359: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 716 B/s wr, 9 op/s 2024-04-22T13:43:19.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:19 smithi087 ceph-mon[27999]: pgmap v359: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 716 B/s wr, 9 op/s 2024-04-22T13:43:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:21 smithi120 ceph-mon[31101]: pgmap v360: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:21.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:21 smithi132 ceph-mon[31531]: pgmap v360: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:21.974 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:43:21.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:21 smithi087 ceph-mon[27999]: pgmap v360: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 -- 172.21.15.87:0/1452932049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29041010a0 msgr2=0x7f2904101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 --2- 172.21.15.87:0/1452932049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29041010a0 0x7f2904101480 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009900 tx=0x7f28f802bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 -- 172.21.15.87:0/1452932049 shutdown_connections 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 --2- 172.21.15.87:0/1452932049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2904106170 0x7f290410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 --2- 172.21.15.87:0/1452932049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2904101a50 0x7f2904105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 --2- 172.21.15.87:0/1452932049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29041010a0 0x7f2904101480 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 -- 172.21.15.87:0/1452932049 >> 172.21.15.87:0/1452932049 conn(0x7f29040771d0 msgr2=0x7f29040775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 -- 172.21.15.87:0/1452932049 shutdown_connections 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 -- 172.21.15.87:0/1452932049 wait complete. 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.088+0000 7f290a7a4640 1 Processor -- start 2024-04-22T13:43:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 -- start start 2024-04-22T13:43:22.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29041010a0 0x7f2904197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 0x7f290419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f29041110b0 con 0x7f2904106170 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2904110f30 con 0x7f29041010a0 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f290a7a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2904111230 con 0x7f290419bef0 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33708/0 (socket says 172.21.15.87:33708) 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 -- 172.21.15.87:0/3080761071 learned_addr learned my addr 172.21.15.87:0/3080761071 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f2908d41640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 0x7f290419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f2903fff640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29041010a0 0x7f2904197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 -- 172.21.15.87:0/3080761071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 msgr2=0x7f290419c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 0x7f290419c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 -- 172.21.15.87:0/3080761071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29041010a0 msgr2=0x7f2904197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29041010a0 0x7f2904197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f29037fe640 1 -- 172.21.15.87:0/3080761071 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290419c9e0 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.089+0000 7f2908d41640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 0x7f290419c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f29037fe640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4017070 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290419ccd0 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28f4009c30 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4003d30 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.090+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29041a44c0 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.091+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2904069230 con 0x7f2904106170 2024-04-22T13:43:22.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.092+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28f4003ed0 con 0x7f2904106170 2024-04-22T13:43:22.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.092+0000 7f29017fa640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 0x7f28ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.092+0000 7f2903fff640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 0x7f28ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.093+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f28f40d04c0 con 0x7f2904106170 2024-04-22T13:43:22.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.093+0000 7f2903fff640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 0x7f28ec0a61e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f802c6b0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.097+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28f4020030 con 0x7f2904106170 2024-04-22T13:43:22.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.313+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2904101a50 con 0x7f2904106170 2024-04-22T13:43:22.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.313+0000 7f29017fa640 1 -- 172.21.15.87:0/3080761071 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f28f4009de0 con 0x7f2904106170 2024-04-22T13:43:22.309 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:43:22.310 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:43:22.310 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:43:22.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 msgr2=0x7f28ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 0x7f28ec0a61e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f802c6b0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 msgr2=0x7f2904197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f400dde0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 shutdown_connections 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f28ec0a3d20 0x7f28ec0a61e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f290419bef0 0x7f290419c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2904106170 0x7f2904197b70 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 --2- 172.21.15.87:0/3080761071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29041010a0 0x7f2904197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 >> 172.21.15.87:0/3080761071 conn(0x7f29040771d0 msgr2=0x7f29040fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 shutdown_connections 2024-04-22T13:43:22.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.315+0000 7f290a7a4640 1 -- 172.21.15.87:0/3080761071 wait complete. 2024-04-22T13:43:22.323 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --set-id b582c357 --release --await' 2024-04-22T13:43:22.323 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --set-id b582c357 --release --await 2024-04-22T13:43:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 -- 172.21.15.87:0/4150304195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94105ea0 msgr2=0x7f4c94106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/4150304195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94105ea0 0x7f4c94106280 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009900 tx=0x7f4c8402bc80 comp rx=0 tx=0).stop 2024-04-22T13:43:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 -- 172.21.15.87:0/4150304195 shutdown_connections 2024-04-22T13:43:22.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/4150304195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 0x7f4c9410dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/4150304195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94101130 0x7f4c94101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/4150304195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94105ea0 0x7f4c94106280 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 -- 172.21.15.87:0/4150304195 >> 172.21.15.87:0/4150304195 conn(0x7f4c94077e30 msgr2=0x7f4c94078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 -- 172.21.15.87:0/4150304195 shutdown_connections 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.472+0000 7f4c98f2b640 1 -- 172.21.15.87:0/4150304195 wait complete. 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 Processor -- start 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 -- start start 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 0x7f4c94197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 0x7f4c9419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941111d0 con 0x7f4c94101130 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c94111050 con 0x7f4c94101ad0 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c98f2b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c94111350 con 0x7f4c94105ea0 2024-04-22T13:43:22.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c9259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c9259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33720/0 (socket says 172.21.15.87:33720) 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c9259c640 1 -- 172.21.15.87:0/3013153959 learned_addr learned my addr 172.21.15.87:0/3013153959 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 0x7f4c94197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.473+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 0x7f4c9419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 -- 172.21.15.87:0/3013153959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 msgr2=0x7f4c9419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 0x7f4c9419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 -- 172.21.15.87:0/3013153959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 msgr2=0x7f4c94197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 0x7f4c94197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 -- 172.21.15.87:0/3013153959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9419c730 con 0x7f4c94101130 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 0x7f4c9419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c9259c640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f4c84006010 tx=0x7f4c84004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c8b7fe640 1 -- 172.21.15.87:0/3013153959 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84043020 con 0x7f4c94101130 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c8b7fe640 1 -- 172.21.15.87:0/3013153959 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c84004440 con 0x7f4c94101130 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9419c9c0 con 0x7f4c94101130 2024-04-22T13:43:22.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c8b7fe640 1 -- 172.21.15.87:0/3013153959 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84003b50 con 0x7f4c94101130 2024-04-22T13:43:22.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.474+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c941a42b0 con 0x7f4c94101130 2024-04-22T13:43:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.476+0000 7f4c8b7fe640 1 -- 172.21.15.87:0/3013153959 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4c84046020 con 0x7f4c94101130 2024-04-22T13:43:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.476+0000 7f4c9bbf7740 1 Processor -- start 2024-04-22T13:43:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 -- start start 2024-04-22T13:43:22.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354c30 0x55df3d35d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d35d810 0x55df3d35fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55df3d2b9500 con 0x55df3d354310 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55df3d25f6a0 con 0x55df3d354c30 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55df3d2b54c0 con 0x55df3d35d810 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c8b7fe640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 0x7f4c800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33726/0 (socket says 172.21.15.87:33726) 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3230979873 learned_addr learned my addr 172.21.15.87:0/3230979873 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d35d810 0x55df3d35fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c9259c640 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354c30 0x55df3d35d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 0x7f4c800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c8b7fe640 1 -- 172.21.15.87:0/3013153959 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f4c840f2360 con 0x7f4c94101130 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3230979873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d35d810 msgr2=0x55df3d35fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d35d810 0x55df3d35fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3230979873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354c30 msgr2=0x55df3d35d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354c30 0x55df3d35d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.477+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3230979873 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3d27e5e0 con 0x55df3d354310 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c012920 tx=0x7f4c8c012de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29544f9c17546c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c88ff9640 1 -- 172.21.15.87:0/3230979873 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c022070 con 0x55df3d354310 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c88ff9640 1 -- 172.21.15.87:0/3230979873 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c012030 con 0x55df3d354310 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c88ff9640 1 -- 172.21.15.87:0/3230979873 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c004730 con 0x55df3d354310 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3230979873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 msgr2=0x55df3d3546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 0x7f4c800a61e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c020000 tx=0x7f4c8c01f000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c012920 tx=0x7f4c8c012de0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3230979873 shutdown_connections 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d35d810 0x55df3d35fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354c30 0x55df3d35d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3230979873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3546f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.478+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3230979873 >> 172.21.15.87:0/3230979873 conn(0x55df3d34ea40 msgr2=0x55df3d34ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3230979873 shutdown_connections 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3230979873 wait complete. 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- start start 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 Processor -- start 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d35d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d35a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55df3d360600 con 0x55df3d35d810 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55df3d272a60 con 0x55df3d354310 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55df3d360780 con 0x55df3d354c30 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9bbf7740 10 client.0 _ll_register_callbacks cb 0x55df3cfcb110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33742/0 (socket says 172.21.15.87:33742) 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c92d9d640 1 -- 172.21.15.87:0/1357583328 learned_addr learned my addr 172.21.15.87:0/1357583328 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.479+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d35d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d35a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 -- 172.21.15.87:0/1357583328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 msgr2=0x55df3d35a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d35a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 -- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 msgr2=0x55df3d35d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d35d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 -- 172.21.15.87:0/1357583328 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3cfcb920 con 0x55df3d35d810 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d35a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c003cc0 tx=0x7f4c8c004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1174fb3de861ad7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c036060 con 0x55df3d35d810 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8c012030 con 0x55df3d35d810 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f4c8c004390 con 0x55df3d35d810 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4c8c022050 con 0x55df3d35d810 2024-04-22T13:43:22.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c9bbf7740 10 client.19119 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55df3d097d30 con 0x55df3d35d810 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c012030 con 0x55df3d35d810 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4c8c026c40 con 0x55df3d35d810 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c9bbf7740 10 client.19119 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.480+0000 7f4c9bbf7740 10 client.19119 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9bbf7740 20 client.19119 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d36d8a0 0x55df3d36fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9bbf7740 4 client.19119 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df3d28ca10 con 0x55df3d36d8a0 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d36d8a0 0x55df3d36fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c79ffb640 10 client.19119 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.481+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d36d8a0 0x55df3d36fc90 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da1972cb911cc187 server_cookie=f983801fea973b53 in_seq=0 out_seq=0 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1357583328 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df3d28ca10 con 0x55df3d36d8a0 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c79ffb640 10 client.19119 handle_command_reply: tid=0 2024-04-22T13:43:22.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 client.19119 shutdown 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 msgr2=0x55df3d35ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c003cc0 tx=0x7f4c8c004a70 comp rx=0 tx=0).stop 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 shutdown_connections 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d36d8a0 0x55df3d36fc90 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d35ad50 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d35a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c79ffb640 0 client.19119 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1357583328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d35d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 >> 172.21.15.87:0/1357583328 conn(0x55df3d34ea40 msgr2=0x55df3d350790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 shutdown_connections 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1357583328 wait complete. 2024-04-22T13:43:22.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.482+0000 7f4c9bbf7740 20 client.19119 trim_cache size 0 max 16384 2024-04-22T13:43:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 Processor -- start 2024-04-22T13:43:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 -- start start 2024-04-22T13:43:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d3694f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d3505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941af1c0 con 0x55df3d35d810 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941aca90 con 0x55df3d354310 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941aa900 con 0x55df3d354c30 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33746/0 (socket says 172.21.15.87:33746) 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3491734627 learned_addr learned my addr 172.21.15.87:0/3491734627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d3505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c9259c640 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d3694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3491734627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 msgr2=0x55df3d3505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d3505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3491734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 msgr2=0x55df3d3694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d3694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.499+0000 7f4c92d9d640 1 -- 172.21.15.87:0/3491734627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3d36a160 con 0x55df3d35d810 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00afb0 tx=0x7f4c8c00a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f7abd5b6879d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c7bfff640 1 -- 172.21.15.87:0/3491734627 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c036070 con 0x55df3d35d810 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c7bfff640 1 -- 172.21.15.87:0/3491734627 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c003cf0 con 0x55df3d35d810 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c7bfff640 1 -- 172.21.15.87:0/3491734627 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c004190 con 0x55df3d35d810 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3491734627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 msgr2=0x55df3d3754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00afb0 tx=0x7f4c8c00a8b0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3491734627 shutdown_connections 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d35d810 0x55df3d3754c0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d3505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/3491734627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d354310 0x55df3d3694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3491734627 >> 172.21.15.87:0/3491734627 conn(0x55df3d34ea40 msgr2=0x55df3d34d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3491734627 shutdown_connections 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.500+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/3491734627 wait complete. 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 -- start start 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 Processor -- start 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d360f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d35d810 0x55df3d361440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941aaf50 con 0x55df3d354310 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941ac3c0 con 0x55df3d35d810 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941ab620 con 0x55df3d354c30 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9bbf7740 10 client.0 _ll_register_callbacks cb 0x7f4c7c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33748/0 (socket says 172.21.15.87:33748) 2024-04-22T13:43:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 -- 172.21.15.87:0/1260730111 learned_addr learned my addr 172.21.15.87:0/1260730111 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d360f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c92d9d640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d35d810 0x55df3d361440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 -- 172.21.15.87:0/1260730111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 msgr2=0x55df3d360f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d360f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 -- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d35d810 msgr2=0x55df3d361440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d35d810 0x55df3d361440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 -- 172.21.15.87:0/1260730111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3d3d3eb0 con 0x55df3d354310 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c9259c640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4c840be150 tx=0x7f4c8404ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddb1b0f403027da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.501+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c840be450 con 0x55df3d354310 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f4c8404ed40 con 0x55df3d354310 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8c003cf0 con 0x55df3d354310 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4c840c2800 con 0x55df3d354310 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 10 client.19125 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:22.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55df3d3d5230 con 0x55df3d354310 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c003cf0 con 0x55df3d354310 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4c840efd70 con 0x55df3d354310 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 10 client.19125 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 10 client.19125 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 20 client.19125 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d3d5900 0x55df3d3d7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 4 client.19125 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b582c357", "release": true, "await": true}] 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b582c357", "release": true, "await": true}) -- 0x55df3d2c6f20 con 0x55df3d3d5900 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.502+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d3d5900 0x55df3d3d7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.503+0000 7f4c79ffb640 10 client.19125 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.503+0000 7f4c8bfff640 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d3d5900 0x55df3d3d7cf0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9326ff517e22b19 server_cookie=1e7fcd7c371a066 in_seq=0 out_seq=0 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c79ffb640 1 -- 172.21.15.87:0/1260730111 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55df3d2c6f20 con 0x55df3d3d5900 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c79ffb640 10 client.19125 handle_command_reply: tid=0 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 client.19125 shutdown 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 msgr2=0x55df3d3609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4c840be150 tx=0x7f4c8404ecd0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 shutdown_connections 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55df3d3d5900 0x55df3d3d7cf0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df3d35d810 0x55df3d361440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c79ffb640 0 client.19125 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df3d354c30 0x55df3d360f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 --2- 172.21.15.87:0/1260730111 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55df3d354310 0x55df3d3609c0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 >> 172.21.15.87:0/1260730111 conn(0x55df3d34ea40 msgr2=0x55df3d350430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 shutdown_connections 2024-04-22T13:43:22.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 1 -- 172.21.15.87:0/1260730111 wait complete. 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c9bbf7740 20 client.19125 trim_cache size 0 max 16384 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 44, 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": { 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "b582c357": { 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "version": 44, 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "age_ref": 0.0, 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:43:22.508 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "timeout": 60.0, 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "expiration": 79.7, 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "members": { 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "file:/": { 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "excluded": false, 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "state": { 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RELEASED", 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 0.0 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:22.509 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:22.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 msgr2=0x7f4c800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 0x7f4c800a61e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c020000 tx=0x7f4c8c01f000 comp rx=0 tx=0).stop 2024-04-22T13:43:22.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 msgr2=0x7f4c94197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f4c84006010 tx=0x7f4c84004040 comp rx=0 tx=0).stop 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 shutdown_connections 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f4c800a3d20 0x7f4c800a61e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c94105ea0 0x7f4c9419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c94101ad0 0x7f4c94197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 --2- 172.21.15.87:0/3013153959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c94101130 0x7f4c94197790 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 >> 172.21.15.87:0/3013153959 conn(0x7f4c94077e30 msgr2=0x7f4c940fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.507+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 shutdown_connections 2024-04-22T13:43:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:22.508+0000 7f4c98f2b640 1 -- 172.21.15.87:0/3013153959 wait complete. 2024-04-22T13:43:22.514 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: b582c357, seconds elapsed: 20.8 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 -- 172.21.15.87:0/3787104977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100240 msgr2=0x7f383c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 --2- 172.21.15.87:0/3787104977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100240 0x7f383c1006a0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f382c009900 tx=0x7f382c02bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 -- 172.21.15.87:0/3787104977 shutdown_connections 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 --2- 172.21.15.87:0/3787104977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c100be0 0x7f383c105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 --2- 172.21.15.87:0/3787104977 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100240 0x7f383c1006a0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 --2- 172.21.15.87:0/3787104977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c106d10 0x7f383c1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 -- 172.21.15.87:0/3787104977 >> 172.21.15.87:0/3787104977 conn(0x7f383c0771c0 msgr2=0x7f383c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 -- 172.21.15.87:0/3787104977 shutdown_connections 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.794+0000 7f3842b9d640 1 -- 172.21.15.87:0/3787104977 wait complete. 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 Processor -- start 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 -- start start 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c100240 0x7f383c1a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 0x7f383c19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f383c118680 con 0x7f383c100be0 2024-04-22T13:43:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f383c118500 con 0x7f383c106d10 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f3842b9d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f383c118800 con 0x7f383c100240 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33762/0 (socket says 172.21.15.87:33762) 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.795+0000 7f384139a640 1 -- 172.21.15.87:0/4291572273 learned_addr learned my addr 172.21.15.87:0/4291572273 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3841b9b640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c100240 0x7f383c1a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384239c640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 0x7f383c19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 -- 172.21.15.87:0/4291572273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c100240 msgr2=0x7f383c1a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c100240 0x7f383c1a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 -- 172.21.15.87:0/4291572273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 msgr2=0x7f383c19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 0x7f383c19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 -- 172.21.15.87:0/4291572273 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c19f120 con 0x7f383c100be0 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384239c640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 0x7f383c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f384139a640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c02d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3832ffd640 1 -- 172.21.15.87:0/4291572273 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c043020 con 0x7f383c100be0 2024-04-22T13:43:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c19f3b0 con 0x7f383c100be0 2024-04-22T13:43:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3832ffd640 1 -- 172.21.15.87:0/4291572273 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c02dd50 con 0x7f383c100be0 2024-04-22T13:43:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3832ffd640 1 -- 172.21.15.87:0/4291572273 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c003af0 con 0x7f383c100be0 2024-04-22T13:43:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.796+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1ab6b0 con 0x7f383c100be0 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f3832ffd640 1 -- 172.21.15.87:0/4291572273 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f382c046020 con 0x7f383c100be0 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 Processor -- start 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 -- start start 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbe41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e8bb488d0 con 0x559e8bbed2d0 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e8bae5e80 con 0x559e8bbe3d50 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.799+0000 7f38469f2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e8bb44890 con 0x559e8bbe4ac0 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33772/0 (socket says 172.21.15.87:33772) 2024-04-22T13:43:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 -- 172.21.15.87:0/2993661317 learned_addr learned my addr 172.21.15.87:0/2993661317 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384239c640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3832ffd640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 0x7f38280a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3841b9b640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbe41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384239c640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 0x7f38280a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3832ffd640 1 -- 172.21.15.87:0/4291572273 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f382c0f2820 con 0x7f383c100be0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 -- 172.21.15.87:0/2993661317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe4ac0 msgr2=0x559e8bbe3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 -- 172.21.15.87:0/2993661317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 msgr2=0x559e8bbe41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbe41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 -- 172.21.15.87:0/2993661317 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8bb07320 con 0x559e8bbed2d0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3841b9b640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbe41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384139a640 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f382c0be730 tx=0x7f382c02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d87feb340a6f156 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3827fff640 1 -- 172.21.15.87:0/2993661317 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0be800 con 0x559e8bbed2d0 2024-04-22T13:43:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f384239c640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 0x7f38280a61d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3838005d00 tx=0x7f3838004730 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3827fff640 1 -- 172.21.15.87:0/2993661317 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c0bf130 con 0x559e8bbed2d0 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.800+0000 7f3827fff640 1 -- 172.21.15.87:0/2993661317 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0be390 con 0x559e8bbed2d0 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 -- 172.21.15.87:0/2993661317 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 msgr2=0x559e8bbef6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f382c0be730 tx=0x7f382c02bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 -- 172.21.15.87:0/2993661317 shutdown_connections 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbef6c0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbe41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 --2- 172.21.15.87:0/2993661317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 -- 172.21.15.87:0/2993661317 >> 172.21.15.87:0/2993661317 conn(0x559e8bbaac90 msgr2=0x559e8bbdea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 -- 172.21.15.87:0/2993661317 shutdown_connections 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.801+0000 7f38469f2740 1 -- 172.21.15.87:0/2993661317 wait complete. 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 -- start start 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 Processor -- start 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x559e8bae5e80 con 0x559e8bbed2d0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e8bad0eb0 con 0x559e8bbe4ac0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e8bb488d0 con 0x559e8bbe3d50 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f38469f2740 10 client.0 _ll_register_callbacks cb 0x559e8bafdb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33786/0 (socket says 172.21.15.87:33786) 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 -- 172.21.15.87:0/3619272841 learned_addr learned my addr 172.21.15.87:0/3619272841 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f3841b9b640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384139a640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 -- 172.21.15.87:0/3619272841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 msgr2=0x559e8bbecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 -- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 msgr2=0x559e8bbe9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384239c640 1 -- 172.21.15.87:0/3619272841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8b85b210 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.802+0000 7f384139a640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f384239c640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3838003ff0 tx=0x7f38380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad026a8a3f6c6bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838023620 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f3838014410 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c0bf130 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3838014610 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 10 client.19137 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e8b927620 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c0bf130 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f383800c070 con 0x559e8bbed2d0 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 10 client.19137 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 10 client.19137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 20 client.19137 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bbfce60 0x559e8bbff250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 4 client.19137 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.803+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e8bb5c880 con 0x559e8bbfce60 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.804+0000 7f3841b9b640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bbfce60 0x559e8bbff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.804+0000 7f3841b9b640 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bbfce60 0x559e8bbff250 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45576bd147726b23 server_cookie=93e8c988606d7e38 in_seq=0 out_seq=0 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.804+0000 7f38257fa640 10 client.19137 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38257fa640 1 -- 172.21.15.87:0/3619272841 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e8bb5c880 con 0x559e8bbfce60 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38257fa640 10 client.19137 handle_command_reply: tid=0 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 client.19137 shutdown 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 msgr2=0x559e8bbea530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3838003ff0 tx=0x7f38380046c0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 shutdown_connections 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bbfce60 0x559e8bbff250 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbed2d0 0x559e8bbea530 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe4ac0 0x559e8bbe9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38257fa640 0 client.19137 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 --2- 172.21.15.87:0/3619272841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 >> 172.21.15.87:0/3619272841 conn(0x559e8bbaac90 msgr2=0x559e8bbe0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 shutdown_connections 2024-04-22T13:43:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 1 -- 172.21.15.87:0/3619272841 wait complete. 2024-04-22T13:43:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.805+0000 7f38469f2740 20 client.19137 trim_cache size 0 max 16384 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 Processor -- start 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 -- start start 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbf89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 0x559e8bbecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b68c0 con 0x559e8bbe4ac0 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b4190 con 0x559e8bbed2d0 2024-04-22T13:43:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38469f2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b2000 con 0x559e8bbe3d50 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33796/0 (socket says 172.21.15.87:33796) 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 -- 172.21.15.87:0/1244434658 learned_addr learned my addr 172.21.15.87:0/1244434658 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 -- 172.21.15.87:0/1244434658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 msgr2=0x559e8bbf89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f3841b9b640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbf89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384239c640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 0x559e8bbecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbf89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 -- 172.21.15.87:0/1244434658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 msgr2=0x559e8bbecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 0x559e8bbecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 -- 172.21.15.87:0/1244434658 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8bbff8d0 con 0x559e8bbe4ac0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384239c640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 0x559e8bbecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f384139a640 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f382c0f0170 tx=0x7f382c005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3093b4b1a05dd3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38277fe640 1 -- 172.21.15.87:0/1244434658 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0be570 con 0x559e8bbe4ac0 2024-04-22T13:43:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38277fe640 1 -- 172.21.15.87:0/1244434658 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c0bf130 con 0x559e8bbe4ac0 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.820+0000 7f38277fe640 1 -- 172.21.15.87:0/1244434658 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c0f0230 con 0x559e8bbe4ac0 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- 172.21.15.87:0/1244434658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 msgr2=0x559e8bbec700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f382c0f0170 tx=0x7f382c005e20 comp rx=0 tx=0).stop 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- 172.21.15.87:0/1244434658 shutdown_connections 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbed2d0 0x559e8bbecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbec700 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 --2- 172.21.15.87:0/1244434658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbe3d50 0x559e8bbf89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- 172.21.15.87:0/1244434658 >> 172.21.15.87:0/1244434658 conn(0x559e8bbaac90 msgr2=0x559e8bbe5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- 172.21.15.87:0/1244434658 shutdown_connections 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- 172.21.15.87:0/1244434658 wait complete. 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 -- start start 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.821+0000 7f38469f2740 1 Processor -- start 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbf4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 0x559e8bbf1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b2650 con 0x559e8bbe4ac0 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b3ac0 con 0x559e8bbe3d50 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1b2d20 con 0x559e8bbed2d0 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38469f2740 10 client.0 _ll_register_callbacks cb 0x7f3838003c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33804/0 (socket says 172.21.15.87:33804) 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 -- 172.21.15.87:0/2523559684 learned_addr learned my addr 172.21.15.87:0/2523559684 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f3841b9b640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbf4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384239c640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 0x559e8bbf1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 -- 172.21.15.87:0/2523559684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 msgr2=0x559e8bbf1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 0x559e8bbf1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 -- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 msgr2=0x559e8bbf4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbf4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 -- 172.21.15.87:0/2523559684 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8bbe3020 con 0x559e8bbe4ac0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384239c640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 0x559e8bbf1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f384139a640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f382c0bb400 tx=0x7f382c050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4ced4057a6ef8d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.822+0000 7f38257fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c087020 con 0x559e8bbe4ac0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f382c003c90 con 0x559e8bbe4ac0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c0bf130 con 0x559e8bbe4ac0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f382c0be800 con 0x559e8bbe4ac0 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 10 client.19143 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e8bc63d50 con 0x559e8bbe4ac0 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c0bf130 con 0x559e8bbe4ac0 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f382c0c7210 con 0x559e8bbe4ac0 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 10 client.19143 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 10 client.19143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 20 client.19143 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bc64230 0x559e8bc66620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 4 client.19143 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e8bae83d0 con 0x559e8bc64230 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.823+0000 7f3841b9b640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bc64230 0x559e8bc66620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.824+0000 7f38257fa640 10 client.19143 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.824+0000 7f3841b9b640 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bc64230 0x559e8bc66620 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83ec10bfeed3b94 server_cookie=a93bd61289dad8a in_seq=0 out_seq=0 2024-04-22T13:43:22.826 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38257fa640 1 -- 172.21.15.87:0/2523559684 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e8bae83d0 con 0x559e8bc64230 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38257fa640 10 client.19143 handle_command_reply: tid=0 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 client.19143 shutdown 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 msgr2=0x559e8bbf5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f382c0bb400 tx=0x7f382c050fc0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 shutdown_connections 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x559e8bc64230 0x559e8bc66620 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8bbed2d0 0x559e8bbf1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x559e8bbe4ac0 0x559e8bbf5440 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38257fa640 0 client.19143 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.829+0000 7f38469f2740 1 --2- 172.21.15.87:0/2523559684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e8bbe3d50 0x559e8bbf4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 >> 172.21.15.87:0/2523559684 conn(0x559e8bbaac90 msgr2=0x559e8bbdd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 shutdown_connections 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f38469f2740 1 -- 172.21.15.87:0/2523559684 wait complete. 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f38469f2740 20 client.19143 trim_cache size 0 max 16384 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:43:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:43:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 msgr2=0x7f38280a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 0x7f38280a61d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3838005d00 tx=0x7f3838004730 comp rx=0 tx=0).stop 2024-04-22T13:43:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 msgr2=0x7f383c1a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c02d890 comp rx=0 tx=0).stop 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 shutdown_connections 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f38280a3d10 0x7f38280a61d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f383c106d10 0x7f383c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f383c100be0 0x7f383c1a4d30 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 --2- 172.21.15.87:0/4291572273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f383c100240 0x7f383c1a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 >> 172.21.15.87:0/4291572273 conn(0x7f383c0771c0 msgr2=0x7f383c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 shutdown_connections 2024-04-22T13:43:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:22.830+0000 7f3842b9d640 1 -- 172.21.15.87:0/4291572273 wait complete. 2024-04-22T13:43:22.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:22 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3080761071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:22.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:22 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3080761071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:22.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:22 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3080761071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:23.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:23 smithi120 ceph-mon[31101]: pgmap v361: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:23.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:23 smithi132 ceph-mon[31531]: pgmap v361: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:23.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:23 smithi087 ceph-mon[27999]: pgmap v361: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 716 B/s wr, 10 op/s 2024-04-22T13:43:25.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:25 smithi120 ceph-mon[31101]: pgmap v362: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 307 B/s wr, 9 op/s 2024-04-22T13:43:25.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:25 smithi132 ceph-mon[31531]: pgmap v362: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 307 B/s wr, 9 op/s 2024-04-22T13:43:25.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:25 smithi087 ceph-mon[27999]: pgmap v362: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 307 B/s wr, 9 op/s 2024-04-22T13:43:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:27 smithi120 ceph-mon[31101]: pgmap v363: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 261 B/s wr, 8 op/s 2024-04-22T13:43:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:27 smithi132 ceph-mon[31531]: pgmap v363: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 261 B/s wr, 8 op/s 2024-04-22T13:43:27.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:27 smithi087 ceph-mon[27999]: pgmap v363: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 261 B/s wr, 8 op/s 2024-04-22T13:43:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:29 smithi120 ceph-mon[31101]: pgmap v364: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:43:29.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:29 smithi132 ceph-mon[31531]: pgmap v364: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:43:29.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:29 smithi087 ceph-mon[27999]: pgmap v364: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T13:43:31.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:31 smithi120 ceph-mon[31101]: pgmap v365: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:43:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:31 smithi132 ceph-mon[31531]: pgmap v365: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:43:31.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:31 smithi087 ceph-mon[27999]: pgmap v365: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T13:43:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:33 smithi120 ceph-mon[31101]: pgmap v366: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:33.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:33 smithi132 ceph-mon[31531]: pgmap v366: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:33.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:33 smithi087 ceph-mon[27999]: pgmap v366: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:35.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:35 smithi120 ceph-mon[31101]: pgmap v367: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 -- 172.21.15.87:0/4274141518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93041072e0 msgr2=0x7f930410ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 --2- 172.21.15.87:0/4274141518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93041072e0 0x7f930410ee80 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009ca0 tx=0x7f92fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 -- 172.21.15.87:0/4274141518 shutdown_connections 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 --2- 172.21.15.87:0/4274141518 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93041072e0 0x7f930410ee80 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 --2- 172.21.15.87:0/4274141518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f930406b3e0 0x7f9304106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 --2- 172.21.15.87:0/4274141518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f930406aa30 0x7f930406ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 -- 172.21.15.87:0/4274141518 >> 172.21.15.87:0/4274141518 conn(0x7f9304077ba0 msgr2=0x7f9304077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 -- 172.21.15.87:0/4274141518 shutdown_connections 2024-04-22T13:43:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 -- 172.21.15.87:0/4274141518 wait complete. 2024-04-22T13:43:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.926+0000 7f930a0d2640 1 Processor -- start 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 -- start start 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93041072e0 0x7f9304101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9304101c80 0x7f93041a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f9304112220 con 0x7f930406aa30 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93041120a0 con 0x7f93041072e0 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f930a0d2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93041123a0 con 0x7f9304101c80 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38802/0 (socket says 172.21.15.87:38802) 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 -- 172.21.15.87:0/266598685 learned_addr learned my addr 172.21.15.87:0/266598685 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f9302ffd640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93041072e0 0x7f9304101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f9303fff640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9304101c80 0x7f93041a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 -- 172.21.15.87:0/266598685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9304101c80 msgr2=0x7f93041a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9304101c80 0x7f93041a6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 -- 172.21.15.87:0/266598685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93041072e0 msgr2=0x7f9304101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93041072e0 0x7f9304101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.927+0000 7f93037fe640 1 -- 172.21.15.87:0/266598685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93041a6ef0 con 0x7f930406aa30 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f93037fe640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f92f400c970 tx=0x7f92f400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f9300ff9640 1 -- 172.21.15.87:0/266598685 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4007bb0 con 0x7f930406aa30 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93041a71e0 con 0x7f930406aa30 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f9300ff9640 1 -- 172.21.15.87:0/266598685 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f4007d50 con 0x7f930406aa30 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f9300ff9640 1 -- 172.21.15.87:0/266598685 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4004de0 con 0x7f930406aa30 2024-04-22T13:43:35.926 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.928+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93041a76e0 con 0x7f930406aa30 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.930+0000 7f9300ff9640 1 -- 172.21.15.87:0/266598685 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f92f4004f80 con 0x7f930406aa30 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.930+0000 7f930cda7740 1 Processor -- start 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.930+0000 7f930cda7740 1 -- start start 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed72bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed7350e0 0x5654ed7374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5654ed6908d0 con 0x5654ed72c8d0 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654ed62de80 con 0x5654ed7350e0 2024-04-22T13:43:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f930cda7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5654ed68c890 con 0x5654ed72bb60 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9300ff9640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 0x7f92ec0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38816/0 (socket says 172.21.15.87:38816) 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 -- 172.21.15.87:0/2338079792 learned_addr learned my addr 172.21.15.87:0/2338079792 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 0x7f92ec0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9300ff9640 1 -- 172.21.15.87:0/266598685 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f92f40d0540 con 0x7f930406aa30 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f93037fe640 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed72bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9302ffd640 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed7350e0 0x5654ed7374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 -- 172.21.15.87:0/2338079792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 msgr2=0x5654ed72bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed72bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 -- 172.21.15.87:0/2338079792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed7350e0 msgr2=0x5654ed7374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed7350e0 0x5654ed7374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.931+0000 7f9303fff640 1 -- 172.21.15.87:0/2338079792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654ed64f320 con 0x5654ed72c8d0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f9303fff640 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f92fc037040 tx=0x7f92fc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef28e5590f329e2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f9303fff640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 0x7f92ec0a6280 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009a60 tx=0x7f92fc042000 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f92e9ffb640 1 -- 172.21.15.87:0/2338079792 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc051070 con 0x5654ed72c8d0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f92e9ffb640 1 -- 172.21.15.87:0/2338079792 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92fc004040 con 0x5654ed72c8d0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f92e9ffb640 1 -- 172.21.15.87:0/2338079792 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc039040 con 0x5654ed72c8d0 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 -- 172.21.15.87:0/2338079792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 msgr2=0x5654ed72b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f92fc037040 tx=0x7f92fc032670 comp rx=0 tx=0).stop 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 -- 172.21.15.87:0/2338079792 shutdown_connections 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed7350e0 0x5654ed7374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed72bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 --2- 172.21.15.87:0/2338079792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72c8d0 0x5654ed72b620 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 -- 172.21.15.87:0/2338079792 >> 172.21.15.87:0/2338079792 conn(0x5654ed6f2c90 msgr2=0x5654ed726850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 -- 172.21.15.87:0/2338079792 shutdown_connections 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.932+0000 7f930cda7740 1 -- 172.21.15.87:0/2338079792 wait complete. 2024-04-22T13:43:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 -- start start 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 Processor -- start 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72bb60 0x5654ed734ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72c8d0 0x5654ed731e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5654ed737db0 con 0x5654ed7350e0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654ed618eb0 con 0x5654ed72bb60 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5654ed737f30 con 0x5654ed72c8d0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f930cda7740 10 client.0 _ll_register_callbacks cb 0x5654ed645b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38830/0 (socket says 172.21.15.87:38830) 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f9303fff640 1 -- 172.21.15.87:0/2186620118 learned_addr learned my addr 172.21.15.87:0/2186620118 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.933+0000 7f93037fe640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72bb60 0x5654ed734ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9302ffd640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72c8d0 0x5654ed731e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 -- 172.21.15.87:0/2186620118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72c8d0 msgr2=0x5654ed731e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72c8d0 0x5654ed731e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 -- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72bb60 msgr2=0x5654ed734ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72bb60 0x5654ed734ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 -- 172.21.15.87:0/2186620118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ed3a3210 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f9303fff640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f92fc02c630 tx=0x7f92fc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5da6bac963fd136 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f92e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc039050 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f92fc0041f0 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92fc004040 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f92fc051040 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f930cda7740 10 client.19155 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5654ed46f620 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.934+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92fc004040 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f92fc04b030 con 0x5654ed7350e0 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 10 client.19155 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 10 client.19155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 20 client.19155 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed744f60 0x5654ed747350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 4 client.19155 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654ed6a4880 con 0x5654ed744f60 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.935+0000 7f93037fe640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed744f60 0x5654ed747350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f93037fe640 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed744f60 0x5654ed747350 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50c6b1798641d313 server_cookie=7329852c892802bc in_seq=0 out_seq=0 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f92e2ffd640 10 client.19155 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f92e2ffd640 1 -- 172.21.15.87:0/2186620118 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654ed6a4880 con 0x5654ed744f60 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f92e2ffd640 10 client.19155 handle_command_reply: tid=0 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f930cda7740 1 client.19155 shutdown 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 msgr2=0x5654ed732480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.936+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f92fc02c630 tx=0x7f92fc0095c0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 shutdown_connections 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed744f60 0x5654ed747350 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed732480 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f92e2ffd640 0 client.19155 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72c8d0 0x5654ed731e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 --2- 172.21.15.87:0/2186620118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72bb60 0x5654ed734ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 >> 172.21.15.87:0/2186620118 conn(0x5654ed6f2c90 msgr2=0x5654ed7285a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 shutdown_connections 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 1 -- 172.21.15.87:0/2186620118 wait complete. 2024-04-22T13:43:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.937+0000 7f930cda7740 20 client.19155 trim_cache size 0 max 16384 2024-04-22T13:43:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 Processor -- start 2024-04-22T13:43:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 -- start start 2024-04-22T13:43:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed740cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed7345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93041b24b0 con 0x5654ed7350e0 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93041afd80 con 0x5654ed72c8d0 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.951+0000 7f930cda7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93041adbf0 con 0x5654ed72bb60 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38846/0 (socket says 172.21.15.87:38846) 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 -- 172.21.15.87:0/1101048398 learned_addr learned my addr 172.21.15.87:0/1101048398 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9302ffd640 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed7345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f93037fe640 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed740cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 -- 172.21.15.87:0/1101048398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 msgr2=0x5654ed740cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed740cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 -- 172.21.15.87:0/1101048398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 msgr2=0x5654ed7345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed7345a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 -- 172.21.15.87:0/1101048398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654ed72ae30 con 0x5654ed7350e0 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f9303fff640 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f92fc006010 tx=0x7f92fc055410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15584f705c69e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f92e97fa640 1 -- 172.21.15.87:0/1101048398 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc039070 con 0x5654ed7350e0 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f92e97fa640 1 -- 172.21.15.87:0/1101048398 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92fc02ce20 con 0x5654ed7350e0 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.952+0000 7f92e97fa640 1 -- 172.21.15.87:0/1101048398 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0042a0 con 0x5654ed7350e0 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- 172.21.15.87:0/1101048398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 msgr2=0x5654ed734ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f92fc006010 tx=0x7f92fc055410 comp rx=0 tx=0).stop 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- 172.21.15.87:0/1101048398 shutdown_connections 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed7350e0 0x5654ed734ae0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed7345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 --2- 172.21.15.87:0/1101048398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed72bb60 0x5654ed740cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- 172.21.15.87:0/1101048398 >> 172.21.15.87:0/1101048398 conn(0x5654ed6f2c90 msgr2=0x5654ed730740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- 172.21.15.87:0/1101048398 shutdown_connections 2024-04-22T13:43:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- 172.21.15.87:0/1101048398 wait complete. 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.953+0000 7f930cda7740 1 -- start start 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 Processor -- start 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed73d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed7350e0 0x5654ed7361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f93041ae240 con 0x5654ed72bb60 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93041af6b0 con 0x5654ed72c8d0 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f93041ae910 con 0x5654ed7350e0 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f930cda7740 10 client.0 _ll_register_callbacks cb 0x7f92fc04f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38850/0 (socket says 172.21.15.87:38850) 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 -- 172.21.15.87:0/405950323 learned_addr learned my addr 172.21.15.87:0/405950323 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f9303fff640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed7350e0 0x5654ed7361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f9302ffd640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed73d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 -- 172.21.15.87:0/405950323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed7350e0 msgr2=0x5654ed7361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed7350e0 0x5654ed7361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 -- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 msgr2=0x5654ed73d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed73d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 -- 172.21.15.87:0/405950323 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ed7477b0 con 0x5654ed72bb60 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f9302ffd640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed73d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f93037fe640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f92f4098920 tx=0x7f92f400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da78a70ae45e5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f92e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.954+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f40ba060 con 0x5654ed72bb60 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f92f4098b30 con 0x5654ed72bb60 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92fc02ce20 con 0x5654ed72bb60 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f92f401d040 con 0x5654ed72bb60 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 10 client.19161 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5654ed7ab830 con 0x5654ed72bb60 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92fc02ce20 con 0x5654ed72bb60 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f92f40bb250 con 0x5654ed72bb60 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 10 client.19161 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 10 client.19161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 20 client.19161 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed7abee0 0x5654ed7ae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 4 client.19161 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654ed6303d0 con 0x5654ed7abee0 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.955+0000 7f9302ffd640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed7abee0 0x5654ed7ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.956+0000 7f92e2ffd640 10 client.19161 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.956+0000 7f9302ffd640 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed7abee0 0x5654ed7ae300 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fadf991af38a8 server_cookie=ef4add92fb0a0d15 in_seq=0 out_seq=0 2024-04-22T13:43:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f92e2ffd640 1 -- 172.21.15.87:0/405950323 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654ed6303d0 con 0x5654ed7abee0 2024-04-22T13:43:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f92e2ffd640 10 client.19161 handle_command_reply: tid=0 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 client.19161 shutdown 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 msgr2=0x5654ed73d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f92f4098920 tx=0x7f92f400cf60 comp rx=0 tx=0).stop 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 shutdown_connections 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5654ed7abee0 0x5654ed7ae300 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654ed7350e0 0x5654ed7361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654ed72c8d0 0x5654ed73d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f92e2ffd640 0 client.19161 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 --2- 172.21.15.87:0/405950323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5654ed72bb60 0x5654ed73d1c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 >> 172.21.15.87:0/405950323 conn(0x5654ed6f2c90 msgr2=0x5654ed725040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 shutdown_connections 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.961+0000 7f930cda7740 1 -- 172.21.15.87:0/405950323 wait complete. 2024-04-22T13:43:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930cda7740 20 client.19161 trim_cache size 0 max 16384 2024-04-22T13:43:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:43:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:43:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:43:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 msgr2=0x7f92ec0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 0x7f92ec0a6280 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009a60 tx=0x7f92fc042000 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 msgr2=0x7f9304101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f92f400c970 tx=0x7f92f400cd70 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 shutdown_connections 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f92ec0a3dc0 0x7f92ec0a6280 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9304101c80 0x7f93041a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93041072e0 0x7f9304101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 --2- 172.21.15.87:0/266598685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f930406aa30 0x7f9304101200 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 >> 172.21.15.87:0/266598685 conn(0x7f9304077ba0 msgr2=0x7f93040ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 shutdown_connections 2024-04-22T13:43:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:35.962+0000 7f930a0d2640 1 -- 172.21.15.87:0/266598685 wait complete. 2024-04-22T13:43:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:35 smithi132 ceph-mon[31531]: pgmap v367: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:35.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:35 smithi087 ceph-mon[27999]: pgmap v367: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:37.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:37 smithi120 ceph-mon[31101]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:43:37.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:37 smithi132 ceph-mon[31531]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:43:37.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:37 smithi087 ceph-mon[27999]: pgmap v368: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:43:39.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:39 smithi120 ceph-mon[31101]: pgmap v369: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:39.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:39 smithi132 ceph-mon[31531]: pgmap v369: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:39.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:39 smithi087 ceph-mon[27999]: pgmap v369: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:41.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:41 smithi120 ceph-mon[31101]: pgmap v370: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:41.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:41 smithi132 ceph-mon[31531]: pgmap v370: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:41.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:41 smithi087 ceph-mon[27999]: pgmap v370: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:43.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:43 smithi120 ceph-mon[31101]: pgmap v371: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:43.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:43 smithi132 ceph-mon[31531]: pgmap v371: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:43.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:43 smithi087 ceph-mon[27999]: pgmap v371: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:45.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:43:45.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 -- 172.21.15.87:0/3187638649 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198106ed0 msgr2=0x7f719806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 --2- 172.21.15.87:0/3187638649 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198106ed0 0x7f719806a5f0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f7194009900 tx=0x7f719402bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 -- 172.21.15.87:0/3187638649 shutdown_connections 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 --2- 172.21.15.87:0/3187638649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7198104960 0x7f7198104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 --2- 172.21.15.87:0/3187638649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f719806ab30 0x7f719806af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 --2- 172.21.15.87:0/3187638649 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198106ed0 0x7f719806a5f0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 -- 172.21.15.87:0/3187638649 >> 172.21.15.87:0/3187638649 conn(0x7f7198077c40 msgr2=0x7f7198078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 -- 172.21.15.87:0/3187638649 shutdown_connections 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.695+0000 7f719fd67640 1 -- 172.21.15.87:0/3187638649 wait complete. 2024-04-22T13:43:45.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 Processor -- start 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 -- start start 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 0x7f71981a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7198106ed0 0x7f719819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f71981186a0 con 0x7f7198104960 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7198118520 con 0x7f719806ab30 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719fd67640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7198118820 con 0x7f7198106ed0 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55724/0 (socket says 172.21.15.87:55724) 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 -- 172.21.15.87:0/1361185633 learned_addr learned my addr 172.21.15.87:0/1361185633 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719e304640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7198106ed0 0x7f719819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719db03640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 0x7f71981a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 -- 172.21.15.87:0/1361185633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7198106ed0 msgr2=0x7f719819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7198106ed0 0x7f719819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 -- 172.21.15.87:0/1361185633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 msgr2=0x7f71981a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 0x7f71981a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719d302640 1 -- 172.21.15.87:0/1361185633 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719819f120 con 0x7f7198104960 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.696+0000 7f719db03640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 0x7f71981a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f719d302640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f718800d950 tx=0x7f718800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f718effd640 1 -- 172.21.15.87:0/1361185633 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188017030 con 0x7f7198104960 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f718effd640 1 -- 172.21.15.87:0/1361185633 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7188004020 con 0x7f7198104960 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719819f410 con 0x7f7198104960 2024-04-22T13:43:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f718effd640 1 -- 172.21.15.87:0/1361185633 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880047a0 con 0x7f7198104960 2024-04-22T13:43:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.697+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71981ab6b0 con 0x7f7198104960 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.699+0000 7f718effd640 1 -- 172.21.15.87:0/1361185633 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7188009a30 con 0x7f7198104960 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 Processor -- start 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 -- start start 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 0x55dd455cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55dd455251e0 con 0x55dd455c0ea0 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dd454c2390 con 0x55dd455c9a80 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f71a2a39740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dd455211a0 con 0x55dd455c0580 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f718effd640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 0x7f71840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55730/0 (socket says 172.21.15.87:55730) 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 -- 172.21.15.87:0/119056870 learned_addr learned my addr 172.21.15.87:0/119056870 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719e304640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 0x7f71840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719d302640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 0x55dd455cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719e304640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f718effd640 1 -- 172.21.15.87:0/1361185633 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f71880d0320 con 0x7f7198104960 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 -- 172.21.15.87:0/119056870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 msgr2=0x55dd455c0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 -- 172.21.15.87:0/119056870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 msgr2=0x55dd455cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 0x55dd455cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719db03640 1 -- 172.21.15.87:0/119056870 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd453017d0 con 0x55dd455c0ea0 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.700+0000 7f719d302640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 0x55dd455cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.701+0000 7f719e304640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 0x7f71840a61e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7190005c40 tx=0x7f71900129b0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.701+0000 7f719db03640 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7194036040 tx=0x7f719402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815e00a5ff83723a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.701+0000 7f7183fff640 1 -- 172.21.15.87:0/119056870 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194043020 con 0x55dd455c0ea0 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.701+0000 7f7183fff640 1 -- 172.21.15.87:0/119056870 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719402c7e0 con 0x55dd455c0ea0 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.701+0000 7f7183fff640 1 -- 172.21.15.87:0/119056870 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194005570 con 0x55dd455c0ea0 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- 172.21.15.87:0/119056870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 msgr2=0x55dd455c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f7194036040 tx=0x7f719402c650 comp rx=0 tx=0).stop 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- 172.21.15.87:0/119056870 shutdown_connections 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c9a80 0x55dd455cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455c9540 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 --2- 172.21.15.87:0/119056870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- 172.21.15.87:0/119056870 >> 172.21.15.87:0/119056870 conn(0x55dd455bacb0 msgr2=0x55dd455bb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- 172.21.15.87:0/119056870 shutdown_connections 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- 172.21.15.87:0/119056870 wait complete. 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 -- start start 2024-04-22T13:43:45.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.702+0000 7f71a2a39740 1 Processor -- start 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55dd455cc870 con 0x55dd455c9a80 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dd454de2f0 con 0x55dd455c0ea0 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dd455cc9f0 con 0x55dd455c0580 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 10 client.0 _ll_register_callbacks cb 0x55dd454eb580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55746/0 (socket says 172.21.15.87:55746) 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 -- 172.21.15.87:0/1847430498 learned_addr learned my addr 172.21.15.87:0/1847430498 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719db03640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719d302640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 -- 172.21.15.87:0/1847430498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 msgr2=0x55dd455c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 -- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 msgr2=0x55dd455c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 -- 172.21.15.87:0/1847430498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd452373f0 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719db03640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f719e304640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f71900109e0 tx=0x7f719001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37525c6da31bbdd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71817fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719001cd30 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f719402c7e0 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f7190021460 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f719002f050 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 10 client.19173 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.703+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dd451f70f0 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719402c7e0 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f719001f680 con 0x55dd455c9a80 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 10 client.19173 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 10 client.19173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 20 client.19173 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd455d9a30 0x55dd455dbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 4 client.19173 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd45532c00 con 0x55dd455d9a30 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.704+0000 7f719db03640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd455d9a30 0x55dd455dbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.705+0000 7f71817fa640 10 client.19173 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.705+0000 7f719db03640 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd455d9a30 0x55dd455dbe80 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0b15428715ed9c server_cookie=b02b529e17cf7d91 in_seq=0 out_seq=0 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.705+0000 7f71817fa640 1 -- 172.21.15.87:0/1847430498 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd45532c00 con 0x55dd455d9a30 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.705+0000 7f71817fa640 10 client.19173 handle_command_reply: tid=0 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.705+0000 7f71a2a39740 1 client.19173 shutdown 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 msgr2=0x55dd455c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f71900109e0 tx=0x7f719001a000 comp rx=0 tx=0).stop 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 shutdown_connections 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd455d9a30 0x55dd455dbe80 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c9a80 0x55dd455c7420 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71817fa640 0 client.19173 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1847430498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c0580 0x55dd455c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 >> 172.21.15.87:0/1847430498 conn(0x55dd455bacb0 msgr2=0x55dd455bca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 shutdown_connections 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 1 -- 172.21.15.87:0/1847430498 wait complete. 2024-04-22T13:43:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.706+0000 7f71a2a39740 20 client.19173 trim_cache size 0 max 16384 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 Processor -- start 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 -- start start 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0580 0x55dd455c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b67f0 con 0x55dd455c0ea0 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b40c0 con 0x55dd455c0580 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.723+0000 7f71a2a39740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b1f30 con 0x55dd455c9a80 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55750/0 (socket says 172.21.15.87:55750) 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 -- 172.21.15.87:0/350061417 learned_addr learned my addr 172.21.15.87:0/350061417 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719db03640 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0580 0x55dd455c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 -- 172.21.15.87:0/350061417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 msgr2=0x55dd455bc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 -- 172.21.15.87:0/350061417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0580 msgr2=0x55dd455c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0580 0x55dd455c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 -- 172.21.15.87:0/350061417 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd455ba7e0 con 0x55dd455c0ea0 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f719d302640 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f71880cdb10 tx=0x7f7188091790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70f44155a43c9be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f71837fe640 1 -- 172.21.15.87:0/350061417 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718809bc60 con 0x55dd455c0ea0 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f71837fe640 1 -- 172.21.15.87:0/350061417 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718809ca00 con 0x55dd455c0ea0 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.724+0000 7f71837fe640 1 -- 172.21.15.87:0/350061417 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880cdd30 con 0x55dd455c0ea0 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- 172.21.15.87:0/350061417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 msgr2=0x55dd455d5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f71880cdb10 tx=0x7f7188091790 comp rx=0 tx=0).stop 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- 172.21.15.87:0/350061417 shutdown_connections 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455bc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0ea0 0x55dd455d5760 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 --2- 172.21.15.87:0/350061417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0580 0x55dd455c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- 172.21.15.87:0/350061417 >> 172.21.15.87:0/350061417 conn(0x55dd455bacb0 msgr2=0x55dd455bc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- 172.21.15.87:0/350061417 shutdown_connections 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- 172.21.15.87:0/350061417 wait complete. 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 -- start start 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.725+0000 7f71a2a39740 1 Processor -- start 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b2580 con 0x55dd455c0580 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b39f0 con 0x55dd455c0ea0 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f71981b2c50 con 0x55dd455c9a80 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 10 client.0 _ll_register_callbacks cb 0x7f71900106e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55762/0 (socket says 172.21.15.87:55762) 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 -- 172.21.15.87:0/1889832535 learned_addr learned my addr 172.21.15.87:0/1889832535 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719d302640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 -- 172.21.15.87:0/1889832535 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 msgr2=0x55dd455ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719e304640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 -- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 msgr2=0x55dd455cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 -- 172.21.15.87:0/1889832535 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd4563efc0 con 0x55dd455c0580 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719d302640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f719db03640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f7194036040 tx=0x7f7194005150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ebb88049916a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71817fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194043040 con 0x55dd455c0580 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f719404fd60 con 0x55dd455c0580 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f718809ca00 con 0x55dd455c0580 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f719405e050 con 0x55dd455c0580 2024-04-22T13:43:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 10 client.19179 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.726+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dd45640310 con 0x55dd455c0580 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718809ca00 con 0x55dd455c0580 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f719404f550 con 0x55dd455c0580 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 10 client.19179 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 10 client.19179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 20 client.19179 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd45640670 0x55dd45642a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 4 client.19179 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd4552ca70 con 0x55dd45640670 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f719d302640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd45640670 0x55dd45642a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f71817fa640 10 client.19179 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.727+0000 7f719d302640 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd45640670 0x55dd45642a90 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b149bf7ffc9445 server_cookie=a4ef354c156e0257 in_seq=0 out_seq=0 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71817fa640 1 -- 172.21.15.87:0/1889832535 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dd4552ca70 con 0x55dd45640670 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71817fa640 10 client.19179 handle_command_reply: tid=0 2024-04-22T13:43:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 client.19179 shutdown 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 msgr2=0x55dd455cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f7194036040 tx=0x7f7194005150 comp rx=0 tx=0).stop 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 shutdown_connections 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55dd45640670 0x55dd45642a90 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd455c9a80 0x55dd455ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd455c0ea0 0x55dd455cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 --2- 172.21.15.87:0/1889832535 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55dd455c0580 0x55dd455cd690 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 >> 172.21.15.87:0/1889832535 conn(0x55dd455bacb0 msgr2=0x55dd455c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71817fa640 0 client.19179 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 shutdown_connections 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 1 -- 172.21.15.87:0/1889832535 wait complete. 2024-04-22T13:43:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f71a2a39740 20 client.19179 trim_cache size 0 max 16384 2024-04-22T13:43:45.731 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:45.731 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:43:45.731 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:43:45.731 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 msgr2=0x7f71840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 0x7f71840a61e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7190005c40 tx=0x7f71900129b0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 msgr2=0x7f71981a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f718800d950 tx=0x7f718800de10 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 shutdown_connections 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f71840a3d20 0x7f71840a61e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7198106ed0 0x7f719819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7198104960 0x7f71981a4d30 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 --2- 172.21.15.87:0/1361185633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f719806ab30 0x7f71981a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.728+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 >> 172.21.15.87:0/1361185633 conn(0x7f7198077c40 msgr2=0x7f7198103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.729+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 shutdown_connections 2024-04-22T13:43:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.729+0000 7f719fd67640 1 -- 172.21.15.87:0/1361185633 wait complete. 2024-04-22T13:43:45.735 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:43:45.736 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:43:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 -- 172.21.15.87:0/4289681242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc06a8f0 msgr2=0x7fb5fc06acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 --2- 172.21.15.87:0/4289681242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc06acd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009900 tx=0x7fb5ec02bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 -- 172.21.15.87:0/4289681242 shutdown_connections 2024-04-22T13:43:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 --2- 172.21.15.87:0/4289681242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 --2- 172.21.15.87:0/4289681242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc06b2a0 0x7fb5fc106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 --2- 172.21.15.87:0/4289681242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc06acd0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 -- 172.21.15.87:0/4289681242 >> 172.21.15.87:0/4289681242 conn(0x7fb5fc077e30 msgr2=0x7fb5fc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 -- 172.21.15.87:0/4289681242 shutdown_connections 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 -- 172.21.15.87:0/4289681242 wait complete. 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.838+0000 7fb6013ef640 1 Processor -- start 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 -- start start 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc19cfd0 0x7fb5fc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc112210 con 0x7fb5fc1072f0 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc112090 con 0x7fb5fc19cfd0 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb6013ef640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc112390 con 0x7fb5fc06a8f0 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55772/0 (socket says 172.21.15.87:55772) 2024-04-22T13:43:45.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 -- 172.21.15.87:0/2046506845 learned_addr learned my addr 172.21.15.87:0/2046506845 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fb7fe640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc19cfd0 0x7fb5fc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5faffd640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 -- 172.21.15.87:0/2046506845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc06a8f0 msgr2=0x7fb5fc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 -- 172.21.15.87:0/2046506845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc19cfd0 msgr2=0x7fb5fc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc19cfd0 0x7fb5fc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 -- 172.21.15.87:0/2046506845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5fc19dac0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.839+0000 7fb5fa7fc640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c8c0 tx=0x7fb5f000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.840+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0007bb0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.840+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5fc19ddb0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.840+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f0007d50 con 0x7fb5fc1072f0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.840+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00187c0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.840+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5fc1a55f0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.841+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5fc069230 con 0x7fb5fc1072f0 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.842+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb5f000fa20 con 0x7fb5fc1072f0 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.842+0000 7fb5e3fff640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 0x7fb5e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.842+0000 7fb5faffd640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 0x7fb5e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.843+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fb5f00d0ab0 con 0x7fb5fc1072f0 2024-04-22T13:43:45.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.843+0000 7fb5faffd640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 0x7fb5e40a61e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec006010 tx=0x7fb5ec005e40 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:45.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:45.846+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5f0014030 con 0x7fb5fc1072f0 2024-04-22T13:43:45.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:45 smithi120 ceph-mon[31101]: pgmap v372: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:45.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:45 smithi132 ceph-mon[31531]: pgmap v372: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:45.986 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:45 smithi087 ceph-mon[27999]: pgmap v372: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T13:43:46.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.061+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5fc199a60 con 0x7fb5fc1072f0 2024-04-22T13:43:46.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.061+0000 7fb5e3fff640 1 -- 172.21.15.87:0/2046506845 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fb5f009d220 con 0x7fb5fc1072f0 2024-04-22T13:43:46.058 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:43:46.058 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:43:46.058 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:43:46.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 msgr2=0x7fb5e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 0x7fb5e40a61e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec006010 tx=0x7fb5ec005e40 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 msgr2=0x7fb5fc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c8c0 tx=0x7fb5f000cd80 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 shutdown_connections 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb5e40a3d20 0x7fb5e40a61e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5fc19cfd0 0x7fb5fc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5fc1072f0 0x7fb5fc198c50 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 --2- 172.21.15.87:0/2046506845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5fc06a8f0 0x7fb5fc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.063+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 >> 172.21.15.87:0/2046506845 conn(0x7fb5fc077e30 msgr2=0x7fb5fc0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.064+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 shutdown_connections 2024-04-22T13:43:46.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.064+0000 7fb6013ef640 1 -- 172.21.15.87:0/2046506845 wait complete. 2024-04-22T13:43:46.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 -- 172.21.15.87:0/2155680062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d301021a0 msgr2=0x7f1d30102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 --2- 172.21.15.87:0/2155680062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d301021a0 0x7f1d30102580 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009900 tx=0x7f1d1802bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 -- 172.21.15.87:0/2155680062 shutdown_connections 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 --2- 172.21.15.87:0/2155680062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d30107270 0x7f1d3010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 --2- 172.21.15.87:0/2155680062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d30102b50 0x7f1d30106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 --2- 172.21.15.87:0/2155680062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d301021a0 0x7f1d30102580 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 -- 172.21.15.87:0/2155680062 >> 172.21.15.87:0/2155680062 conn(0x7f1d30079350 msgr2=0x7f1d30079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 -- 172.21.15.87:0/2155680062 shutdown_connections 2024-04-22T13:43:46.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.175+0000 7f1d34a81640 1 -- 172.21.15.87:0/2155680062 wait complete. 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 Processor -- start 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 -- start start 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d301021a0 0x7f1d3019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 0x7f1d30198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301121b0 con 0x7f1d30107270 2024-04-22T13:43:46.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30112030 con 0x7f1d301021a0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d34a81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30112330 con 0x7f1d301985b0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55794/0 (socket says 172.21.15.87:55794) 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/517255769 learned_addr learned my addr 172.21.15.87:0/517255769 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d301021a0 0x7f1d3019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 0x7f1d30198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/517255769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 msgr2=0x7f1d30198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 0x7f1d30198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/517255769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d301021a0 msgr2=0x7f1d3019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d301021a0 0x7f1d3019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/517255769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d301990d0 con 0x7f1d30107270 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 0x7f1d30198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.176+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d770 tx=0x7f1d2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.177+0000 7f1d277fe640 1 -- 172.21.15.87:0/517255769 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20017070 con 0x7f1d30107270 2024-04-22T13:43:46.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.177+0000 7f1d277fe640 1 -- 172.21.15.87:0/517255769 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d20009c30 con 0x7f1d30107270 2024-04-22T13:43:46.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.177+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d301993c0 con 0x7f1d30107270 2024-04-22T13:43:46.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.177+0000 7f1d277fe640 1 -- 172.21.15.87:0/517255769 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2000f620 con 0x7f1d30107270 2024-04-22T13:43:46.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.177+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301a5660 con 0x7f1d30107270 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.179+0000 7f1d277fe640 1 -- 172.21.15.87:0/517255769 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1d2001a070 con 0x7f1d30107270 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.179+0000 7f1d37677740 1 Processor -- start 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.179+0000 7f1d37677740 1 -- start start 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.179+0000 7f1d277fe640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 0x7f1d1c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a50838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 0x7f1d1c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a508ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5561a4fe81f0 con 0x5561a5083e80 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5561a4f854b0 con 0x5561a50834d0 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d37677740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561a4fe41b0 con 0x5561a508ca00 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d277fe640 1 -- 172.21.15.87:0/517255769 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f1d200d0360 con 0x7f1d30107270 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55796/0 (socket says 172.21.15.87:55796) 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/239806056 learned_addr learned my addr 172.21.15.87:0/239806056 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a508ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a50838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 0x7f1d1c0a6260 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005fd0 tx=0x7f1d2800e6c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/239806056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 msgr2=0x5561a508ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a508ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/239806056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 msgr2=0x5561a50838b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a50838b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/239806056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a4f771b0 con 0x5561a5083e80 2024-04-22T13:43:46.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a508ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1d200ce0f0 tx=0x7f1d20009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d987244d88a79685 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d24ff9640 1 -- 172.21.15.87:0/239806056 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20021040 con 0x5561a5083e80 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.180+0000 7f1d24ff9640 1 -- 172.21.15.87:0/239806056 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d200a06f0 con 0x5561a5083e80 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d24ff9640 1 -- 172.21.15.87:0/239806056 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d200982f0 con 0x5561a5083e80 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 -- 172.21.15.87:0/239806056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 msgr2=0x5561a508c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1d200ce0f0 tx=0x7f1d20009ea0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 -- 172.21.15.87:0/239806056 shutdown_connections 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a508ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a508c430 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 --2- 172.21.15.87:0/239806056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a50838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 -- 172.21.15.87:0/239806056 >> 172.21.15.87:0/239806056 conn(0x5561a507dba0 msgr2=0x5561a507df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 -- 172.21.15.87:0/239806056 shutdown_connections 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.181+0000 7f1d37677740 1 -- 172.21.15.87:0/239806056 wait complete. 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 -- start start 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 Processor -- start 2024-04-22T13:43:46.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a5089d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5561a508f910 con 0x5561a5083e80 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5561a4fa1500 con 0x5561a50834d0 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561a508fa90 con 0x5561a508ca00 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d37677740 10 client.0 _ll_register_callbacks cb 0x5561a4f6fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55798/0 (socket says 172.21.15.87:55798) 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/923790712 learned_addr learned my addr 172.21.15.87:0/923790712 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a5089d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 msgr2=0x5561a5089d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a5089d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/923790712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 msgr2=0x5561a508c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 -- 172.21.15.87:0/923790712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a4cfa3f0 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a5089d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d740 tx=0x7f1d200ce020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9612d78585be0292 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.182+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d200ba070 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f1d200bb270 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d200a06f0 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1d20085050 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 10 client.19197 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561a4dc47d0 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d200a06f0 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1d200a8080 con 0x5561a5083e80 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 10 client.19197 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 10 client.19197 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 20 client.19197 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a509ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 4 client.19197 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.183+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561a4f87930 con 0x5561a509ca80 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.184+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a509ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.184+0000 7f1d15ffb640 10 client.19197 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.184+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a509ee70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4d446e725a81e92 server_cookie=988e658ef723f05d in_seq=0 out_seq=0 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d15ffb640 1 -- 172.21.15.87:0/923790712 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561a4f87930 con 0x5561a509ca80 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d15ffb640 10 client.19197 handle_command_reply: tid=0 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 client.19197 shutdown 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 msgr2=0x5561a5089700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d740 tx=0x7f1d200ce020 comp rx=0 tx=0).stop 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 shutdown_connections 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a509ee70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a508ca00 0x5561a5089d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d15ffb640 0 client.19197 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a5083e80 0x5561a5089700 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 --2- 172.21.15.87:0/923790712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 >> 172.21.15.87:0/923790712 conn(0x5561a507dba0 msgr2=0x5561a507f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 shutdown_connections 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 1 -- 172.21.15.87:0/923790712 wait complete. 2024-04-22T13:43:46.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.185+0000 7f1d37677740 20 client.19197 trim_cache size 0 max 16384 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 Processor -- start 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 -- start start 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a50950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 0x5561a50885d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b0450 con 0x5561a50834d0 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301add20 con 0x5561a508ca00 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.198+0000 7f1d37677740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301abb90 con 0x5561a5083e80 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55806/0 (socket says 172.21.15.87:55806) 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 -- 172.21.15.87:0/3873013683 learned_addr learned my addr 172.21.15.87:0/3873013683 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a50950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 -- 172.21.15.87:0/3873013683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 msgr2=0x5561a50950c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 0x5561a50885d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a50950c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 -- 172.21.15.87:0/3873013683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 msgr2=0x5561a50885d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 0x5561a50885d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 -- 172.21.15.87:0/3873013683 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a508a080 con 0x5561a50834d0 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a50950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 0x5561a50885d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005940 tx=0x7f1d180058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325e359bc4ab0a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d17fff640 1 -- 172.21.15.87:0/3873013683 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1804d020 con 0x5561a50834d0 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d17fff640 1 -- 172.21.15.87:0/3873013683 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d1802cc90 con 0x5561a50834d0 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.199+0000 7f1d17fff640 1 -- 172.21.15.87:0/3873013683 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18047070 con 0x5561a50834d0 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- 172.21.15.87:0/3873013683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 msgr2=0x5561a50987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005940 tx=0x7f1d180058b0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- 172.21.15.87:0/3873013683 shutdown_connections 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a508ca00 0x5561a50885d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a50950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- 172.21.15.87:0/3873013683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a50834d0 0x5561a50987d0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- 172.21.15.87:0/3873013683 >> 172.21.15.87:0/3873013683 conn(0x5561a507dba0 msgr2=0x5561a5087ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- 172.21.15.87:0/3873013683 shutdown_connections 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- 172.21.15.87:0/3873013683 wait complete. 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- start start 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 Processor -- start 2024-04-22T13:43:46.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a508bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ac1e0 con 0x5561a508ca00 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ad650 con 0x5561a50834d0 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ac8b0 con 0x5561a5083e80 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d37677740 10 client.0 _ll_register_callbacks cb 0x7f1d20010f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55814/0 (socket says 172.21.15.87:55814) 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.200+0000 7f1d2ed9d640 1 -- 172.21.15.87:0/1164304051 learned_addr learned my addr 172.21.15.87:0/1164304051 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a508bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 -- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 msgr2=0x5561a508bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a508bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 -- 172.21.15.87:0/1164304051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 msgr2=0x5561a508b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 -- 172.21.15.87:0/1164304051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a5105d10 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2dd9b640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a508bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d2ed9d640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1d28012c10 tx=0x7f1d28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc15f80adc50ee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28012c40 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f1d28012de0 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1802cc90 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1d28003970 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d37677740 10 client.19203 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561a5106fa0 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.201+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d1802cc90 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1d2800f040 con 0x5561a508ca00 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 10 client.19203 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 10 client.19203 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:43:46.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 20 client.19203 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a5109790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 4 client.19203 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561a4ff5c10 con 0x5561a509ca80 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a5109790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d15ffb640 10 client.19203 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d2e59c640 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a5109790 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d72adcf9217ab82c server_cookie=3bf1bcf30791b807 in_seq=0 out_seq=0 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d15ffb640 1 -- 172.21.15.87:0/1164304051 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561a4ff5c10 con 0x5561a509ca80 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d15ffb640 10 client.19203 handle_command_reply: tid=0 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.202+0000 7f1d37677740 1 client.19203 shutdown 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 msgr2=0x5561a508c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1d28012c10 tx=0x7f1d28004ab0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 shutdown_connections 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x5561a509ca80 0x5561a5109790 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5561a508ca00 0x5561a508c2b0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d15ffb640 0 client.19203 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561a5083e80 0x5561a508bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 --2- 172.21.15.87:0/1164304051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5561a50834d0 0x5561a508b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 >> 172.21.15.87:0/1164304051 conn(0x5561a507dba0 msgr2=0x5561a5088750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 shutdown_connections 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 1 -- 172.21.15.87:0/1164304051 wait complete. 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d37677740 20 client.19203 trim_cache size 0 max 16384 2024-04-22T13:43:46.206 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 msgr2=0x7f1d1c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 0x7f1d1c0a6260 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005fd0 tx=0x7f1d2800e6c0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 msgr2=0x7f1d3019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000d770 tx=0x7f1d2000dc30 comp rx=0 tx=0).stop 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 shutdown_connections 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1d1c0a3da0 0x7f1d1c0a6260 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d301985b0 0x7f1d30198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1d30107270 0x7f1d3019e940 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 --2- 172.21.15.87:0/517255769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d301021a0 0x7f1d3019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 >> 172.21.15.87:0/517255769 conn(0x7f1d30079350 msgr2=0x7f1d30100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 shutdown_connections 2024-04-22T13:43:46.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.203+0000 7f1d34a81640 1 -- 172.21.15.87:0/517255769 wait complete. 2024-04-22T13:43:46.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:43:46.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.314+0000 7f45dc152640 1 -- 172.21.15.87:0/1311398716 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 msgr2=0x7f45d4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.314+0000 7f45dc152640 1 --2- 172.21.15.87:0/1311398716 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d4101130 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009ca0 tx=0x7f45cc02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- 172.21.15.87:0/1311398716 shutdown_connections 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- 172.21.15.87:0/1311398716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45d410e900 0x7f45d4110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- 172.21.15.87:0/1311398716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d4101670 0x7f45d410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- 172.21.15.87:0/1311398716 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d4101130 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- 172.21.15.87:0/1311398716 >> 172.21.15.87:0/1311398716 conn(0x7f45d4077f50 msgr2=0x7f45d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- 172.21.15.87:0/1311398716 shutdown_connections 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- 172.21.15.87:0/1311398716 wait complete. 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 Processor -- start 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- start start 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45d4101670 0x7f45d41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d410e900 0x7f45d419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4114430 con 0x7f45d4100d50 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41142b0 con 0x7f45d410e900 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45dc152640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41145b0 con 0x7f45d4101670 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.315+0000 7f45d9eee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55834/0 (socket says 172.21.15.87:55834) 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 -- 172.21.15.87:0/86104691 learned_addr learned my addr 172.21.15.87:0/86104691 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 -- 172.21.15.87:0/86104691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45d4101670 msgr2=0x7f45d41a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45d4101670 0x7f45d41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 -- 172.21.15.87:0/86104691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d410e900 msgr2=0x7f45d419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d410e900 0x7f45d419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 -- 172.21.15.87:0/86104691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d419afe0 con 0x7f45d4100d50 2024-04-22T13:43:46.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45d9eee640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f45cc005e30 tx=0x7f45cc034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45caffd640 1 -- 172.21.15.87:0/86104691 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc039040 con 0x7f45d4100d50 2024-04-22T13:43:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45caffd640 1 -- 172.21.15.87:0/86104691 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc044070 con 0x7f45d4100d50 2024-04-22T13:43:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d419b270 con 0x7f45d4100d50 2024-04-22T13:43:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45caffd640 1 -- 172.21.15.87:0/86104691 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0042a0 con 0x7f45d4100d50 2024-04-22T13:43:46.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.316+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a74f0 con 0x7f45d4100d50 2024-04-22T13:43:46.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.318+0000 7f45caffd640 1 -- 172.21.15.87:0/86104691 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f45cc004440 con 0x7f45d4100d50 2024-04-22T13:43:46.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 Processor -- start 2024-04-22T13:43:46.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 -- start start 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45caffd640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 0x7f45c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb34b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb34ba00 0x55c1eb34de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb2a71f0 con 0x55c1eb3424d0 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb2444b0 con 0x55c1eb342e80 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45dee18740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb2a31b0 con 0x55c1eb34ba00 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45da6ef640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 0x7f45c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45caffd640 1 -- 172.21.15.87:0/86104691 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f45cc0f2ac0 con 0x7f45d4100d50 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55848/0 (socket says 172.21.15.87:55848) 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 -- 172.21.15.87:0/1719583877 learned_addr learned my addr 172.21.15.87:0/1719583877 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d96ed640 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb34b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45da6ef640 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb34ba00 0x55c1eb34de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 -- 172.21.15.87:0/1719583877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb34ba00 msgr2=0x55c1eb34de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb34ba00 0x55c1eb34de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 -- 172.21.15.87:0/1719583877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 msgr2=0x55c1eb34b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb34b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.319+0000 7f45d9eee640 1 -- 172.21.15.87:0/1719583877 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1eb2361b0 con 0x55c1eb3424d0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45da6ef640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 0x7f45c00a61e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f45d0005fd0 tx=0x7f45d00147e0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45d9eee640 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0f03c0 tx=0x7f45cc003bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e6a95a7f579f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45bffff640 1 -- 172.21.15.87:0/1719583877 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc03e3a0 con 0x55c1eb3424d0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45bffff640 1 -- 172.21.15.87:0/1719583877 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc003e30 con 0x55c1eb3424d0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45bffff640 1 -- 172.21.15.87:0/1719583877 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc04d9c0 con 0x55c1eb3424d0 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 -- 172.21.15.87:0/1719583877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 msgr2=0x55c1eb3428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0f03c0 tx=0x7f45cc003bc0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 -- 172.21.15.87:0/1719583877 shutdown_connections 2024-04-22T13:43:46.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb34ba00 0x55c1eb34de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb34b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 --2- 172.21.15.87:0/1719583877 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb3428b0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.320+0000 7f45dee18740 1 -- 172.21.15.87:0/1719583877 >> 172.21.15.87:0/1719583877 conn(0x55c1eb33cba0 msgr2=0x55c1eb33cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- 172.21.15.87:0/1719583877 shutdown_connections 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- 172.21.15.87:0/1719583877 wait complete. 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- start start 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 Processor -- start 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb3424d0 0x55c1eb34b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb348610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb34e730 con 0x55c1eb34ba00 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb260500 con 0x55c1eb342e80 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c1eb34e8b0 con 0x55c1eb3424d0 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45dee18740 10 client.0 _ll_register_callbacks cb 0x55c1eb22eb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45da6ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45da6ef640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55858/0 (socket says 172.21.15.87:55858) 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.321+0000 7f45da6ef640 1 -- 172.21.15.87:0/2384374393 learned_addr learned my addr 172.21.15.87:0/2384374393 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45d96ed640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb348610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45d9eee640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb3424d0 0x55c1eb34b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 -- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb3424d0 msgr2=0x55c1eb34b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb3424d0 0x55c1eb34b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 -- 172.21.15.87:0/2384374393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 msgr2=0x55c1eb348610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb348610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 -- 172.21.15.87:0/2384374393 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1eafb93f0 con 0x55c1eb34ba00 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45da6ef640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f45d0012b50 tx=0x7f45d0016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8f9956fcfa99a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0023070 con 0x55c1eb34ba00 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f45d0004220 con 0x55c1eb34ba00 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc003e30 con 0x55c1eb34ba00 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f45d002e050 con 0x55c1eb34ba00 2024-04-22T13:43:46.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45dee18740 10 client.19215 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c1eb0837d0 con 0x55c1eb34ba00 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc003e30 con 0x55c1eb34ba00 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.322+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f45d001e070 con 0x55c1eb34ba00 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 10 client.19215 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 10 client.19215 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 20 client.19215 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb35b850 0x55c1eb35dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 4 client.19215 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1eb246930 con 0x55c1eb35b850 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45d9eee640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb35b850 0x55c1eb35dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45d9eee640 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb35b850 0x55c1eb35dcb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25125d981d347f43 server_cookie=c7587045e107618e in_seq=0 out_seq=0 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.323+0000 7f45bd7fa640 10 client.19215 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:46.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45bd7fa640 1 -- 172.21.15.87:0/2384374393 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1eb246930 con 0x55c1eb35b850 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45bd7fa640 10 client.19215 handle_command_reply: tid=0 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 client.19215 shutdown 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 msgr2=0x55c1eb348b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f45d0012b50 tx=0x7f45d0016040 comp rx=0 tx=0).stop 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 shutdown_connections 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb35b850 0x55c1eb35dcb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb34ba00 0x55c1eb348b50 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb348610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45bd7fa640 0 client.19215 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 --2- 172.21.15.87:0/2384374393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb3424d0 0x55c1eb34b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 >> 172.21.15.87:0/2384374393 conn(0x55c1eb33cba0 msgr2=0x55c1eb33e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 shutdown_connections 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 1 -- 172.21.15.87:0/2384374393 wait complete. 2024-04-22T13:43:46.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.324+0000 7f45dee18740 20 client.19215 trim_cache size 0 max 16384 2024-04-22T13:43:46.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.337+0000 7f45dee18740 1 Processor -- start 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 -- start start 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 0x55c1eb3574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb342e80 0x55c1eb363ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41b2680 con 0x55c1eb3424d0 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41aff50 con 0x55c1eb33cba0 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45dee18740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41addc0 con 0x55c1eb342e80 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55862/0 (socket says 172.21.15.87:55862) 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 -- 172.21.15.87:0/2410060653 learned_addr learned my addr 172.21.15.87:0/2410060653 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45da6ef640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb342e80 0x55c1eb363ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d9eee640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 0x55c1eb3574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 -- 172.21.15.87:0/2410060653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb342e80 msgr2=0x55c1eb363ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb342e80 0x55c1eb363ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 -- 172.21.15.87:0/2410060653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 msgr2=0x55c1eb3574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 0x55c1eb3574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 -- 172.21.15.87:0/2410060653 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1eb34e210 con 0x55c1eb3424d0 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d9eee640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 0x55c1eb3574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45d96ed640 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f45d419bb60 tx=0x7f45c4009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb89f1666c93e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45bf7fe640 1 -- 172.21.15.87:0/2410060653 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4019070 con 0x55c1eb3424d0 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45bf7fe640 1 -- 172.21.15.87:0/2410060653 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4011e70 con 0x55c1eb3424d0 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.338+0000 7f45bf7fe640 1 -- 172.21.15.87:0/2410060653 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c40143e0 con 0x55c1eb3424d0 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- 172.21.15.87:0/2410060653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 msgr2=0x55c1eb363580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f45d419bb60 tx=0x7f45c4009110 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- 172.21.15.87:0/2410060653 shutdown_connections 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb342e80 0x55c1eb363ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb363580 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 --2- 172.21.15.87:0/2410060653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb33cba0 0x55c1eb3574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- 172.21.15.87:0/2410060653 >> 172.21.15.87:0/2410060653 conn(0x55c1eb33bfa0 msgr2=0x55c1eb34a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- 172.21.15.87:0/2410060653 shutdown_connections 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- 172.21.15.87:0/2410060653 wait complete. 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 -- start start 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.339+0000 7f45dee18740 1 Processor -- start 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb33cba0 0x55c1eb353d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb351660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ae410 con 0x55c1eb3424d0 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41af880 con 0x55c1eb342e80 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41aeae0 con 0x55c1eb33cba0 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45dee18740 10 client.0 _ll_register_callbacks cb 0x7f45d0012700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55878/0 (socket says 172.21.15.87:55878) 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 -- 172.21.15.87:0/815892579 learned_addr learned my addr 172.21.15.87:0/815892579 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d9eee640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb33cba0 0x55c1eb353d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 -- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb33cba0 msgr2=0x55c1eb353d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb33cba0 0x55c1eb353d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45da6ef640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb351660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 -- 172.21.15.87:0/815892579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 msgr2=0x55c1eb351660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb351660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 -- 172.21.15.87:0/815892579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1eb351e70 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45d96ed640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f45c40095b0 tx=0x7f45c4015a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e29806094b1300b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.340+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4019040 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f45c4011bd0 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c4011e70 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f45c401e450 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 10 client.19221 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c1eb3c1d40 con 0x55c1eb3424d0 2024-04-22T13:43:46.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4011e70 con 0x55c1eb3424d0 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f45c40145e0 con 0x55c1eb3424d0 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 10 client.19221 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 10 client.19221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 20 client.19221 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb34ba00 0x55c1eb3c4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 4 client.19221 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1eb2b4c10 con 0x55c1eb34ba00 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.341+0000 7f45d9eee640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb34ba00 0x55c1eb3c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45bd7fa640 10 client.19221 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45d9eee640 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb34ba00 0x55c1eb3c4390 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7d61b3013d02539 server_cookie=714b1efdb0946a3b in_seq=0 out_seq=0 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45bd7fa640 1 -- 172.21.15.87:0/815892579 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1eb2b4c10 con 0x55c1eb34ba00 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45bd7fa640 10 client.19221 handle_command_reply: tid=0 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 client.19221 shutdown 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 msgr2=0x55c1eb351120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f45c40095b0 tx=0x7f45c4015a40 comp rx=0 tx=0).stop 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 shutdown_connections 2024-04-22T13:43:46.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55c1eb34ba00 0x55c1eb3c4390 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1eb342e80 0x55c1eb351660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c1eb3424d0 0x55c1eb351120 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 --2- 172.21.15.87:0/815892579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c1eb33cba0 0x55c1eb353d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45bd7fa640 0 client.19221 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 >> 172.21.15.87:0/815892579 conn(0x55c1eb33bfa0 msgr2=0x55c1eb343260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 shutdown_connections 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 1 -- 172.21.15.87:0/815892579 wait complete. 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.342+0000 7f45dee18740 20 client.19221 trim_cache size 0 max 16384 2024-04-22T13:43:46.344 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 msgr2=0x7f45c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 0x7f45c00a61e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f45d0005fd0 tx=0x7f45d00147e0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 msgr2=0x7f45d41a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f45cc005e30 tx=0x7f45cc034760 comp rx=0 tx=0).stop 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 shutdown_connections 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f45c00a3d20 0x7f45c00a61e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d410e900 0x7f45d419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45d4101670 0x7f45d41a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 --2- 172.21.15.87:0/86104691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45d4100d50 0x7f45d41a06a0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 >> 172.21.15.87:0/86104691 conn(0x7f45d4077f50 msgr2=0x7f45d4110000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 shutdown_connections 2024-04-22T13:43:46.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.343+0000 7f45dc152640 1 -- 172.21.15.87:0/86104691 wait complete. 2024-04-22T13:43:46.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:43:46.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 -- 172.21.15.87:0/3610893194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 msgr2=0x7fe2f8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 --2- 172.21.15.87:0/3610893194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f8100ed0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009900 tx=0x7fe2f402bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:46.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 -- 172.21.15.87:0/3610893194 shutdown_connections 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 --2- 172.21.15.87:0/3610893194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 0x7fe2f810eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 --2- 172.21.15.87:0/3610893194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f81014a0 0x7fe2f8106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 --2- 172.21.15.87:0/3610893194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f8100ed0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 -- 172.21.15.87:0/3610893194 >> 172.21.15.87:0/3610893194 conn(0x7fe2f80760e0 msgr2=0x7fe2f80764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 -- 172.21.15.87:0/3610893194 shutdown_connections 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 -- 172.21.15.87:0/3610893194 wait complete. 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.454+0000 7fe2fff75640 1 Processor -- start 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 -- start start 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 0x7fe2f819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 0x7fe2f8198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f8112230 con 0x7fe2f8100af0 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81120b0 con 0x7fe2f81072b0 2024-04-22T13:43:46.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fff75640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81123b0 con 0x7fe2f8198650 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55894/0 (socket says 172.21.15.87:55894) 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 -- 172.21.15.87:0/1568931568 learned_addr learned my addr 172.21.15.87:0/1568931568 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 0x7fe2f8198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fd510640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 0x7fe2f819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 -- 172.21.15.87:0/1568931568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 msgr2=0x7fe2f8198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 0x7fe2f8198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 -- 172.21.15.87:0/1568931568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 msgr2=0x7fe2f819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 0x7fe2f819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 -- 172.21.15.87:0/1568931568 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f8199250 con 0x7fe2f8100af0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 0x7fe2f8198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005940 tx=0x7fe2f402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.455+0000 7fe2eeffd640 1 -- 172.21.15.87:0/1568931568 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4043020 con 0x7fe2f8100af0 2024-04-22T13:43:46.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.456+0000 7fe2eeffd640 1 -- 172.21.15.87:0/1568931568 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2f402cbe0 con 0x7fe2f8100af0 2024-04-22T13:43:46.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.456+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f81994e0 con 0x7fe2f8100af0 2024-04-22T13:43:46.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.456+0000 7fe2eeffd640 1 -- 172.21.15.87:0/1568931568 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4003bf0 con 0x7fe2f8100af0 2024-04-22T13:43:46.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.456+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f81a5680 con 0x7fe2f8100af0 2024-04-22T13:43:46.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe2eeffd640 1 -- 172.21.15.87:0/1568931568 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe2f4046020 con 0x7fe2f8100af0 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 Processor -- start 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 -- start start 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe2eeffd640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 0x7fe2e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134250 0x55cea1134630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea1144220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cea10991f0 con 0x55cea1134c00 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cea10364b0 con 0x55cea1141e10 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.458+0000 7fe3027fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cea10951b0 con 0x55cea1134250 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 0x7fe2e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2eeffd640 1 -- 172.21.15.87:0/1568931568 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fe2f40f29c0 con 0x7fe2f8100af0 2024-04-22T13:43:46.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55896/0 (socket says 172.21.15.87:55896) 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 -- 172.21.15.87:0/146556610 learned_addr learned my addr 172.21.15.87:0/146556610 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134250 0x55cea1134630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 -- 172.21.15.87:0/146556610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134250 msgr2=0x55cea1134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134250 0x55cea1134630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 -- 172.21.15.87:0/146556610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 msgr2=0x55cea1144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea1144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 -- 172.21.15.87:0/146556610 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea10281b0 con 0x55cea1134c00 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 0x7fe2e40a61e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f00098a0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2fd510640 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800ba40 tx=0x7fe2e800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9474860a0a4840b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2e3fff640 1 -- 172.21.15.87:0/146556610 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8017420 con 0x55cea1134c00 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2e3fff640 1 -- 172.21.15.87:0/146556610 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8017a70 con 0x55cea1134c00 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.459+0000 7fe2e3fff640 1 -- 172.21.15.87:0/146556610 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800f8c0 con 0x55cea1134c00 2024-04-22T13:43:46.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- 172.21.15.87:0/146556610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 msgr2=0x55cea1141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800ba40 tx=0x7fe2e800bf00 comp rx=0 tx=0).stop 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- 172.21.15.87:0/146556610 shutdown_connections 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea1144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1141840 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 --2- 172.21.15.87:0/146556610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134250 0x55cea1134630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- 172.21.15.87:0/146556610 >> 172.21.15.87:0/146556610 conn(0x55cea112eba0 msgr2=0x55cea112ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- 172.21.15.87:0/146556610 shutdown_connections 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- 172.21.15.87:0/146556610 wait complete. 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 -- start start 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.460+0000 7fe3027fd740 1 Processor -- start 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea113e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea113ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55cea1144d40 con 0x55cea1134250 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cea1052500 con 0x55cea1141e10 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cea1144ec0 con 0x55cea1134c00 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe3027fd740 10 client.0 _ll_register_callbacks cb 0x55cea1020b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55898/0 (socket says 172.21.15.87:55898) 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 -- 172.21.15.87:0/16430806 learned_addr learned my addr 172.21.15.87:0/16430806 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fe512640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea113ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fd510640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea113e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 -- 172.21.15.87:0/16430806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 msgr2=0x55cea113e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea113e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 -- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 msgr2=0x55cea113ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea113ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 -- 172.21.15.87:0/16430806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea0dab3f0 con 0x55cea1134250 2024-04-22T13:43:46.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fe512640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea113ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40f0750 tx=0x7fe2f40bf1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70875bbfcb15812c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.461+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f40c7720 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fe2f40bcdb0 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2e8017a70 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe2f40054d0 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 10 client.19233 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cea0e757d0 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8017a70 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe2f40c7320 con 0x55cea1134250 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 10 client.19233 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 10 client.19233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:46.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 20 client.19233 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea1151ea0 0x55cea1154310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 4 client.19233 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cea1038930 con 0x55cea1151ea0 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.462+0000 7fe2fd510640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea1151ea0 0x55cea1154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe2e17fa640 10 client.19233 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe2fd510640 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea1151ea0 0x55cea1154310 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a99f07056b5387 server_cookie=9f66057f48504467 in_seq=0 out_seq=0 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe2e17fa640 1 -- 172.21.15.87:0/16430806 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cea1038930 con 0x55cea1151ea0 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe2e17fa640 10 client.19233 handle_command_reply: tid=0 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe3027fd740 1 client.19233 shutdown 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 msgr2=0x55cea1141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.463+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40f0750 tx=0x7fe2f40bf1b0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 shutdown_connections 2024-04-22T13:43:46.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea1151ea0 0x55cea1154310 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1141e10 0x55cea113ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea113e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe2e17fa640 0 client.19233 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 --2- 172.21.15.87:0/16430806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134250 0x55cea1141640 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 >> 172.21.15.87:0/16430806 conn(0x55cea112eba0 msgr2=0x55cea11308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 shutdown_connections 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 1 -- 172.21.15.87:0/16430806 wait complete. 2024-04-22T13:43:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.464+0000 7fe3027fd740 20 client.19233 trim_cache size 0 max 16384 2024-04-22T13:43:46.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 Processor -- start 2024-04-22T13:43:46.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 -- start start 2024-04-22T13:43:46.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1141e10 0x55cea1141100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81ae130 con 0x55cea1134c00 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81aba00 con 0x55cea1134250 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe3027fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81a9870 con 0x55cea1141e10 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55900/0 (socket says 172.21.15.87:55900) 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 -- 172.21.15.87:0/753304346 learned_addr learned my addr 172.21.15.87:0/753304346 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 -- 172.21.15.87:0/753304346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1141e10 msgr2=0x55cea1141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1141e10 0x55cea1141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 -- 172.21.15.87:0/753304346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 msgr2=0x55cea114dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.477+0000 7fe2fd510640 1 -- 172.21.15.87:0/753304346 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea113cf20 con 0x55cea1134c00 2024-04-22T13:43:46.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe2fd510640 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8017ce0 tx=0x7fe2e800f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85f6c500005ff9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe2e37fe640 1 -- 172.21.15.87:0/753304346 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8005a10 con 0x55cea1134c00 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe2e37fe640 1 -- 172.21.15.87:0/753304346 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8017e10 con 0x55cea1134c00 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe2e37fe640 1 -- 172.21.15.87:0/753304346 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8017420 con 0x55cea1134c00 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 -- 172.21.15.87:0/753304346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 msgr2=0x55cea1139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8017ce0 tx=0x7fe2e800f560 comp rx=0 tx=0).stop 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 -- 172.21.15.87:0/753304346 shutdown_connections 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1141e10 0x55cea1141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1134c00 0x55cea1139ef0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 --2- 172.21.15.87:0/753304346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 -- 172.21.15.87:0/753304346 >> 172.21.15.87:0/753304346 conn(0x55cea112eba0 msgr2=0x55cea11397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 -- 172.21.15.87:0/753304346 shutdown_connections 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.478+0000 7fe3027fd740 1 -- 172.21.15.87:0/753304346 wait complete. 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 -- start start 2024-04-22T13:43:46.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 Processor -- start 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea11459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81a9ec0 con 0x55cea1141e10 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81ab330 con 0x55cea1134250 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81aa590 con 0x55cea1134c00 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe3027fd740 10 client.0 _ll_register_callbacks cb 0x7fe2f40be880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55904/0 (socket says 172.21.15.87:55904) 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 -- 172.21.15.87:0/1911817821 learned_addr learned my addr 172.21.15.87:0/1911817821 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fd510640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea11459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 -- 172.21.15.87:0/1911817821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 msgr2=0x55cea11459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea11459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 -- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 msgr2=0x55cea114a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.479+0000 7fe2fe512640 1 -- 172.21.15.87:0/1911817821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea1146770 con 0x55cea1141e10 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2fd510640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea11459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2fe512640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0012740 tx=0x7fe2f0012b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a182f8f4b564587 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0023070 con 0x55cea1141e10 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fe2f0004d20 con 0x55cea1141e10 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2e8017e10 con 0x55cea1141e10 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe2f002e050 con 0x55cea1141e10 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 10 client.19239 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:46.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cea11b8580 con 0x55cea1141e10 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8017e10 con 0x55cea1141e10 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe2f001f930 con 0x55cea1141e10 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 10 client.19239 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 10 client.19239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 20 client.19239 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea11b8d30 0x55cea11bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 4 client.19239 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.480+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cea113f2a0 con 0x55cea11b8d30 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea11b8d30 0x55cea11bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2e17fa640 10 client.19239 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2fdd11640 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea11b8d30 0x55cea11bb150 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3173f429a1768685 server_cookie=3244ef47ce5fb9ad in_seq=0 out_seq=0 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2e17fa640 1 -- 172.21.15.87:0/1911817821 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cea113f2a0 con 0x55cea11b8d30 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2e17fa640 10 client.19239 handle_command_reply: tid=0 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 client.19239 shutdown 2024-04-22T13:43:46.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 msgr2=0x55cea1145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0012740 tx=0x7fe2f0012b40 comp rx=0 tx=0).stop 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 shutdown_connections 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55cea11b8d30 0x55cea11bb150 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55cea1141e10 0x55cea1145f40 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cea1134c00 0x55cea11459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe2e17fa640 0 client.19239 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 --2- 172.21.15.87:0/1911817821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cea1134250 0x55cea114a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 >> 172.21.15.87:0/1911817821 conn(0x55cea112eba0 msgr2=0x55cea113a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 shutdown_connections 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.481+0000 7fe3027fd740 1 -- 172.21.15.87:0/1911817821 wait complete. 2024-04-22T13:43:46.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe3027fd740 20 client.19239 trim_cache size 0 max 16384 2024-04-22T13:43:46.485 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 msgr2=0x7fe2e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 0x7fe2e40a61e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f00098a0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 msgr2=0x7fe2f819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005940 tx=0x7fe2f402c7e0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 shutdown_connections 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe2e40a3d20 0x7fe2e40a61e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2f8198650 0x7fe2f8198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f81072b0 0x7fe2f819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 --2- 172.21.15.87:0/1568931568 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2f8100af0 0x7fe2f819e590 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 >> 172.21.15.87:0/1568931568 conn(0x7fe2f80760e0 msgr2=0x7fe2f80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 shutdown_connections 2024-04-22T13:43:46.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:46.482+0000 7fe2fff75640 1 -- 172.21.15.87:0/1568931568 wait complete. 2024-04-22T13:43:46.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:46 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2046506845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:46.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:46 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2046506845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:46.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:46 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2046506845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:47.490 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T13:43:47.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:43:47.490 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T13:43:47.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 -- 172.21.15.87:0/850602974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc06a820 msgr2=0x7ff5bc06ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 --2- 172.21.15.87:0/850602974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc06a820 0x7ff5bc06ac00 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009900 tx=0x7ff5b002bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 -- 172.21.15.87:0/850602974 shutdown_connections 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 --2- 172.21.15.87:0/850602974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 --2- 172.21.15.87:0/850602974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06b1d0 0x7ff5bc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 --2- 172.21.15.87:0/850602974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc06a820 0x7ff5bc06ac00 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 -- 172.21.15.87:0/850602974 >> 172.21.15.87:0/850602974 conn(0x7ff5bc077eb0 msgr2=0x7ff5bc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 -- 172.21.15.87:0/850602974 shutdown_connections 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.609+0000 7ff5c3243640 1 -- 172.21.15.87:0/850602974 wait complete. 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 Processor -- start 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 -- start start 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06a820 0x7ff5bc1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1013c0 con 0x7ff5bc19ad40 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc101240 con 0x7ff5bc06a820 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c3243640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc101540 con 0x7ff5bc107270 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c17e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c17e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55910/0 (socket says 172.21.15.87:55910) 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c17e0640 1 -- 172.21.15.87:0/3164785636 learned_addr learned my addr 172.21.15.87:0/3164785636 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06a820 0x7ff5bc1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.610+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 -- 172.21.15.87:0/3164785636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 msgr2=0x7ff5bc1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 -- 172.21.15.87:0/3164785636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06a820 msgr2=0x7ff5bc1a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06a820 0x7ff5bc1a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 -- 172.21.15.87:0/3164785636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5bc19b920 con 0x7ff5bc19ad40 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400b970 tx=0x7ff5b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5b9ffb640 1 -- 172.21.15.87:0/3164785636 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4019070 con 0x7ff5bc19ad40 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5bc1a7520 con 0x7ff5bc19ad40 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5b9ffb640 1 -- 172.21.15.87:0/3164785636 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5b4014440 con 0x7ff5bc19ad40 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5b9ffb640 1 -- 172.21.15.87:0/3164785636 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b40075d0 con 0x7ff5bc19ad40 2024-04-22T13:43:47.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.611+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc1a78d0 con 0x7ff5bc19ad40 2024-04-22T13:43:47.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.613+0000 7ff5b9ffb640 1 -- 172.21.15.87:0/3164785636 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff5b401c070 con 0x7ff5bc19ad40 2024-04-22T13:43:47.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 Processor -- start 2024-04-22T13:43:47.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 -- start start 2024-04-22T13:43:47.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5b9ffb640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 0x7ff5ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5b9ffb640 1 -- 172.21.15.87:0/3164785636 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff5b40cfbe0 con 0x7ff5bc19ad40 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 0x55bca2cbd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 0x7ff5ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bca2c15f00 con 0x55bca2cb2290 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bca2bb3590 con 0x55bca2cbae10 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bca2c11ec0 con 0x55bca2cb18e0 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55918/0 (socket says 172.21.15.87:55918) 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 -- 172.21.15.87:0/1310795249 learned_addr learned my addr 172.21.15.87:0/1310795249 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 0x55bca2cbd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 -- 172.21.15.87:0/1310795249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 msgr2=0x55bca2cb1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.614+0000 7ff5bbfff640 1 -- 172.21.15.87:0/1310795249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 msgr2=0x55bca2cbd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 0x55bca2cbd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5bbfff640 1 -- 172.21.15.87:0/1310795249 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca29f07e0 con 0x55bca2cb2290 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 0x55bca2cbd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 0x7ff5ac0a61e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc06a400 tx=0x7ff5a800ab60 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8010740 tx=0x7ff5a801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb446d66a5c728a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5a6ffd640 1 -- 172.21.15.87:0/1310795249 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8023070 con 0x55bca2cb2290 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5a6ffd640 1 -- 172.21.15.87:0/1310795249 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5a800a460 con 0x55bca2cb2290 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.615+0000 7ff5a6ffd640 1 -- 172.21.15.87:0/1310795249 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8004a00 con 0x55bca2cb2290 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1310795249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 msgr2=0x55bca2cba840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8010740 tx=0x7ff5a801a000 comp rx=0 tx=0).stop 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1310795249 shutdown_connections 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cbae10 0x55bca2cbd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cb2290 0x55bca2cba840 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1310795249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1310795249 >> 172.21.15.87:0/1310795249 conn(0x55bca2cabfb0 msgr2=0x55bca2cac390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1310795249 shutdown_connections 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1310795249 wait complete. 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- start start 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 Processor -- start 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cb8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55bca2cbdd20 con 0x55bca2cbae10 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bca2bcf580 con 0x55bca2cb2290 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bca2cbdea0 con 0x55bca2cb18e0 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.616+0000 7ff5c5e3b740 10 client.0 _ll_register_callbacks cb 0x55bca2bd5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55934/0 (socket says 172.21.15.87:55934) 2024-04-22T13:43:47.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 -- 172.21.15.87:0/1876662980 learned_addr learned my addr 172.21.15.87:0/1876662980 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cb8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 -- 172.21.15.87:0/1876662980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 msgr2=0x55bca2cb7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 -- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 msgr2=0x55bca2cb8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cb8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 -- 172.21.15.87:0/1876662980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bca2a50f70 con 0x55bca2cbae10 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cb8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40cd970 tx=0x7ff5b40a0080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ae766f010bcf9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff59bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b40b5060 con 0x55bca2cbae10 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff5b40a44b0 con 0x55bca2cbae10 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bca2c23920 con 0x55bca2cbae10 2024-04-22T13:43:47.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff5b409a140 con 0x55bca2cbae10 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c5e3b740 10 client.19251 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bca28e8170 con 0x55bca2cbae10 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.617+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bca2c23920 con 0x55bca2cbae10 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff5b407aba0 con 0x55bca2cbae10 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 10 client.19251 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 10 client.19251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 20 client.19251 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2ccb3b0 0x55bca2ccd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 4 client.19251 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bca2c1d790 con 0x55bca2ccb3b0 2024-04-22T13:43:47.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.618+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2ccb3b0 0x55bca2ccd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff59bfff640 10 client.19251 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2ccb3b0 0x55bca2ccd7a0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d3d5df5dad477f1 server_cookie=405eb6c592e92e37 in_seq=0 out_seq=0 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff59bfff640 1 -- 172.21.15.87:0/1876662980 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bca2c1d790 con 0x55bca2ccb3b0 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff59bfff640 10 client.19251 handle_command_reply: tid=0 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 client.19251 shutdown 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 msgr2=0x55bca2cb8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40cd970 tx=0x7ff5b40a0080 comp rx=0 tx=0).stop 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 shutdown_connections 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2ccb3b0 0x55bca2ccd7a0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cbae10 0x55bca2cb8850 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cb8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff59bfff640 0 client.19251 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/1876662980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.619+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 >> 172.21.15.87:0/1876662980 conn(0x55bca2cabfb0 msgr2=0x55bca2cadd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.620+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 shutdown_connections 2024-04-22T13:43:47.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.620+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/1876662980 wait complete. 2024-04-22T13:43:47.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.620+0000 7ff5c5e3b740 20 client.19251 trim_cache size 0 max 16384 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 Processor -- start 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 -- start start 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cd3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1b24a0 con 0x55bca2cabfb0 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1afd70 con 0x55bca2cb2290 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1adbe0 con 0x55bca2cb18e0 2024-04-22T13:43:47.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55950/0 (socket says 172.21.15.87:55950) 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/211179203 learned_addr learned my addr 172.21.15.87:0/211179203 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cd3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/211179203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 msgr2=0x55bca2cb84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/211179203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 msgr2=0x55bca2cd3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cd3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/211179203 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca2ccddd0 con 0x55bca2cabfb0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.638+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cd3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.639+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0004390 tx=0x7ff5b0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3eba19ef5edf27 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.639+0000 7ff5a67fc640 1 -- 172.21.15.87:0/211179203 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b004e020 con 0x55bca2cabfb0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.639+0000 7ff5a67fc640 1 -- 172.21.15.87:0/211179203 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5b002cc90 con 0x55bca2cabfb0 2024-04-22T13:43:47.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.639+0000 7ff5a67fc640 1 -- 172.21.15.87:0/211179203 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0048070 con 0x55bca2cabfb0 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/211179203 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 msgr2=0x55bca2cc6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0004390 tx=0x7ff5b0005fd0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/211179203 shutdown_connections 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cd3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cb84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/211179203 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc6c60 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/211179203 >> 172.21.15.87:0/211179203 conn(0x55bca2cb6a10 msgr2=0x55bca2cb7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/211179203 shutdown_connections 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/211179203 wait complete. 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- start start 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 Processor -- start 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cc0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cc0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1ae230 con 0x55bca2cabfb0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1af6a0 con 0x55bca2cb2290 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc1ae900 con 0x55bca2cb18e0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c5e3b740 10 client.0 _ll_register_callbacks cb 0x7ff5b409b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.640+0000 7ff5c0fdf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55952/0 (socket says 172.21.15.87:55952) 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/4130163886 learned_addr learned my addr 172.21.15.87:0/4130163886 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cc0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cc0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/4130163886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 msgr2=0x55bca2cc0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cc0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 msgr2=0x55bca2cc0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cc0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 -- 172.21.15.87:0/4130163886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bca2cc0d20 con 0x55bca2cabfb0 2024-04-22T13:43:47.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c17e0640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cc0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cc0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c0fdf640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0005190 tx=0x7ff5b004abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20430b95a24f7e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff59bfff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0059060 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff5b00052d0 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5b002cc90 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff5b004e030 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c5e3b740 10 client.19257 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bca2d32770 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.641+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5b002cc90 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff5b0003710 con 0x55bca2cabfb0 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 10 client.19257 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 10 client.19257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 20 client.19257 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2cbae10 0x55bca2d34f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 4 client.19257 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T13:43:47.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bca2b99c50 con 0x55bca2cbae10 2024-04-22T13:43:47.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2cbae10 0x55bca2d34f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff59bfff640 10 client.19257 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.642+0000 7ff5bbfff640 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2cbae10 0x55bca2d34f60 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38334e60fff107fc server_cookie=9c56e08226021922 in_seq=0 out_seq=0 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff59bfff640 1 -- 172.21.15.87:0/4130163886 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bca2b99c50 con 0x55bca2cbae10 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff59bfff640 10 client.19257 handle_command_reply: tid=0 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 client.19257 shutdown 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 msgr2=0x55bca2cc3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0005190 tx=0x7ff5b004abd0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 shutdown_connections 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55bca2cbae10 0x55bca2d34f60 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bca2cb2290 0x55bca2cc0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bca2cb18e0 0x55bca2cc0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 --2- 172.21.15.87:0/4130163886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55bca2cabfb0 0x55bca2cc3410 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 >> 172.21.15.87:0/4130163886 conn(0x55bca2cb6a10 msgr2=0x55bca2cb8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff59bfff640 0 client.19257 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 shutdown_connections 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 1 -- 172.21.15.87:0/4130163886 wait complete. 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.644+0000 7ff5c5e3b740 20 client.19257 trim_cache size 0 max 16384 2024-04-22T13:43:47.646 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:47.647 INFO:teuthology.orchestra.run.smithi087.stdout: "return_code": 0, 2024-04-22T13:43:47.647 INFO:teuthology.orchestra.run.smithi087.stdout: "scrub_tag": "d4818ad6-945c-45f0-8daf-b85c8411b27d", 2024-04-22T13:43:47.647 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": "asynchronous" 2024-04-22T13:43:47.647 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 msgr2=0x7ff5ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 0x7ff5ac0a61e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc06a400 tx=0x7ff5a800ab60 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 msgr2=0x7ff5bc19b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400b970 tx=0x7ff5b400be30 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 shutdown_connections 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5ac0a3d20 0x7ff5ac0a61e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5bc19ad40 0x7ff5bc19b1f0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5bc107270 0x7ff5bc1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 --2- 172.21.15.87:0/3164785636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5bc06a820 0x7ff5bc1a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 >> 172.21.15.87:0/3164785636 conn(0x7ff5bc077eb0 msgr2=0x7ff5bc105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 shutdown_connections 2024-04-22T13:43:47.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.645+0000 7ff5c3243640 1 -- 172.21.15.87:0/3164785636 wait complete. 2024-04-22T13:43:47.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:43:47.652 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:43:47.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 -- 172.21.15.87:0/3107175829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88601010b0 msgr2=0x7f8860101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 --2- 172.21.15.87:0/3107175829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88601010b0 0x7f8860101490 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f885c009900 tx=0x7f885c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:47.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 -- 172.21.15.87:0/3107175829 shutdown_connections 2024-04-22T13:43:47.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 --2- 172.21.15.87:0/3107175829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8860106180 0x7f886010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 --2- 172.21.15.87:0/3107175829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860101a60 0x7f8860105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 --2- 172.21.15.87:0/3107175829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88601010b0 0x7f8860101490 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 -- 172.21.15.87:0/3107175829 >> 172.21.15.87:0/3107175829 conn(0x7f88600771d0 msgr2=0x7f88600775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 -- 172.21.15.87:0/3107175829 shutdown_connections 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.758+0000 7f8867081640 1 -- 172.21.15.87:0/3107175829 wait complete. 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 Processor -- start 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 -- start start 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 0x7f8860197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860106180 0x7f8860197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f88601110f0 con 0x7f886019bf10 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8860110f70 con 0x7f8860106180 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8867081640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8860111270 con 0x7f88601010b0 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55968/0 (socket says 172.21.15.87:55968) 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 -- 172.21.15.87:0/1738795951 learned_addr learned my addr 172.21.15.87:0/1738795951 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8857fff640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860106180 0x7f8860197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8864e1d640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 0x7f8860197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 -- 172.21.15.87:0/1738795951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 msgr2=0x7f8860197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 0x7f8860197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 -- 172.21.15.87:0/1738795951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860106180 msgr2=0x7f8860197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860106180 0x7f8860197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 -- 172.21.15.87:0/1738795951 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886019ca00 con 0x7f886019bf10 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f8864e1d640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 0x7f8860197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.759+0000 7f886561e640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f885800eaa0 tx=0x7f885800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.760+0000 7f8855ffb640 1 -- 172.21.15.87:0/1738795951 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800cdb0 con 0x7f886019bf10 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.760+0000 7f8855ffb640 1 -- 172.21.15.87:0/1738795951 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8858005020 con 0x7f886019bf10 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.760+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886019ccf0 con 0x7f886019bf10 2024-04-22T13:43:47.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.760+0000 7f8855ffb640 1 -- 172.21.15.87:0/1738795951 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88580116d0 con 0x7f886019bf10 2024-04-22T13:43:47.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.760+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601a44e0 con 0x7f886019bf10 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8855ffb640 1 -- 172.21.15.87:0/1738795951 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f88580051d0 con 0x7f886019bf10 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8869c75740 1 Processor -- start 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8869c75740 1 -- start start 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8855ffb640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 0x7f88480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8869c75740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8869c75740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 0x55a9627a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.762+0000 7f8857fff640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 0x7f88480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8855ffb640 1 -- 172.21.15.87:0/1738795951 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f88580d09b0 con 0x7f886019bf10 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55978/0 (socket says 172.21.15.87:55978) 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8869c75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a9627a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a9627021e0 con 0x55a96279d580 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 -- 172.21.15.87:0/186784482 learned_addr learned my addr 172.21.15.87:0/186784482 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a96269f390 con 0x55a9627a6a80 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a9626fe1a0 con 0x55a96279dea0 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8857fff640 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 0x55a9627a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 -- 172.21.15.87:0/186784482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 msgr2=0x55a9627a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 0x55a9627a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 -- 172.21.15.87:0/186784482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 msgr2=0x55a9627a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a9627a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 -- 172.21.15.87:0/186784482 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9624de7d0 con 0x55a96279d580 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8857fff640 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 0x55a9627a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8857fff640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 0x7f88480a61e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8850005d00 tx=0x7f885000bf90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8864e1d640 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c02c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73e76d5aee14923 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8846ffd640 1 -- 172.21.15.87:0/186784482 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c043020 con 0x55a96279d580 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8846ffd640 1 -- 172.21.15.87:0/186784482 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885c02c7e0 con 0x55a96279d580 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.763+0000 7f8846ffd640 1 -- 172.21.15.87:0/186784482 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c005520 con 0x55a96279d580 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 msgr2=0x55a96279d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f885c036040 tx=0x7f885c02c680 comp rx=0 tx=0).stop 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 shutdown_connections 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a9627a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279dea0 0x55a9627a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 --2- 172.21.15.87:0/186784482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a96279d960 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 >> 172.21.15.87:0/186784482 conn(0x55a962797cb0 msgr2=0x55a962798090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 shutdown_connections 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- 172.21.15.87:0/186784482 wait complete. 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 -- start start 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.764+0000 7f8869c75740 1 Processor -- start 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55a9627a9810 con 0x55a9627a6a80 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a9626bb2f0 con 0x55a96279dea0 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a9627a9990 con 0x55a96279d580 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 10 client.0 _ll_register_callbacks cb 0x55a9626c8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55994/0 (socket says 172.21.15.87:55994) 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 -- 172.21.15.87:0/4133080911 learned_addr learned my addr 172.21.15.87:0/4133080911 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8864e1d640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8857fff640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 -- 172.21.15.87:0/4133080911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 msgr2=0x55a9627a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 -- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 msgr2=0x55a9627a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 -- 172.21.15.87:0/4133080911 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9624143f0 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8857fff640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8864e1d640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f886561e640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f88580ce740 tx=0x7f885809c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec684637b21907b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f883ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885801f040 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f88580870c0 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.765+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f885c02c7e0 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f88580a5600 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 10 client.19269 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a9623d40f0 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885c02c7e0 con 0x55a9627a6a80 2024-04-22T13:43:47.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f885800f040 con 0x55a9627a6a80 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 10 client.19269 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 10 client.19269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 20 client.19269 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a9627b6b10 0x55a9627b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 4 client.19269 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a96270fc00 con 0x55a9627b6b10 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.766+0000 7f8864e1d640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a9627b6b10 0x55a9627b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f883ffff640 10 client.19269 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8864e1d640 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a9627b6b10 0x55a9627b8f30 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4fe9ab7fac89e37 server_cookie=1a69d94d4ccbd5a6 in_seq=0 out_seq=0 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f883ffff640 1 -- 172.21.15.87:0/4133080911 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a96270fc00 con 0x55a9627b6b10 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f883ffff640 10 client.19269 handle_command_reply: tid=0 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 client.19269 shutdown 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 msgr2=0x55a9627a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f88580ce740 tx=0x7f885809c3e0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 shutdown_connections 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a9627b6b10 0x55a9627b8f30 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a9627a6a80 0x55a9627a44c0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f883ffff640 0 client.19269 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 --2- 172.21.15.87:0/4133080911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.767+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 >> 172.21.15.87:0/4133080911 conn(0x55a962797cb0 msgr2=0x55a962799a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.768+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 shutdown_connections 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.768+0000 7f8869c75740 1 -- 172.21.15.87:0/4133080911 wait complete. 2024-04-22T13:43:47.771 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.768+0000 7f8869c75740 20 client.19269 trim_cache size 0 max 16384 2024-04-22T13:43:47.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 Processor -- start 2024-04-22T13:43:47.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 -- start start 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627b2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a9627a6a80 0x55a9627a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f88601af2e0 con 0x55a96279d580 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f88601acbb0 con 0x55a96279dea0 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8869c75740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f88601aaa20 con 0x55a9627a6a80 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56002/0 (socket says 172.21.15.87:56002) 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 -- 172.21.15.87:0/2367896874 learned_addr learned my addr 172.21.15.87:0/2367896874 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f886561e640 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a9627a6a80 0x55a9627a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 -- 172.21.15.87:0/2367896874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a9627a6a80 msgr2=0x55a9627a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a9627a6a80 0x55a9627a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 -- 172.21.15.87:0/2367896874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 msgr2=0x55a9627b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.785+0000 7f8864e1d640 1 -- 172.21.15.87:0/2367896874 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9627b5660 con 0x55a96279d580 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8864e1d640 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f885c0051f0 tx=0x7f885c005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80682d7c6d9c884 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f88467fc640 1 -- 172.21.15.87:0/2367896874 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c043040 con 0x55a96279d580 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f88467fc640 1 -- 172.21.15.87:0/2367896874 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885c005420 con 0x55a96279d580 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f88467fc640 1 -- 172.21.15.87:0/2367896874 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c034440 con 0x55a96279d580 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 -- 172.21.15.87:0/2367896874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 msgr2=0x55a9627a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f885c0051f0 tx=0x7f885c005220 comp rx=0 tx=0).stop 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 -- 172.21.15.87:0/2367896874 shutdown_connections 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a9627a6a80 0x55a9627a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a96279dea0 0x55a9627b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 --2- 172.21.15.87:0/2367896874 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279d580 0x55a9627a6170 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 -- 172.21.15.87:0/2367896874 >> 172.21.15.87:0/2367896874 conn(0x55a962797cb0 msgr2=0x55a962799100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 -- 172.21.15.87:0/2367896874 shutdown_connections 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.786+0000 7f8869c75740 1 -- 172.21.15.87:0/2367896874 wait complete. 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 -- start start 2024-04-22T13:43:47.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 Processor -- start 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a96279fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f88601ab070 con 0x55a96279dea0 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f88601ac4e0 con 0x55a9627a6a80 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f88601ab740 con 0x55a96279d580 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8869c75740 10 client.0 _ll_register_callbacks cb 0x7f885800cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56012/0 (socket says 172.21.15.87:56012) 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 -- 172.21.15.87:0/1915923400 learned_addr learned my addr 172.21.15.87:0/1915923400 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f886561e640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a96279fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8864e1d640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 -- 172.21.15.87:0/1915923400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 msgr2=0x55a9627a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 -- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 msgr2=0x55a96279fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a96279fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.787+0000 7f8857fff640 1 -- 172.21.15.87:0/1915923400 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9627a8e70 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8857fff640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8860100770 tx=0x7f8850004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63acece2cd360f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850024070 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f8850029660 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a962709a70 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8850029880 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 10 client.19275 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a96281f9b0 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a962709a70 con 0x55a96279dea0 2024-04-22T13:43:47.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8850020b00 con 0x55a96279dea0 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 10 client.19275 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 10 client.19275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 20 client.19275 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a962820110 0x55a962822530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 4 client.19275 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.788+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a962685590 con 0x55a962820110 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8864e1d640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a962820110 0x55a962822530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8864e1d640 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a962820110 0x55a962822530 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d016ccfdd19ac98 server_cookie=ac70e5131dddae90 in_seq=0 out_seq=0 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f883ffff640 10 client.19275 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f883ffff640 1 -- 172.21.15.87:0/1915923400 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a962685590 con 0x55a962820110 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f883ffff640 10 client.19275 handle_command_reply: tid=0 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 client.19275 shutdown 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 msgr2=0x55a9627a14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8860100770 tx=0x7f8850004730 comp rx=0 tx=0).stop 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 shutdown_connections 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55a962820110 0x55a962822530 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a9627a6a80 0x55a96279fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55a96279dea0 0x55a9627a14f0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f883ffff640 0 client.19275 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 --2- 172.21.15.87:0/1915923400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a96279d580 0x55a9627a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 >> 172.21.15.87:0/1915923400 conn(0x55a962797cb0 msgr2=0x55a962796d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 shutdown_connections 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 1 -- 172.21.15.87:0/1915923400 wait complete. 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.789+0000 7f8869c75740 20 client.19275 trim_cache size 0 max 16384 2024-04-22T13:43:47.792 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": { 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "d4818ad6-945c-45f0-8daf-b85c8411b27d": { 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "path": "/", 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "tag": "d4818ad6-945c-45f0-8daf-b85c8411b27d", 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: "options": "recursive,force" 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:43:47.793 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 msgr2=0x7f88480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 0x7f88480a61e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8850005d00 tx=0x7f885000bf90 comp rx=0 tx=0).stop 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 msgr2=0x7f886019c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f885800eaa0 tx=0x7f885800ef60 comp rx=0 tx=0).stop 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 shutdown_connections 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f88480a3d20 0x7f88480a61e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f886019bf10 0x7f886019c3c0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8860106180 0x7f8860197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 --2- 172.21.15.87:0/1738795951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88601010b0 0x7f8860197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:47.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 >> 172.21.15.87:0/1738795951 conn(0x7f88600771d0 msgr2=0x7f88600fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:47.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 shutdown_connections 2024-04-22T13:43:47.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:47.790+0000 7f8867081640 1 -- 172.21.15.87:0/1738795951 wait complete. 2024-04-22T13:43:47.796 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:47 smithi087 ceph-mon[27999]: pgmap v373: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:47.797 INFO:tasks.cephfs.filesystem:scrub status for tag:d4818ad6-945c-45f0-8daf-b85c8411b27d - {'path': '/', 'tag': 'd4818ad6-945c-45f0-8daf-b85c8411b27d', 'options': 'recursive,force'} 2024-04-22T13:43:47.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:47 smithi120 ceph-mon[31101]: pgmap v373: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:47.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:47 smithi132 ceph-mon[31531]: pgmap v373: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T13:43:48.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:48 smithi120 ceph-mon[31101]: scrub queued for path: / 2024-04-22T13:43:48.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:48 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:48.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:48 smithi120 ceph-mon[31101]: scrub summary: active paths [/] 2024-04-22T13:43:48.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:48 smithi120 ceph-mon[31101]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:48.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:48 smithi132 ceph-mon[31531]: scrub queued for path: / 2024-04-22T13:43:48.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:48 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:48.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:48 smithi132 ceph-mon[31531]: scrub summary: active paths [/] 2024-04-22T13:43:48.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:48 smithi132 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:48.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:48 smithi087 ceph-mon[27999]: scrub queued for path: / 2024-04-22T13:43:48.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:48 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:48.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:48 smithi087 ceph-mon[27999]: scrub summary: active paths [/] 2024-04-22T13:43:48.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:48 smithi087 ceph-mon[27999]: scrub summary: idle+waiting paths [/] 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 -- 172.21.15.87:0/3130931642 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff834107fe0 msgr2=0x7ff8341083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 --2- 172.21.15.87:0/3130931642 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff834107fe0 0x7ff8341083c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff828009930 tx=0x7ff82802bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 -- 172.21.15.87:0/3130931642 shutdown_connections 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 --2- 172.21.15.87:0/3130931642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406b120 0x7ff83410edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 --2- 172.21.15.87:0/3130931642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff83406a780 0x7ff83406abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 --2- 172.21.15.87:0/3130931642 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff834107fe0 0x7ff8341083c0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 -- 172.21.15.87:0/3130931642 >> 172.21.15.87:0/3130931642 conn(0x7ff834076c50 msgr2=0x7ff834077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 -- 172.21.15.87:0/3130931642 shutdown_connections 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.058+0000 7ff83ab50640 1 -- 172.21.15.87:0/3130931642 wait complete. 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 Processor -- start 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 -- start start 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 0x7ff834198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff834107fe0 0x7ff83419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff834112190 con 0x7ff83406b120 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff834112010 con 0x7ff83406a780 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff83ab50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff834112310 con 0x7ff834107fe0 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff833fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff833fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56032/0 (socket says 172.21.15.87:56032) 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff833fff640 1 -- 172.21.15.87:0/4243310627 learned_addr learned my addr 172.21.15.87:0/4243310627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.059+0000 7ff8390ed640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff834107fe0 0x7ff83419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff8388ec640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 0x7ff834198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 -- 172.21.15.87:0/4243310627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff834107fe0 msgr2=0x7ff83419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff834107fe0 0x7ff83419cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 -- 172.21.15.87:0/4243310627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 msgr2=0x7ff834198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 0x7ff834198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 -- 172.21.15.87:0/4243310627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83419d660 con 0x7ff83406b120 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff8388ec640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 0x7ff834198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff833fff640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff82000d770 tx=0x7ff82000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff831ffb640 1 -- 172.21.15.87:0/4243310627 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820017070 con 0x7ff83406b120 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff831ffb640 1 -- 172.21.15.87:0/4243310627 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8200054d0 con 0x7ff83406b120 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83419d950 con 0x7ff83406b120 2024-04-22T13:43:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff831ffb640 1 -- 172.21.15.87:0/4243310627 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820004c30 con 0x7ff83406b120 2024-04-22T13:43:49.060 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.060+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8341a51e0 con 0x7ff83406b120 2024-04-22T13:43:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.062+0000 7ff831ffb640 1 -- 172.21.15.87:0/4243310627 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff820004e10 con 0x7ff83406b120 2024-04-22T13:43:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff831ffb640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 0x7ff8240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 Processor -- start 2024-04-22T13:43:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 -- start start 2024-04-22T13:43:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff8388ec640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 0x7ff8240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff831ffb640 1 -- 172.21.15.87:0/4243310627 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff8200d0290 con 0x7ff83406b120 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b67620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b67fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9acc8d0 con 0x55c5c9b710e0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9a69e80 con 0x55c5c9b688d0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff83d819740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9ac8890 con 0x55c5c9b67b60 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56036/0 (socket says 172.21.15.87:56036) 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.063+0000 7ff8390ed640 1 -- 172.21.15.87:0/979879289 learned_addr learned my addr 172.21.15.87:0/979879289 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8388ec640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 0x7ff8240a6280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff828006010 tx=0x7ff828009610 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff833fff640 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b67fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8388ec640 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b67620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 -- 172.21.15.87:0/979879289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 msgr2=0x55c5c9b67fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b67fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 -- 172.21.15.87:0/979879289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 msgr2=0x55c5c9b67620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b67620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 -- 172.21.15.87:0/979879289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5c9a8b320 con 0x55c5c9b710e0 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff8390ed640 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009fd0 tx=0x7ff82c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa823ad50ddd36e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff81effd640 1 -- 172.21.15.87:0/979879289 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c004d20 con 0x55c5c9b710e0 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff81effd640 1 -- 172.21.15.87:0/979879289 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c004050 con 0x55c5c9b710e0 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.064+0000 7ff81effd640 1 -- 172.21.15.87:0/979879289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c005260 con 0x55c5c9b710e0 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- 172.21.15.87:0/979879289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 msgr2=0x55c5c9b734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009fd0 tx=0x7ff82c0118d0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- 172.21.15.87:0/979879289 shutdown_connections 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b734d0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b67fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 --2- 172.21.15.87:0/979879289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b67620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- 172.21.15.87:0/979879289 >> 172.21.15.87:0/979879289 conn(0x55c5c9b2ec90 msgr2=0x55c5c9b62850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- 172.21.15.87:0/979879289 shutdown_connections 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- 172.21.15.87:0/979879289 wait complete. 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 -- start start 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.065+0000 7ff83d819740 1 Processor -- start 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b70a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b6de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9b73e10 con 0x55c5c9b710e0 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9a54eb0 con 0x55c5c9b688d0 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5c9b73f90 con 0x55c5c9b67b60 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff83d819740 10 client.0 _ll_register_callbacks cb 0x55c5c9a81b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56048/0 (socket says 172.21.15.87:56048) 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 -- 172.21.15.87:0/641261324 learned_addr learned my addr 172.21.15.87:0/641261324 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff833fff640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b6de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8388ec640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b70a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 -- 172.21.15.87:0/641261324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 msgr2=0x55c5c9b70a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b70a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 -- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 msgr2=0x55c5c9b6de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b6de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 -- 172.21.15.87:0/641261324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5c97df210 con 0x55c5c9b710e0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff8390ed640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff82c006010 tx=0x7ff82c003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a515c1f9daa126f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff813fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c028020 con 0x55c5c9b710e0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.066+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff82c004410 con 0x55c5c9b710e0 2024-04-22T13:43:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c004050 con 0x55c5c9b710e0 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff82c033050 con 0x55c5c9b710e0 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 10 client.19287 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c5c98ab620 con 0x55c5c9b710e0 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c004050 con 0x55c5c9b710e0 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff82c00f4e0 con 0x55c5c9b710e0 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 10 client.19287 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 10 client.19287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 20 client.19287 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9b81090 0x55c5c9b83480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 4 client.19287 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5c9ae0880 con 0x55c5c9b81090 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.067+0000 7ff8388ec640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9b81090 0x55c5c9b83480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.068+0000 7ff813fff640 10 client.19287 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.068+0000 7ff8388ec640 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9b81090 0x55c5c9b83480 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a6b5fd8f5200c6 server_cookie=eb5ac28d79ce68f4 in_seq=0 out_seq=0 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.068+0000 7ff813fff640 1 -- 172.21.15.87:0/641261324 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5c9ae0880 con 0x55c5c9b81090 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.068+0000 7ff813fff640 10 client.19287 handle_command_reply: tid=0 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.068+0000 7ff83d819740 1 client.19287 shutdown 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 msgr2=0x55c5c9b6e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff82c006010 tx=0x7ff82c003c60 comp rx=0 tx=0).stop 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 shutdown_connections 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9b81090 0x55c5c9b83480 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6e3c0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b6de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff813fff640 0 client.19287 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 --2- 172.21.15.87:0/641261324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b70a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 >> 172.21.15.87:0/641261324 conn(0x55c5c9b2ec90 msgr2=0x55c5c9b645a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 shutdown_connections 2024-04-22T13:43:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 1 -- 172.21.15.87:0/641261324 wait complete. 2024-04-22T13:43:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.069+0000 7ff83d819740 20 client.19287 trim_cache size 0 max 16384 2024-04-22T13:43:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.083+0000 7ff83d819740 1 Processor -- start 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 -- start start 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b7cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b79600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b0310 con 0x55c5c9b710e0 2024-04-22T13:43:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341adbe0 con 0x55c5c9b688d0 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff83d819740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341aba50 con 0x55c5c9b67b60 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56054/0 (socket says 172.21.15.87:56054) 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 -- 172.21.15.87:0/3766168959 learned_addr learned my addr 172.21.15.87:0/3766168959 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff833fff640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b7cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 -- 172.21.15.87:0/3766168959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 msgr2=0x55c5c9b7cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b7cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 -- 172.21.15.87:0/3766168959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 msgr2=0x55c5c9b79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b79600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 -- 172.21.15.87:0/3766168959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5c9b5df30 con 0x55c5c9b710e0 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b7cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff8390ed640 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0099e0 tx=0x7ff82c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b134826f18096c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.084+0000 7ff81e7fc640 1 -- 172.21.15.87:0/3766168959 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c028040 con 0x55c5c9b710e0 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff81e7fc640 1 -- 172.21.15.87:0/3766168959 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c00f6e0 con 0x55c5c9b710e0 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff81e7fc640 1 -- 172.21.15.87:0/3766168959 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c019430 con 0x55c5c9b710e0 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- 172.21.15.87:0/3766168959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 msgr2=0x55c5c9b70550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0099e0 tx=0x7ff82c003970 comp rx=0 tx=0).stop 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- 172.21.15.87:0/3766168959 shutdown_connections 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b710e0 0x55c5c9b70550 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b688d0 0x55c5c9b79600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 --2- 172.21.15.87:0/3766168959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b67b60 0x55c5c9b7cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- 172.21.15.87:0/3766168959 >> 172.21.15.87:0/3766168959 conn(0x55c5c9b2ec90 msgr2=0x55c5c9b69080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- 172.21.15.87:0/3766168959 shutdown_connections 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- 172.21.15.87:0/3766168959 wait complete. 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 -- start start 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.085+0000 7ff83d819740 1 Processor -- start 2024-04-22T13:43:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 0x55c5c9b726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341ac0a0 con 0x55c5c9b688d0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341ad510 con 0x55c5c9b67b60 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341ac770 con 0x55c5c9b710e0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff83d819740 10 client.0 _ll_register_callbacks cb 0x7ff82c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56058/0 (socket says 172.21.15.87:56058) 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 -- 172.21.15.87:0/3567583043 learned_addr learned my addr 172.21.15.87:0/3567583043 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff8390ed640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 0x55c5c9b726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 -- 172.21.15.87:0/3567583043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b710e0 msgr2=0x55c5c9b6f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 -- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 msgr2=0x55c5c9b726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 0x55c5c9b726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 -- 172.21.15.87:0/3567583043 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5c9b6fc10 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 0x55c5c9b726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff833fff640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff8200986d0 tx=0x7ff82009ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a43e84db7832845 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.086+0000 7ff813fff640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200ba060 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7ff8200cdca0 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff82c00f6e0 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff8200b5070 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 10 client.19293 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c5c9beae70 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff82c00f6e0 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff8200a4a50 con 0x55c5c9b688d0 2024-04-22T13:43:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 10 client.19293 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 10 client.19293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 20 client.19293 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9beb5c0 0x55c5c9bed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 4 client.19293 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c5c9a6c3d0 con 0x55c5c9beb5c0 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.087+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9beb5c0 0x55c5c9bed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.088+0000 7ff8388ec640 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9beb5c0 0x55c5c9bed9e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f3c2071cbfe25d5 server_cookie=63947e1ac575b18c in_seq=0 out_seq=0 2024-04-22T13:43:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.088+0000 7ff813fff640 10 client.19293 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff813fff640 1 -- 172.21.15.87:0/3567583043 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c5c9a6c3d0 con 0x55c5c9beb5c0 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff813fff640 10 client.19293 handle_command_reply: tid=0 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 client.19293 shutdown 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 msgr2=0x55c5c9b72c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff8200986d0 tx=0x7ff82009ca30 comp rx=0 tx=0).stop 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 shutdown_connections 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55c5c9beb5c0 0x55c5c9bed9e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5c9b710e0 0x55c5c9b6f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55c5c9b688d0 0x55c5c9b72c00 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff813fff640 0 client.19293 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 --2- 172.21.15.87:0/3567583043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5c9b67b60 0x55c5c9b726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 >> 172.21.15.87:0/3567583043 conn(0x55c5c9b2ec90 msgr2=0x55c5c9b61040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 shutdown_connections 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.093+0000 7ff83d819740 1 -- 172.21.15.87:0/3567583043 wait complete. 2024-04-22T13:43:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83d819740 20 client.19293 trim_cache size 0 max 16384 2024-04-22T13:43:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:{ 2024-04-22T13:43:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "message": "", 2024-04-22T13:43:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout: "return_code": 0 2024-04-22T13:43:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stdout:} 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 msgr2=0x7ff8240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 0x7ff8240a6280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff828006010 tx=0x7ff828009610 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 msgr2=0x7ff834198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff82000d770 tx=0x7ff82000dc30 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 shutdown_connections 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff8240a3dc0 0x7ff8240a6280 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff834107fe0 0x7ff83419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff83406b120 0x7ff834198bb0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 --2- 172.21.15.87:0/4243310627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff83406a780 0x7ff834198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 >> 172.21.15.87:0/4243310627 conn(0x7ff834076c50 msgr2=0x7ff8340fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 shutdown_connections 2024-04-22T13:43:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi087.front.sepia.ceph.com.smithi087.stderr:2024-04-22T13:43:49.094+0000 7ff83ab50640 1 -- 172.21.15.87:0/4243310627 wait complete. 2024-04-22T13:43:49.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:49 smithi120 ceph-mon[31101]: pgmap v374: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:43:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:49 smithi132 ceph-mon[31531]: pgmap v374: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:43:49.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:49 smithi087 ceph-mon[27999]: pgmap v374: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T13:43:51.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:51 smithi120 ceph-mon[31101]: pgmap v375: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:51.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:51 smithi120 ceph-mon[31101]: scrub summary: idle 2024-04-22T13:43:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:51 smithi132 ceph-mon[31531]: pgmap v375: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:51.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:51 smithi132 ceph-mon[31531]: scrub summary: idle 2024-04-22T13:43:51.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:51 smithi087 ceph-mon[27999]: pgmap v375: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:51.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:51 smithi087 ceph-mon[27999]: scrub summary: idle 2024-04-22T13:43:53.578 INFO:tasks.workunit.client.0.smithi087.stdout:stopping iogen 2024-04-22T13:43:53.578 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'stopping iogen' 2024-04-22T13:43:53.578 INFO:tasks.workunit.client.0.smithi087.stderr:+ ./iogen -k 2024-04-22T13:43:53.582 INFO:tasks.workunit.client.0.smithi087.stdout:OK 2024-04-22T13:43:53.582 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo OK 2024-04-22T13:43:53.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:53 smithi120 ceph-mon[31101]: pgmap v376: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:53.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:53 smithi132 ceph-mon[31531]: pgmap v376: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:53.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:53 smithi087 ceph-mon[27999]: pgmap v376: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T13:43:55.602 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T13:43:55.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T13:43:55.692 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-22T13:43:55.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T13:43:55.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:55 smithi120 ceph-mon[31101]: pgmap v377: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T13:43:55.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:55 smithi132 ceph-mon[31531]: pgmap v377: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T13:43:55.963 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:55 smithi087 ceph-mon[27999]: pgmap v377: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T13:43:55.965 DEBUG:teuthology.parallel:result is None 2024-04-22T13:43:55.966 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T13:43:55.994 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T13:43:55.994 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-22T13:43:56.006 INFO:tasks.quiescer:joining Quiescers 2024-04-22T13:43:56.006 INFO:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-22T13:43:56.006 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:43:56.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 -- 172.21.15.87:0/4224361308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c1011a0 msgr2=0x7f811c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 --2- 172.21.15.87:0/4224361308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c1011a0 0x7f811c101620 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f8110009900 tx=0x7f811002bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 -- 172.21.15.87:0/4224361308 shutdown_connections 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 --2- 172.21.15.87:0/4224361308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c101b60 0x7f811c10dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 --2- 172.21.15.87:0/4224361308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c1011a0 0x7f811c101620 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 --2- 172.21.15.87:0/4224361308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c106fb0 0x7f811c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 -- 172.21.15.87:0/4224361308 >> 172.21.15.87:0/4224361308 conn(0x7f811c077e30 msgr2=0x7f811c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 -- 172.21.15.87:0/4224361308 shutdown_connections 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.137+0000 7f81224c8640 1 -- 172.21.15.87:0/4224361308 wait complete. 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 Processor -- start 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 -- start start 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 0x7f811c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c101b60 0x7f811c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1110d0 con 0x7f811c106fb0 2024-04-22T13:43:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f811c110f50 con 0x7f811c101b60 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f81224c8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f811c111250 con 0x7f811c1011a0 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44216/0 (socket says 172.21.15.87:44216) 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 -- 172.21.15.87:0/2383463849 learned_addr learned my addr 172.21.15.87:0/2383463849 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f811b7fe640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c101b60 0x7f811c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f811bfff640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 0x7f811c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 -- 172.21.15.87:0/2383463849 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 msgr2=0x7f811c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 0x7f811c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 -- 172.21.15.87:0/2383463849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c101b60 msgr2=0x7f811c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c101b60 0x7f811c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f8120a65640 1 -- 172.21.15.87:0/2383463849 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c19c620 con 0x7f811c106fb0 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.138+0000 7f811bfff640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 0x7f811c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f8120a65640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f811400eaa0 tx=0x7f811400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811400cdb0 con 0x7f811c106fb0 2024-04-22T13:43:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c19c910 con 0x7f811c106fb0 2024-04-22T13:43:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8114005020 con 0x7f811c106fb0 2024-04-22T13:43:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81140116d0 con 0x7f811c106fb0 2024-04-22T13:43:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.139+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c1a41a0 con 0x7f811c106fb0 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.140+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f811c069230 con 0x7f811c106fb0 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.141+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f81140054a0 con 0x7f811c106fb0 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.141+0000 7f81197fa640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 0x7f81040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.142+0000 7f811bfff640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 0x7f81040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.142+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7f81140d0950 con 0x7f811c106fb0 2024-04-22T13:43:56.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.142+0000 7f811bfff640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 0x7f81040a61e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f810c009930 tx=0x7f810c008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.146+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8114016030 con 0x7f811c106fb0 2024-04-22T13:43:56.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.364+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f811c109300 con 0x7f811c106fb0 2024-04-22T13:43:56.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.364+0000 7f81197fa640 1 -- 172.21.15.87:0/2383463849 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f811409d030 con 0x7f811c106fb0 2024-04-22T13:43:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:43:56.361 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:43:56.361 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 msgr2=0x7f81040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 0x7f81040a61e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f810c009930 tx=0x7f810c008040 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 msgr2=0x7f811c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f811400eaa0 tx=0x7f811400ef60 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 shutdown_connections 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f81040a3d20 0x7f81040a61e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f811c106fb0 0x7f811c19bef0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f811c101b60 0x7f811c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 --2- 172.21.15.87:0/2383463849 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f811c1011a0 0x7f811c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.366+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 >> 172.21.15.87:0/2383463849 conn(0x7f811c077e30 msgr2=0x7f811c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.367+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 shutdown_connections 2024-04-22T13:43:56.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.367+0000 7f81224c8640 1 -- 172.21.15.87:0/2383463849 wait complete. 2024-04-22T13:43:56.374 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24482 quiesce db --cancel --all' 2024-04-22T13:43:56.374 DEBUG:teuthology.orchestra.run.smithi087:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24482 quiesce db --cancel --all 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.481+0000 7fc14a043640 1 -- 172.21.15.87:0/3970525584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 msgr2=0x7fc144103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.481+0000 7fc14a043640 1 --2- 172.21.15.87:0/3970525584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144103590 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc134009900 tx=0x7fc13402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 -- 172.21.15.87:0/3970525584 shutdown_connections 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 --2- 172.21.15.87:0/3970525584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc14410c580 0x7fc14410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 --2- 172.21.15.87:0/3970525584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc144103ad0 0x7fc14410bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 --2- 172.21.15.87:0/3970525584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144103590 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 -- 172.21.15.87:0/3970525584 >> 172.21.15.87:0/3970525584 conn(0x7fc1440771d0 msgr2=0x7fc1440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 -- 172.21.15.87:0/3970525584 shutdown_connections 2024-04-22T13:43:56.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 -- 172.21.15.87:0/3970525584 wait complete. 2024-04-22T13:43:56.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 Processor -- start 2024-04-22T13:43:56.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.482+0000 7fc14a043640 1 -- start start 2024-04-22T13:43:56.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc144103ad0 0x7fc144198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc14410c580 0x7fc14419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc144112200 con 0x7fc1441031b0 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc144112080 con 0x7fc144103ad0 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc144112380 con 0x7fc14410c580 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44234/0 (socket says 172.21.15.87:44234) 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 -- 172.21.15.87:0/4112702855 learned_addr learned my addr 172.21.15.87:0/4112702855 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc13affd640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc144103ad0 0x7fc144198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 -- 172.21.15.87:0/4112702855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc14410c580 msgr2=0x7fc14419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc14410c580 0x7fc14419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 -- 172.21.15.87:0/4112702855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc144103ad0 msgr2=0x7fc144198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc144103ad0 0x7fc144198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 -- 172.21.15.87:0/4112702855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14419d620 con 0x7fc1441031b0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1437fe640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc134005ab0 tx=0x7fc13402db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1417fa640 1 -- 172.21.15.87:0/4112702855 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134043020 con 0x7fc1441031b0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1417fa640 1 -- 172.21.15.87:0/4112702855 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1340052b0 con 0x7fc1441031b0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14419d8b0 con 0x7fc1441031b0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc1417fa640 1 -- 172.21.15.87:0/4112702855 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134003c70 con 0x7fc1441031b0 2024-04-22T13:43:56.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.483+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1441a51a0 con 0x7fc1441031b0 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.485+0000 7fc1417fa640 1 -- 172.21.15.87:0/4112702855 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc134046020 con 0x7fc1441031b0 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 Processor -- start 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 -- start start 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3687ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561a367e0000 con 0x561a3687b440 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561a3677cb20 con 0x561a36888650 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561a367dbfc0 con 0x561a3687aa90 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1417fa640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 0x7fc1300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44250/0 (socket says 172.21.15.87:44250) 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 -- 172.21.15.87:0/3566895416 learned_addr learned my addr 172.21.15.87:0/3566895416 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc143fff640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3687ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc13affd640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc143fff640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 0x7fc1300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1417fa640 1 -- 172.21.15.87:0/4112702855 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fc1340f29f0 con 0x7fc1441031b0 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 -- 172.21.15.87:0/3566895416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 msgr2=0x561a3687ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3687ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 -- 172.21.15.87:0/3566895416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 msgr2=0x561a3688aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc1437fe640 1 -- 172.21.15.87:0/3566895416 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a36796ea0 con 0x561a3687b440 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.486+0000 7fc143fff640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3687ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc1340be7e0 tx=0x7fc1340be750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ee82b339d658d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc139ffb640 1 -- 172.21.15.87:0/3566895416 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1340bf1e0 con 0x561a3687b440 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc143fff640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 0x7fc1300a61e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc13c005fd0 tx=0x7fc13c005e90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc139ffb640 1 -- 172.21.15.87:0/3566895416 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1340beae0 con 0x561a3687b440 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc139ffb640 1 -- 172.21.15.87:0/3566895416 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134005460 con 0x561a3687b440 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3566895416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 msgr2=0x561a36888080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.487+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc1340be7e0 tx=0x7fc1340be750 comp rx=0 tx=0).stop 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3566895416 shutdown_connections 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a36888080 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3566895416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3687ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3566895416 >> 172.21.15.87:0/3566895416 conn(0x561a368753e0 msgr2=0x561a368757c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3566895416 shutdown_connections 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3566895416 wait complete. 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- start start 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 Processor -- start 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36885b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a36888650 0x561a36886060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x561a3688b340 con 0x561a3687aa90 2024-04-22T13:43:56.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561a3677a510 con 0x561a3687b440 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561a3688b4c0 con 0x561a36888650 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc14cc3b740 10 client.0 _ll_register_callbacks cb 0x561a36779f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44264/0 (socket says 172.21.15.87:44264) 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.488+0000 7fc1437fe640 1 -- 172.21.15.87:0/996845949 learned_addr learned my addr 172.21.15.87:0/996845949 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc13affd640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36885b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 -- 172.21.15.87:0/996845949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a36888650 msgr2=0x561a36886060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a36888650 0x561a36886060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 -- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 msgr2=0x561a36885b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36885b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 -- 172.21.15.87:0/996845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a365bea20 con 0x561a3687aa90 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc1437fe640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc1340f0340 tx=0x7fc13402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c1063bfbfee472 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134003e10 con 0x561a3687aa90 2024-04-22T13:43:56.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fc13408d1d0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1340beae0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc1340bf1e0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 10 client.19311 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561a3661f1b0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1340beae0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc1340c72c0 con 0x561a3687aa90 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 10 client.19311 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 10 client.19311 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.489+0000 7fc14cc3b740 20 client.19311 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a36898670 0x561a3689aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc14cc3b740 4 client.19311 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a367eda20 con 0x561a36898670 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc13affd640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a36898670 0x561a3689aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc13affd640 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a36898670 0x561a3689aa60 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d62799282adcdb3 server_cookie=ff93a5e86e97c700 in_seq=0 out_seq=0 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.490+0000 7fc122ffd640 10 client.19311 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:56.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc122ffd640 1 -- 172.21.15.87:0/996845949 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a367eda20 con 0x561a36898670 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc122ffd640 10 client.19311 handle_command_reply: tid=0 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 client.19311 shutdown 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 msgr2=0x561a368855e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc1340f0340 tx=0x7fc13402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 shutdown_connections 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a36898670 0x561a3689aa60 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a36888650 0x561a36886060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36885b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc122ffd640 0 client.19311 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/996845949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687aa90 0x561a368855e0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 >> 172.21.15.87:0/996845949 conn(0x561a368753e0 msgr2=0x561a36877130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 shutdown_connections 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 1 -- 172.21.15.87:0/996845949 wait complete. 2024-04-22T13:43:56.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.491+0000 7fc14cc3b740 20 client.19311 trim_cache size 0 max 16384 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 Processor -- start 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 -- start start 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3689aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36894200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ae0b0 con 0x561a36888650 2024-04-22T13:43:56.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ab980 con 0x561a3687b440 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441a97f0 con 0x561a3687aa90 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc143fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.508+0000 7fc143fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44272/0 (socket says 172.21.15.87:44272) 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 -- 172.21.15.87:0/2017188789 learned_addr learned my addr 172.21.15.87:0/2017188789 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc13affd640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36894200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc1437fe640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3689aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 -- 172.21.15.87:0/2017188789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 msgr2=0x561a3689aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3689aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 -- 172.21.15.87:0/2017188789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 msgr2=0x561a36894200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36894200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 -- 172.21.15.87:0/2017188789 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a36884590 con 0x561a36888650 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc1437fe640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3689aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc13affd640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36894200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:56.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc143fff640 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc13c011540 tx=0x7fc13c005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47ddbc353d9fb77 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc1397fa640 1 -- 172.21.15.87:0/2017188789 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c021070 con 0x561a36888650 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc1397fa640 1 -- 172.21.15.87:0/2017188789 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13c01d8e0 con 0x561a36888650 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.509+0000 7fc1397fa640 1 -- 172.21.15.87:0/2017188789 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c01c910 con 0x561a36888650 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- 172.21.15.87:0/2017188789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 msgr2=0x561a36876f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc13c011540 tx=0x7fc13c005c60 comp rx=0 tx=0).stop 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- 172.21.15.87:0/2017188789 shutdown_connections 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a36888650 0x561a36876f40 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a3687b440 0x561a36894200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/2017188789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a3689aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- 172.21.15.87:0/2017188789 >> 172.21.15.87:0/2017188789 conn(0x561a368753e0 msgr2=0x561a36876830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- 172.21.15.87:0/2017188789 shutdown_connections 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- 172.21.15.87:0/2017188789 wait complete. 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- start start 2024-04-22T13:43:56.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 Processor -- start 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a36890900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441a9e40 con 0x561a3687b440 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441ab2b0 con 0x561a36888650 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441aa510 con 0x561a3687aa90 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.510+0000 7fc14cc3b740 10 client.0 _ll_register_callbacks cb 0x7fc1340f0200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44282/0 (socket says 172.21.15.87:44282) 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 -- 172.21.15.87:0/3379576474 learned_addr learned my addr 172.21.15.87:0/3379576474 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc143fff640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a36890900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 -- 172.21.15.87:0/3379576474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 msgr2=0x561a36890900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a36890900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 -- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 msgr2=0x561a3688dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 -- 172.21.15.87:0/3379576474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a364b2310 con 0x561a3687b440 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc143fff640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc13affd640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc12c004700 tx=0x7fc12c00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abbaf3f64eed5fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c01c7c0 con 0x561a3687b440 2024-04-22T13:43:56.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fc12c025420 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc13c01d8e0 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc12c025620 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc14cc3b740 10 client.19317 fetch_fsmap learned FSMap version 23 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561a368ff7d0 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.511+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13c01d8e0 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc12c02e820 con 0x561a3687b440 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 10 client.19317 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 10 client.19317 resolve_mds: validated gid 24482 aka daemon mds.b 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 20 client.19317 populate_metadata read hostname 'smithi087' 2024-04-22T13:43:56.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a368fff80 0x561a369023a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 4 client.19317 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x561a367e7890 con 0x561a368fff80 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a368fff80 0x561a369023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc122ffd640 10 client.19317 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.512+0000 7fc1437fe640 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a368fff80 0x561a369023a0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65da3ffd9fd6e87f server_cookie=35e30967417168fe in_seq=0 out_seq=0 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc122ffd640 1 -- 172.21.15.87:0/3379576474 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x561a367e7890 con 0x561a368fff80 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc122ffd640 10 client.19317 handle_command_reply: tid=0 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 client.19317 shutdown 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 msgr2=0x561a3688d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc12c004700 tx=0x7fc12c00f7f0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 shutdown_connections 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x561a368fff80 0x561a369023a0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561a36888650 0x561a3688dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x561a3687b440 0x561a3688d6e0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc122ffd640 0 client.19317 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 --2- 172.21.15.87:0/3379576474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a3687aa90 0x561a36890900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 >> 172.21.15.87:0/3379576474 conn(0x561a368753e0 msgr2=0x561a36885e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 shutdown_connections 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 1 -- 172.21.15.87:0/3379576474 wait complete. 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14cc3b740 20 client.19317 trim_cache size 0 max 16384 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 23, 2024-04-22T13:43:56.517 INFO:teuthology.orchestra.run.smithi087.stdout: "leader": 24482, 2024-04-22T13:43:56.518 INFO:teuthology.orchestra.run.smithi087.stdout: "set_version": 44, 2024-04-22T13:43:56.518 INFO:teuthology.orchestra.run.smithi087.stdout: "sets": {} 2024-04-22T13:43:56.518 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 msgr2=0x7fc1300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 0x7fc1300a61e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc13c005fd0 tx=0x7fc13c005e90 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 msgr2=0x7fc144198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.513+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc134005ab0 tx=0x7fc13402db40 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 shutdown_connections 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc1300a3d20 0x7fc1300a61e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc14410c580 0x7fc14419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc144103ad0 0x7fc144198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 --2- 172.21.15.87:0/4112702855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1441031b0 0x7fc144198680 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 >> 172.21.15.87:0/4112702855 conn(0x7fc1440771d0 msgr2=0x7fc1440fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 shutdown_connections 2024-04-22T13:43:56.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.514+0000 7fc14a043640 1 -- 172.21.15.87:0/4112702855 wait complete. 2024-04-22T13:43:56.521 INFO:tasks.quiescer:done joining Quiescers 2024-04-22T13:43:56.521 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T13:43:56.533 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:43:56.541 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-22T13:43:56.551 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-22T13:43:56.551 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T13:43:56.551 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:43:56.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-22T13:43:56.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-22T13:43:56.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T13:43:56.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 -- 172.21.15.87:0/116570493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c102b80 msgr2=0x7fa63c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 --2- 172.21.15.87:0/116570493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c102b80 0x7fa63c106bc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009900 tx=0x7fa62c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:43:56.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 -- 172.21.15.87:0/116570493 shutdown_connections 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 --2- 172.21.15.87:0/116570493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1072a0 0x7fa63c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 --2- 172.21.15.87:0/116570493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c102b80 0x7fa63c106bc0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 --2- 172.21.15.87:0/116570493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa63c1021d0 0x7fa63c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.815+0000 7fa64445c640 1 -- 172.21.15.87:0/116570493 >> 172.21.15.87:0/116570493 conn(0x7fa63c0793e0 msgr2=0x7fa63c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- 172.21.15.87:0/116570493 shutdown_connections 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- 172.21.15.87:0/116570493 wait complete. 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 Processor -- start 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- start start 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 0x7fa63c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa63c1072a0 0x7fa63c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112290 con 0x7fa63c19cf90 2024-04-22T13:43:56.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112110 con 0x7fa63c1021d0 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa64445c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112410 con 0x7fa63c1072a0 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa6429f9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa6429f9640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44296/0 (socket says 172.21.15.87:44296) 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.816+0000 7fa6429f9640 1 -- 172.21.15.87:0/2504166829 learned_addr learned my addr 172.21.15.87:0/2504166829 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6421f8640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 0x7fa63c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 -- 172.21.15.87:0/2504166829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa63c1072a0 msgr2=0x7fa63c198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa63c1072a0 0x7fa63c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 -- 172.21.15.87:0/2504166829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 msgr2=0x7fa63c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 0x7fa63c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 -- 172.21.15.87:0/2504166829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c19da80 con 0x7fa63c19cf90 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6421f8640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 0x7fa63c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6429f9640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fa63400cab0 tx=0x7fa63400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63400ae20 con 0x7fa63c19cf90 2024-04-22T13:43:56.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c19dd70 con 0x7fa63c19cf90 2024-04-22T13:43:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa63400f900 con 0x7fa63c19cf90 2024-04-22T13:43:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6340189f0 con 0x7fa63c19cf90 2024-04-22T13:43:56.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.817+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c1a5560 con 0x7fa63c19cf90 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.818+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c04fce0 con 0x7fa63c19cf90 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.819+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa63400fab0 con 0x7fa63c19cf90 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.819+0000 7fa6337fe640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 0x7fa6280a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.820+0000 7fa6421f8640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 0x7fa6280a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.820+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fa6340d1070 con 0x7fa63c19cf90 2024-04-22T13:43:56.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.820+0000 7fa6421f8640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 0x7fa6280a6280 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:56.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:56.824+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa634014030 con 0x7fa63c19cf90 2024-04-22T13:43:56.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2383463849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:56.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2383463849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:56.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2383463849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:43:57.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.023+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa63c102b80 con 0x7fa6280a3dc0 2024-04-22T13:43:57.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.030+0000 7fa6337fe640 1 -- 172.21.15.87:0/2504166829 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fa63c102b80 con 0x7fa6280a3dc0 2024-04-22T13:43:57.026 INFO:teuthology.orchestra.run.smithi087.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T13:31:59", "first": "2024-04-22T13:32:00", "last": "2024-04-22T13:43:00", "last_pruned": "2024-04-22T13:43:00", "created_count": 12, "pruned_count": 6, "active": true} 2024-04-22T13:43:57.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 msgr2=0x7fa6280a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 0x7fa6280a6280 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638008040 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 msgr2=0x7fa63c19d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fa63400cab0 tx=0x7fa63400cf70 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 shutdown_connections 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa6280a3dc0 0x7fa6280a6280 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa63c19cf90 0x7fa63c19d440 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa63c1072a0 0x7fa63c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 --2- 172.21.15.87:0/2504166829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa63c1021d0 0x7fa63c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 >> 172.21.15.87:0/2504166829 conn(0x7fa63c0793e0 msgr2=0x7fa63c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 shutdown_connections 2024-04-22T13:43:57.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.032+0000 7fa64445c640 1 -- 172.21.15.87:0/2504166829 wait complete. 2024-04-22T13:43:57.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.143+0000 7fa7737f4640 1 -- 172.21.15.87:0/691914484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 msgr2=0x7fa76c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.143+0000 7fa7737f4640 1 --2- 172.21.15.87:0/691914484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c10e9b0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fa764009900 tx=0x7fa76402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- 172.21.15.87:0/691914484 shutdown_connections 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- 172.21.15.87:0/691914484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c10e9b0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- 172.21.15.87:0/691914484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c10c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- 172.21.15.87:0/691914484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c1035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- 172.21.15.87:0/691914484 >> 172.21.15.87:0/691914484 conn(0x7fa76c077fd0 msgr2=0x7fa76c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- 172.21.15.87:0/691914484 shutdown_connections 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- 172.21.15.87:0/691914484 wait complete. 2024-04-22T13:43:57.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 Processor -- start 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- start start 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1121a0 con 0x7fa76c10c540 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c112020 con 0x7fa76c103200 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa7737f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c112320 con 0x7fa76c103b20 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.144+0000 7fa771d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44308/0 (socket says 172.21.15.87:44308) 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 -- 172.21.15.87:0/2320567039 learned_addr learned my addr 172.21.15.87:0/2320567039 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771590640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa770d8f640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 -- 172.21.15.87:0/2320567039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 msgr2=0x7fa76c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 -- 172.21.15.87:0/2320567039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 msgr2=0x7fa76c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 -- 172.21.15.87:0/2320567039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c19d620 con 0x7fa76c10c540 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771590640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa770d8f640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa771d91640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fa764009230 tx=0x7fa7640045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764043020 con 0x7fa76c10c540 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa76402dc20 con 0x7fa76c10c540 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c19d8b0 con 0x7fa76c10c540 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640038e0 con 0x7fa76c10c540 2024-04-22T13:43:57.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.145+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c1a51a0 con 0x7fa76c10c540 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.146+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa76c04fce0 con 0x7fa76c10c540 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.147+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa764046020 con 0x7fa76c10c540 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.148+0000 7fa7627fc640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 0x7fa7580a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.148+0000 7fa771590640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 0x7fa7580a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.148+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8455+0+0 (secure 0 0 0) 0x7fa7640f2c70 con 0x7fa76c10c540 2024-04-22T13:43:57.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.148+0000 7fa771590640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 0x7fa7580a6280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa768005c40 tx=0x7fa7680082f0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:43:57.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.151+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa76404d020 con 0x7fa76c10c540 2024-04-22T13:43:57.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.351+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 --> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa76c107730 con 0x7fa7580a3dc0 2024-04-22T13:43:57.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.358+0000 7fa7627fc640 1 -- 172.21.15.87:0/2320567039 <== mgr.15303 v2:172.21.15.87:6800/2766661417 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa7680089b0 con 0x7fa7580a3dc0 2024-04-22T13:43:57.355 INFO:teuthology.orchestra.run.smithi087.stdout:/ 1m 6m3h 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 msgr2=0x7fa7580a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 0x7fa7580a6280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa768005c40 tx=0x7fa7680082f0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 msgr2=0x7fa76c19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fa764009230 tx=0x7fa7640045e0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 shutdown_connections 2024-04-22T13:43:57.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fa7580a3dc0 0x7fa7580a6280 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa76c10c540 0x7fa76c19cef0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa76c103b20 0x7fa76c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 --2- 172.21.15.87:0/2320567039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa76c103200 0x7fa76c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 >> 172.21.15.87:0/2320567039 conn(0x7fa76c077fd0 msgr2=0x7fa76c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 shutdown_connections 2024-04-22T13:43:57.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:43:57.361+0000 7fa7737f4640 1 -- 172.21.15.87:0/2320567039 wait complete. 2024-04-22T13:43:57.368 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T13:43:57.379 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T13:43:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:57 smithi120 ceph-mon[31101]: pgmap v378: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T13:43:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:57 smithi120 ceph-mon[31101]: from='client.19323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:57.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:57 smithi132 ceph-mon[31531]: pgmap v378: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T13:43:57.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:57 smithi132 ceph-mon[31531]: from='client.19323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:57.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:57 smithi087 ceph-mon[27999]: pgmap v378: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T13:43:57.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:57 smithi087 ceph-mon[27999]: from='client.19323 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:58.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:58 smithi120 ceph-mon[31101]: from='client.19329 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:58.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:58 smithi132 ceph-mon[31531]: from='client.19329 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:58 smithi087 ceph-mon[27999]: from='client.19329 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T13:43:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:43:59 smithi120 ceph-mon[31101]: pgmap v379: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-04-22T13:43:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:43:59 smithi132 ceph-mon[31531]: pgmap v379: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-04-22T13:43:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:43:59 smithi087 ceph-mon[27999]: pgmap v379: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-04-22T13:44:01.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:01 smithi120 ceph-mon[31101]: pgmap v380: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-22T13:44:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:01 smithi132 ceph-mon[31531]: pgmap v380: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-22T13:44:01.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:01 smithi087 ceph-mon[27999]: pgmap v380: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-22T13:44:03.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:03 smithi132 ceph-mon[31531]: pgmap v381: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 53 op/s 2024-04-22T13:44:03.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:03 smithi087 ceph-mon[27999]: pgmap v381: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 53 op/s 2024-04-22T13:44:04.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:03 smithi120 ceph-mon[31101]: pgmap v381: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 19 KiB/s wr, 53 op/s 2024-04-22T13:44:05.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:05 smithi132 ceph-mon[31531]: pgmap v382: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 212 op/s 2024-04-22T13:44:05.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:05 smithi087 ceph-mon[27999]: pgmap v382: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 212 op/s 2024-04-22T13:44:06.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:05 smithi120 ceph-mon[31101]: pgmap v382: 97 pgs: 97 active+clean; 12 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 212 op/s 2024-04-22T13:44:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:07 smithi132 ceph-mon[31531]: pgmap v383: 97 pgs: 97 active+clean; 12 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 241 op/s 2024-04-22T13:44:07.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:07 smithi132 ceph-mon[31531]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T13:44:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:07 smithi087 ceph-mon[27999]: pgmap v383: 97 pgs: 97 active+clean; 12 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 241 op/s 2024-04-22T13:44:07.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:07 smithi087 ceph-mon[27999]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T13:44:08.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:07 smithi120 ceph-mon[31101]: pgmap v383: 97 pgs: 97 active+clean; 12 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 69 KiB/s wr, 241 op/s 2024-04-22T13:44:08.171 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:07 smithi120 ceph-mon[31101]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T13:44:09.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:09 smithi132 ceph-mon[31531]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 62 KiB/s wr, 289 op/s 2024-04-22T13:44:09.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:09 smithi087 ceph-mon[27999]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 62 KiB/s wr, 289 op/s 2024-04-22T13:44:10.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:09 smithi120 ceph-mon[31101]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 62 KiB/s wr, 289 op/s 2024-04-22T13:44:11.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:11 smithi132 ceph-mon[31531]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 281 op/s 2024-04-22T13:44:11.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:11 smithi087 ceph-mon[27999]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 281 op/s 2024-04-22T13:44:12.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:11 smithi120 ceph-mon[31101]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 281 op/s 2024-04-22T13:44:13.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:13 smithi132 ceph-mon[31531]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 233 op/s 2024-04-22T13:44:13.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:13 smithi087 ceph-mon[27999]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 233 op/s 2024-04-22T13:44:14.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:13 smithi120 ceph-mon[31101]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 62 KiB/s wr, 233 op/s 2024-04-22T13:44:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:15 smithi120 ceph-mon[31101]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 44 op/s 2024-04-22T13:44:15.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:15 smithi120 ceph-mon[31101]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T13:44:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:15 smithi132 ceph-mon[31531]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 44 op/s 2024-04-22T13:44:15.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:15 smithi132 ceph-mon[31531]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T13:44:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:15 smithi087 ceph-mon[27999]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 44 op/s 2024-04-22T13:44:15.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:15 smithi087 ceph-mon[27999]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T13:44:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:16 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:44:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:16 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:16 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:16 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:16 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:16 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:44:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:16 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:16 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:16 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:16 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:16 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T13:44:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:16 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:16 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:16 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:16.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:16 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:17.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T13:44:17.798 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- 172.21.15.87:0/3369262575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc103ae0 msgr2=0x7fd2bc10bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- 172.21.15.87:0/3369262575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc10bfc0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0009900 tx=0x7fd2b002bd60 comp rx=0 tx=0).stop 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- 172.21.15.87:0/3369262575 shutdown_connections 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- 172.21.15.87:0/3369262575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc10c500 0x7fd2bc10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- 172.21.15.87:0/3369262575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc10bfc0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- 172.21.15.87:0/3369262575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 0x7fd2bc1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- 172.21.15.87:0/3369262575 >> 172.21.15.87:0/3369262575 conn(0x7fd2bc078000 msgr2=0x7fd2bc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- 172.21.15.87:0/3369262575 shutdown_connections 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- 172.21.15.87:0/3369262575 wait complete. 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 Processor -- start 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 -- start start 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 0x7fd2bc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.906+0000 7fd2c1477640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc112190 con 0x7fd2bc10c500 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc112010 con 0x7fd2bc103ae0 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc112310 con 0x7fd2bc1031c0 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46306/0 (socket says 172.21.15.87:46306) 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/1116951387 learned_addr learned my addr 172.21.15.87:0/1116951387 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:17.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2baffd640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 0x7fd2bc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/1116951387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 msgr2=0x7fd2bc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 0x7fd2bc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/1116951387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc103ae0 msgr2=0x7fd2bc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/1116951387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc19d6e0 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400d750 tx=0x7fd2b400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2a3fff640 1 -- 172.21.15.87:0/1116951387 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4017070 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2a3fff640 1 -- 172.21.15.87:0/1116951387 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b40054d0 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc19d9d0 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2a3fff640 1 -- 172.21.15.87:0/1116951387 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004c60 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.907+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc1a5260 con 0x7fd2bc10c500 2024-04-22T13:44:17.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.909+0000 7fd2a3fff640 1 -- 172.21.15.87:0/1116951387 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd2b4009bd0 con 0x7fd2bc10c500 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 Processor -- start 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 -- start start 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2a3fff640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 0x7fd2a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 0x55e6d8822960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822ea0 0x55e6d882b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 0x55e6d882de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e6d87871e0 con 0x55e6d8822ea0 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6d8724390 con 0x55e6d882ba80 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2c4060740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6d87831a0 con 0x55e6d8822580 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 0x7fd2a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2a3fff640 1 -- 172.21.15.87:0/1116951387 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fd2b40d00f0 con 0x7fd2bc10c500 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 0x55e6d882de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 0x55e6d882de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.87:46020/0 (socket says 172.21.15.87:46020) 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/753310735 learned_addr learned my addr 172.21.15.87:0/753310735 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2baffd640 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 0x55e6d8822960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.910+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822ea0 0x55e6d882b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 -- 172.21.15.87:0/753310735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 msgr2=0x55e6d882de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 0x55e6d882de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 -- 172.21.15.87:0/753310735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822ea0 msgr2=0x55e6d882b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822ea0 0x55e6d882b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 -- 172.21.15.87:0/753310735 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6d85637d0 con 0x55e6d8822580 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 0x7fd2a40a61e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc10e350 tx=0x7fd2b002c780 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2baffd640 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 0x55e6d8822960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009fb0 tx=0x7fd2ac010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37284a0f8f8c12db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2a17fa640 1 -- 172.21.15.87:0/753310735 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac016070 con 0x55e6d8822580 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2a17fa640 1 -- 172.21.15.87:0/753310735 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ac004d20 con 0x55e6d8822580 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.911+0000 7fd2a17fa640 1 -- 172.21.15.87:0/753310735 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac005350 con 0x55e6d8822580 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- 172.21.15.87:0/753310735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 msgr2=0x55e6d8822960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 0x55e6d8822960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009fb0 tx=0x7fd2ac010800 comp rx=0 tx=0).stop 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- 172.21.15.87:0/753310735 shutdown_connections 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d882ba80 0x55e6d882de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822ea0 0x55e6d882b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- 172.21.15.87:0/753310735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822580 0x55e6d8822960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- 172.21.15.87:0/753310735 >> 172.21.15.87:0/753310735 conn(0x55e6d881ccb0 msgr2=0x55e6d881d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- 172.21.15.87:0/753310735 shutdown_connections 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- 172.21.15.87:0/753310735 wait complete. 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- start start 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 Processor -- start 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822ea0 0x55e6d8828f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d882ba80 0x55e6d8829460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e6d882e870 con 0x55e6d8822580 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6d87402f0 con 0x55e6d8822ea0 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e6d882e9f0 con 0x55e6d882ba80 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2c4060740 10 client.0 _ll_register_callbacks cb 0x55e6d874d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46330/0 (socket says 172.21.15.87:46330) 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2baffd640 1 -- 172.21.15.87:0/2788663139 learned_addr learned my addr 172.21.15.87:0/2788663139 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d882ba80 0x55e6d8829460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.912+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822ea0 0x55e6d8828f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 -- 172.21.15.87:0/2788663139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d882ba80 msgr2=0x55e6d8829460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d882ba80 0x55e6d8829460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 -- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822ea0 msgr2=0x55e6d8828f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822ea0 0x55e6d8828f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 -- 172.21.15.87:0/2788663139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6d84993f0 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2baffd640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac004ca0 tx=0x7fd2ac009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e6ab48e6553081 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac029060 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fd2ac012720 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2ac004d20 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd2ac016070 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 10 client.19341 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e6d84590f0 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ac004d20 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd2ac00f040 con 0x55e6d8822580 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 10 client.19341 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:17.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 10 client.19341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 20 client.19341 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d883b990 0x55e6d883dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 4 client.19341 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.913+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6d8794c00 con 0x55e6d883b990 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.914+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d883b990 0x55e6d883dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.914+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d883b990 0x55e6d883dd80 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19081650dfc241be server_cookie=4a3049fca7d2eb2f in_seq=0 out_seq=0 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.914+0000 7fd29eff5640 10 client.19341 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd29eff5640 1 -- 172.21.15.87:0/2788663139 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6d8794c00 con 0x55e6d883b990 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd29eff5640 10 client.19341 handle_command_reply: tid=0 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 client.19341 shutdown 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 msgr2=0x55e6d88289e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac004ca0 tx=0x7fd2ac009bc0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 shutdown_connections 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d883b990 0x55e6d883dd80 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d882ba80 0x55e6d8829460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822ea0 0x55e6d8828f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd29eff5640 0 client.19341 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2788663139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d8822580 0x55e6d88289e0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 >> 172.21.15.87:0/2788663139 conn(0x55e6d881ccb0 msgr2=0x55e6d881ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 shutdown_connections 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 1 -- 172.21.15.87:0/2788663139 wait complete. 2024-04-22T13:44:17.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.915+0000 7fd2c4060740 20 client.19341 trim_cache size 0 max 16384 2024-04-22T13:44:17.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:17 smithi120 ceph-mon[31101]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-22T13:44:17.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:17 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:44:17.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:17 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:44:17.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:17 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 Processor -- start 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 -- start start 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d882b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8837730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b02d0 con 0x55e6d882ba80 2024-04-22T13:44:17.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1adba0 con 0x55e6d8822580 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.932+0000 7fd2c4060740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1aba10 con 0x55e6d8822ea0 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46340/0 (socket says 172.21.15.87:46340) 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/2799090096 learned_addr learned my addr 172.21.15.87:0/2799090096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/2799090096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 msgr2=0x55e6d8837730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8837730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/2799090096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 msgr2=0x55e6d882b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2baffd640 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d882b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d882b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/2799090096 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6d8823f30 con 0x55e6d882ba80 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2baffd640 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d882b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd2b409c060 tx=0x7fd2b4098370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8593c00a37fcf573 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2a0ff9640 1 -- 172.21.15.87:0/2799090096 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4020040 con 0x55e6d882ba80 2024-04-22T13:44:17.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2a0ff9640 1 -- 172.21.15.87:0/2799090096 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b40a0480 con 0x55e6d882ba80 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.933+0000 7fd2a0ff9640 1 -- 172.21.15.87:0/2799090096 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4098740 con 0x55e6d882ba80 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- 172.21.15.87:0/2799090096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 msgr2=0x55e6d8827940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd2b409c060 tx=0x7fd2b4098370 comp rx=0 tx=0).stop 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- 172.21.15.87:0/2799090096 shutdown_connections 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d8827940 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8837730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- 172.21.15.87:0/2799090096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d882b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- 172.21.15.87:0/2799090096 >> 172.21.15.87:0/2799090096 conn(0x55e6d881ccb0 msgr2=0x55e6d881bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- 172.21.15.87:0/2799090096 shutdown_connections 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- 172.21.15.87:0/2799090096 wait complete. 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- start start 2024-04-22T13:44:17.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 Processor -- start 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d8825a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8825fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1ac060 con 0x55e6d882ba80 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1ad4d0 con 0x55e6d8822580 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1ac730 con 0x55e6d8822ea0 2024-04-22T13:44:17.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2c4060740 10 client.0 _ll_register_callbacks cb 0x7fd2b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46356/0 (socket says 172.21.15.87:46356) 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.934+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/3380548413 learned_addr learned my addr 172.21.15.87:0/3380548413 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2baffd640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d8825a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2ba7fc640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8825fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/3380548413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 msgr2=0x55e6d8825fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8825fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 msgr2=0x55e6d8825a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d8825a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 -- 172.21.15.87:0/3380548413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6d8826870 con 0x55e6d882ba80 2024-04-22T13:44:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2baffd640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d8825a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2bb7fe640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40a4810 tx=0x7fd2b4009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448a02c9ef53794b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4020040 con 0x55e6d882ba80 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fd2b40828a0 con 0x55e6d882ba80 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2b40a0480 con 0x55e6d882ba80 2024-04-22T13:44:17.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd2b4085180 con 0x55e6d882ba80 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 10 client.19347 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e6d88a2210 con 0x55e6d882ba80 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b40a0480 con 0x55e6d882ba80 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fd2b40b0ce0 con 0x55e6d882ba80 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 10 client.19347 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 10 client.19347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 20 client.19347 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d88a2870 0x55e6d88a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 4 client.19347 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.935+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e6d878ea70 con 0x55e6d88a2870 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2baffd640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d88a2870 0x55e6d88a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:17.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd29eff5640 10 client.19347 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2baffd640 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d88a2870 0x55e6d88a4c60 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c84fd85042c8d515 server_cookie=61a1e36f102f377a in_seq=0 out_seq=0 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd29eff5640 1 -- 172.21.15.87:0/3380548413 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e6d878ea70 con 0x55e6d88a2870 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd29eff5640 10 client.19347 handle_command_reply: tid=0 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 client.19347 shutdown 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 msgr2=0x55e6d88a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40a4810 tx=0x7fd2b4009f00 comp rx=0 tx=0).stop 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 shutdown_connections 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x55e6d88a2870 0x55e6d88a4c60 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e6d882ba80 0x55e6d88a0c50 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e6d8822ea0 0x55e6d8825fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd29eff5640 0 client.19347 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:17.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 --2- 172.21.15.87:0/3380548413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6d8822580 0x55e6d8825a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.936+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 >> 172.21.15.87:0/3380548413 conn(0x55e6d881ccb0 msgr2=0x55e6d88287d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 shutdown_connections 2024-04-22T13:44:17.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c4060740 1 -- 172.21.15.87:0/3380548413 wait complete. 2024-04-22T13:44:17.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c4060740 20 client.19347 trim_cache size 0 max 16384 2024-04-22T13:44:17.953 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:17.953 INFO:teuthology.orchestra.run.smithi087.stdout: "status": "no active scrubs running", 2024-04-22T13:44:17.953 INFO:teuthology.orchestra.run.smithi087.stdout: "scrubs": {} 2024-04-22T13:44:17.953 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:17.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 msgr2=0x7fd2a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 0x7fd2a40a61e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc10e350 tx=0x7fd2b002c780 comp rx=0 tx=0).stop 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 msgr2=0x7fd2bc19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400d750 tx=0x7fd2b400dc10 comp rx=0 tx=0).stop 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 shutdown_connections 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2a40a3d20 0x7fd2a40a61e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2bc10c500 0x7fd2bc19cfb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2bc103ae0 0x7fd2bc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 --2- 172.21.15.87:0/1116951387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2bc1031c0 0x7fd2bc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:17.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 >> 172.21.15.87:0/1116951387 conn(0x7fd2bc078000 msgr2=0x7fd2bc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:17.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 shutdown_connections 2024-04-22T13:44:17.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:17.937+0000 7fd2c1477640 1 -- 172.21.15.87:0/1116951387 wait complete. 2024-04-22T13:44:17.956 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:17 smithi087 ceph-mon[27999]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-22T13:44:17.956 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:17 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:44:17.956 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:17 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:44:17.957 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:17 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:17.958 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T13:44:17.958 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:44:17.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:17 smithi132 ceph-mon[31531]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-04-22T13:44:17.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:17 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T13:44:17.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:17 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T13:44:17.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:17 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' 2024-04-22T13:44:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 -- 172.21.15.87:0/1669746482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda341072f0 msgr2=0x7fda3410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 --2- 172.21.15.87:0/1669746482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda341072f0 0x7fda3410ee70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009ca0 tx=0x7fda2c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 -- 172.21.15.87:0/1669746482 shutdown_connections 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 --2- 172.21.15.87:0/1669746482 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda341072f0 0x7fda3410ee70 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 --2- 172.21.15.87:0/1669746482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda3406b2a0 0x7fda34106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 --2- 172.21.15.87:0/1669746482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3406a8f0 0x7fda3406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 -- 172.21.15.87:0/1669746482 >> 172.21.15.87:0/1669746482 conn(0x7fda34077e30 msgr2=0x7fda34078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 -- 172.21.15.87:0/1669746482 shutdown_connections 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.060+0000 7fda39123640 1 -- 172.21.15.87:0/1669746482 wait complete. 2024-04-22T13:44:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 Processor -- start 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 -- start start 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda341072f0 0x7fda34198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3419cfb0 0x7fda3419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fda341121f0 con 0x7fda3406a8f0 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda34112070 con 0x7fda3419cfb0 2024-04-22T13:44:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda39123640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda34112370 con 0x7fda341072f0 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda32d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda32d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46368/0 (socket says 172.21.15.87:46368) 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda32d9d640 1 -- 172.21.15.87:0/486181695 learned_addr learned my addr 172.21.15.87:0/486181695 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.061+0000 7fda3359e640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3419cfb0 0x7fda3419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda2bfff640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda341072f0 0x7fda34198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 -- 172.21.15.87:0/486181695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda341072f0 msgr2=0x7fda34198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda341072f0 0x7fda34198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 -- 172.21.15.87:0/486181695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3419cfb0 msgr2=0x7fda3419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3419cfb0 0x7fda3419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 -- 172.21.15.87:0/486181695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3419daa0 con 0x7fda3406a8f0 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda32d9d640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d950 tx=0x7fda2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24017070 con 0x7fda3406a8f0 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3419dd90 con 0x7fda3406a8f0 2024-04-22T13:44:18.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda24009c30 con 0x7fda3406a8f0 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2400f620 con 0x7fda3406a8f0 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.062+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda341a55d0 con 0x7fda3406a8f0 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.063+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda34069230 con 0x7fda3406a8f0 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.064+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda24010460 con 0x7fda3406a8f0 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.065+0000 7fda30d99640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 0x7fda200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.065+0000 7fda2bfff640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 0x7fda200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.065+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fda240d0400 con 0x7fda3406a8f0 2024-04-22T13:44:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.065+0000 7fda2bfff640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 0x7fda200a61e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fda34100d80 tx=0x7fda1c005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.069+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda2409cae0 con 0x7fda3406a8f0 2024-04-22T13:44:18.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.289+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda3410a4a0 con 0x7fda3406a8f0 2024-04-22T13:44:18.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.290+0000 7fda30d99640 1 -- 172.21.15.87:0/486181695 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fda24009de0 con 0x7fda3406a8f0 2024-04-22T13:44:18.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:18.286 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:44:18.287 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:44:18.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 msgr2=0x7fda200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 0x7fda200a61e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fda34100d80 tx=0x7fda1c005f90 comp rx=0 tx=0).stop 2024-04-22T13:44:18.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 msgr2=0x7fda341986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d950 tx=0x7fda2400de10 comp rx=0 tx=0).stop 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 shutdown_connections 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fda200a3d20 0x7fda200a61e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda3419cfb0 0x7fda3419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda341072f0 0x7fda34198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 --2- 172.21.15.87:0/486181695 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda3406a8f0 0x7fda341986f0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 >> 172.21.15.87:0/486181695 conn(0x7fda34077e30 msgr2=0x7fda340ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 shutdown_connections 2024-04-22T13:44:18.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.292+0000 7fda39123640 1 -- 172.21.15.87:0/486181695 wait complete. 2024-04-22T13:44:18.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 -- 172.21.15.87:0/1011362007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5410e830 msgr2=0x7f1a54110c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 --2- 172.21.15.87:0/1011362007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5410e830 0x7f1a54110c20 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009ca0 tx=0x7f1a4c02c180 comp rx=0 tx=0).stop 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 -- 172.21.15.87:0/1011362007 shutdown_connections 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 --2- 172.21.15.87:0/1011362007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a5410e830 0x7f1a54110c20 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 --2- 172.21.15.87:0/1011362007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a54074b40 0x7f1a5410e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 --2- 172.21.15.87:0/1011362007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074220 0x7f1a54074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 -- 172.21.15.87:0/1011362007 >> 172.21.15.87:0/1011362007 conn(0x7f1a540793a0 msgr2=0x7f1a540797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 -- 172.21.15.87:0/1011362007 shutdown_connections 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.404+0000 7f1a59aba640 1 -- 172.21.15.87:0/1011362007 wait complete. 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 Processor -- start 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 -- start start 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074b40 0x7f1a541a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5410e830 0x7f1a5419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541143a0 con 0x7f1a54074220 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54114220 con 0x7f1a5410e830 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a59aba640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54114520 con 0x7f1a54074b40 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46386/0 (socket says 172.21.15.87:46386) 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a537fe640 1 -- 172.21.15.87:0/3481969762 learned_addr learned my addr 172.21.15.87:0/3481969762 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.405+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074b40 0x7f1a541a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a53fff640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5410e830 0x7f1a5419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 -- 172.21.15.87:0/3481969762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074b40 msgr2=0x7f1a541a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074b40 0x7f1a541a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 -- 172.21.15.87:0/3481969762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5410e830 msgr2=0x7f1a5419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5410e830 0x7f1a5419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 -- 172.21.15.87:0/3481969762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5419b000 con 0x7f1a54074220 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a537fe640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800d950 tx=0x7f1a4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a50ff9640 1 -- 172.21.15.87:0/3481969762 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48017030 con 0x7f1a54074220 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a50ff9640 1 -- 172.21.15.87:0/3481969762 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a48004020 con 0x7f1a54074220 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a5419b2f0 con 0x7f1a54074220 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a50ff9640 1 -- 172.21.15.87:0/3481969762 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a480047a0 con 0x7f1a54074220 2024-04-22T13:44:18.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.406+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a541a7600 con 0x7f1a54074220 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a50ff9640 1 -- 172.21.15.87:0/3481969762 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a48009a30 con 0x7f1a54074220 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 Processor -- start 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 -- start start 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 0x55b7e06fce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b7e06561f0 con 0x55b7e06f1e80 2024-04-22T13:44:18.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b7e05f34b0 con 0x55b7e06f14d0 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a5c784740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b7e06521b0 con 0x55b7e06faa00 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46394/0 (socket says 172.21.15.87:46394) 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 -- 172.21.15.87:0/2742980937 learned_addr learned my addr 172.21.15.87:0/2742980937 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a53fff640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 0x55b7e06fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a50ff9640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 0x7f1a3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 -- 172.21.15.87:0/2742980937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 msgr2=0x55b7e06fce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 0x55b7e06fce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 -- 172.21.15.87:0/2742980937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 msgr2=0x55b7e06f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06f18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a537fe640 1 -- 172.21.15.87:0/2742980937 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e05e51b0 con 0x55b7e06f1e80 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.409+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 0x55b7e06fce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a53fff640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 0x7f1a3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1a480cd920 tx=0x7f1a480c7310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d2e04523bb8cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a50ff9640 1 -- 172.21.15.87:0/3481969762 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f1a480d0220 con 0x7f1a54074220 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a39ffb640 1 -- 172.21.15.87:0/2742980937 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4809bfd0 con 0x55b7e06f1e80 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a39ffb640 1 -- 172.21.15.87:0/2742980937 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a480aa880 con 0x55b7e06f1e80 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.410+0000 7f1a39ffb640 1 -- 172.21.15.87:0/2742980937 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4809bb60 con 0x55b7e06f1e80 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- 172.21.15.87:0/2742980937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 msgr2=0x55b7e06fa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a53fff640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 0x7f1a3c0a61e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c02c790 tx=0x7f1a4c0095c0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1a480cd920 tx=0x7f1a480c7310 comp rx=0 tx=0).stop 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- 172.21.15.87:0/2742980937 shutdown_connections 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06faa00 0x55b7e06fce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e06fa430 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2742980937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- 172.21.15.87:0/2742980937 >> 172.21.15.87:0/2742980937 conn(0x55b7e06ebba0 msgr2=0x55b7e06ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- 172.21.15.87:0/2742980937 shutdown_connections 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- 172.21.15.87:0/2742980937 wait complete. 2024-04-22T13:44:18.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- start start 2024-04-22T13:44:18.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 Processor -- start 2024-04-22T13:44:18.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06f7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55b7e06fd8b0 con 0x55b7e06faa00 2024-04-22T13:44:18.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b7e060f500 con 0x55b7e06f14d0 2024-04-22T13:44:18.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b7e06fda30 con 0x55b7e06f1e80 2024-04-22T13:44:18.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.411+0000 7f1a5c784740 10 client.0 _ll_register_callbacks cb 0x55b7e05ddb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46396/0 (socket says 172.21.15.87:46396) 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 -- 172.21.15.87:0/880260372 learned_addr learned my addr 172.21.15.87:0/880260372 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a537fe640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 -- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 msgr2=0x55b7e06f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 -- 172.21.15.87:0/880260372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 msgr2=0x55b7e06fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06fa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 -- 172.21.15.87:0/880260372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e03683f0 con 0x55b7e06faa00 2024-04-22T13:44:18.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:18.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a53fff640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009ca0 tx=0x7f1a4c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf9ff6af9fa62e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c050020 con 0x55b7e06faa00 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f1a4c032760 con 0x55b7e06faa00 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a480aa880 con 0x55b7e06faa00 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1a4c039040 con 0x55b7e06faa00 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a5c784740 10 client.19365 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b7e04327d0 con 0x55b7e06faa00 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.412+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a480aa880 con 0x55b7e06faa00 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1a4c03f040 con 0x55b7e06faa00 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 10 client.19365 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 10 client.19365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 20 client.19365 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e070aaa0 0x55b7e070cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 4 client.19365 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7e05f5930 con 0x55b7e070aaa0 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a537fe640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e070aaa0 0x55b7e070cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a32ffd640 10 client.19365 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.413+0000 7f1a537fe640 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e070aaa0 0x55b7e070cec0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57dbb6ce540fe3e9 server_cookie=fa946d1262bbd204 in_seq=0 out_seq=0 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a32ffd640 1 -- 172.21.15.87:0/880260372 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7e05f5930 con 0x55b7e070aaa0 2024-04-22T13:44:18.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a32ffd640 10 client.19365 handle_command_reply: tid=0 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 client.19365 shutdown 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 msgr2=0x55b7e06f7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009ca0 tx=0x7f1a4c005d80 comp rx=0 tx=0).stop 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 shutdown_connections 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e070aaa0 0x55b7e070cec0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06faa00 0x55b7e06f7bb0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 --2- 172.21.15.87:0/880260372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06f14d0 0x55b7e06fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a32ffd640 0 client.19365 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 >> 172.21.15.87:0/880260372 conn(0x55b7e06ebba0 msgr2=0x55b7e06ed8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 shutdown_connections 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 1 -- 172.21.15.87:0/880260372 wait complete. 2024-04-22T13:44:18.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.414+0000 7f1a5c784740 20 client.19365 trim_cache size 0 max 16384 2024-04-22T13:44:18.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.427+0000 7f1a5c784740 1 Processor -- start 2024-04-22T13:44:18.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.427+0000 7f1a5c784740 1 -- start start 2024-04-22T13:44:18.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541b2630 con 0x55b7e06f14d0 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541aff00 con 0x55b7e06faa00 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a5c784740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541add70 con 0x55b7e06f1e80 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46408/0 (socket says 172.21.15.87:46408) 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 -- 172.21.15.87:0/906464232 learned_addr learned my addr 172.21.15.87:0/906464232 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 -- 172.21.15.87:0/906464232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 msgr2=0x55b7e06eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 -- 172.21.15.87:0/906464232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 msgr2=0x55b7e06f9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 -- 172.21.15.87:0/906464232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e06f7f20 con 0x55b7e06f14d0 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a537fe640 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1a4809a4b0 tx=0x7f1a4800df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b722171d61b19011 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a397fa640 1 -- 172.21.15.87:0/906464232 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a480d0040 con 0x55b7e06f14d0 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a397fa640 1 -- 172.21.15.87:0/906464232 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4809c900 con 0x55b7e06f14d0 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.428+0000 7f1a397fa640 1 -- 172.21.15.87:0/906464232 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4809bd40 con 0x55b7e06f14d0 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- 172.21.15.87:0/906464232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 msgr2=0x55b7e07067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1a4809a4b0 tx=0x7f1a4800df40 comp rx=0 tx=0).stop 2024-04-22T13:44:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- 172.21.15.87:0/906464232 shutdown_connections 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f1e80 0x55b7e06eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 --2- 172.21.15.87:0/906464232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f14d0 0x55b7e07067a0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- 172.21.15.87:0/906464232 >> 172.21.15.87:0/906464232 conn(0x55b7e06ebba0 msgr2=0x55b7e06ea370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- 172.21.15.87:0/906464232 shutdown_connections 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- 172.21.15.87:0/906464232 wait complete. 2024-04-22T13:44:18.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.429+0000 7f1a5c784740 1 -- start start 2024-04-22T13:44:18.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 Processor -- start 2024-04-22T13:44:18.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 0x55b7e0702cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541ae3c0 con 0x55b7e06f1e80 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541af830 con 0x55b7e06faa00 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541aea90 con 0x55b7e06f14d0 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a5c784740 10 client.0 _ll_register_callbacks cb 0x7f1a4c003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46410/0 (socket says 172.21.15.87:46410) 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 -- 172.21.15.87:0/2578976838 learned_addr learned my addr 172.21.15.87:0/2578976838 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 0x55b7e0702cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a53fff640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 -- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 msgr2=0x55b7e0702cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 0x55b7e0702cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 -- 172.21.15.87:0/2578976838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 msgr2=0x55b7e06f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 -- 172.21.15.87:0/2578976838 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e06f45a0 con 0x55b7e06f1e80 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 0x55b7e0702cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a52ffd640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1a44004700 tx=0x7f1a44004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc101ea29c591237 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.430+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44012450 con 0x55b7e06f1e80 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f1a440125f0 con 0x55b7e06f1e80 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a4809c900 con 0x55b7e06f1e80 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1a440127f0 con 0x55b7e06f1e80 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 10 client.19371 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b7e0770c50 con 0x55b7e06f1e80 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4809c900 con 0x55b7e06f1e80 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1a4401a020 con 0x55b7e06f1e80 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 10 client.19371 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 10 client.19371 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 20 client.19371 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e0771400 0x55b7e0773820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 4 client.19371 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 --> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b7e0663c10 con 0x55b7e0771400 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.431+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e0771400 0x55b7e0773820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a32ffd640 10 client.19371 ms_handle_connect on v2:172.21.15.132:6836/453938062 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a537fe640 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e0771400 0x55b7e0773820 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6c2f8c2b9aacc17 server_cookie=5a9b7dbcc24b732c in_seq=0 out_seq=0 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a32ffd640 1 -- 172.21.15.87:0/2578976838 <== mds.2 v2:172.21.15.132:6836/453938062 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b7e0663c10 con 0x55b7e0771400 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a32ffd640 10 client.19371 handle_command_reply: tid=0 2024-04-22T13:44:18.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 client.19371 shutdown 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 msgr2=0x55b7e07031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1a44004700 tx=0x7f1a44004610 comp rx=0 tx=0).stop 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 shutdown_connections 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:6836/453938062,v1:172.21.15.132:6837/453938062] conn(0x55b7e0771400 0x55b7e0773820 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7e06faa00 0x55b7e06f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a32ffd640 0 client.19371 ms_handle_reset on v2:172.21.15.132:6836/453938062 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55b7e06f1e80 0x55b7e07031f0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 --2- 172.21.15.87:0/2578976838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7e06f14d0 0x55b7e0702cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 >> 172.21.15.87:0/2578976838 conn(0x55b7e06ebba0 msgr2=0x55b7e06ea370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 shutdown_connections 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 1 -- 172.21.15.87:0/2578976838 wait complete. 2024-04-22T13:44:18.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.432+0000 7f1a5c784740 20 client.19371 trim_cache size 0 max 16384 2024-04-22T13:44:18.439 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 msgr2=0x7f1a3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 0x7f1a3c0a61e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c02c790 tx=0x7f1a4c0095c0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 msgr2=0x7f1a541a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800d950 tx=0x7f1a4800de10 comp rx=0 tx=0).stop 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 shutdown_connections 2024-04-22T13:44:18.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f1a3c0a3d20 0x7f1a3c0a61e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5410e830 0x7f1a5419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1a54074b40 0x7f1a541a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 --2- 172.21.15.87:0/3481969762 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1a54074220 0x7f1a541a0680 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 >> 172.21.15.87:0/3481969762 conn(0x7f1a540793a0 msgr2=0x7f1a540fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 shutdown_connections 2024-04-22T13:44:18.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.433+0000 7f1a59aba640 1 -- 172.21.15.87:0/3481969762 wait complete. 2024-04-22T13:44:18.441 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T13:44:18.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.545+0000 7fc61ca4e640 1 -- 172.21.15.87:0/886868896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102210 msgr2=0x7fc6181025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.545+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/886868896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102210 0x7fc6181025f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fc608009900 tx=0x7fc60802bc30 comp rx=0 tx=0).stop 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 -- 172.21.15.87:0/886868896 shutdown_connections 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/886868896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc6181072e0 0x7fc61810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/886868896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc618102bc0 0x7fc618106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/886868896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618102210 0x7fc6181025f0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 -- 172.21.15.87:0/886868896 >> 172.21.15.87:0/886868896 conn(0x7fc6180fdab0 msgr2=0x7fc6180ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 -- 172.21.15.87:0/886868896 shutdown_connections 2024-04-22T13:44:18.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 -- 172.21.15.87:0/886868896 wait complete. 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 Processor -- start 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 -- start start 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.546+0000 7fc61ca4e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102210 0x7fc618074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6181072e0 0x7fc618074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc618112250 con 0x7fc618074cd0 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181120d0 con 0x7fc6181072e0 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181123d0 con 0x7fc618102210 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46428/0 (socket says 172.21.15.87:46428) 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 -- 172.21.15.87:0/1594992644 learned_addr learned my addr 172.21.15.87:0/1594992644 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61659c640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102210 0x7fc618074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc60ffff640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6181072e0 0x7fc618074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 -- 172.21.15.87:0/1594992644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102210 msgr2=0x7fc618074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102210 0x7fc618074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 -- 172.21.15.87:0/1594992644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6181072e0 msgr2=0x7fc618074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6181072e0 0x7fc618074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 -- 172.21.15.87:0/1594992644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6181a70e0 con 0x7fc618074cd0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc616d9d640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fc61000eaa0 tx=0x7fc61000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc60f7fe640 1 -- 172.21.15.87:0/1594992644 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61000cdb0 con 0x7fc618074cd0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc60f7fe640 1 -- 172.21.15.87:0/1594992644 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc610005020 con 0x7fc618074cd0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6181a73d0 con 0x7fc618074cd0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc60f7fe640 1 -- 172.21.15.87:0/1594992644 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100116d0 con 0x7fc618074cd0 2024-04-22T13:44:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.547+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a7920 con 0x7fc618074cd0 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.549+0000 7fc60f7fe640 1 -- 172.21.15.87:0/1594992644 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc6100051d0 con 0x7fc618074cd0 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 Processor -- start 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 -- start start 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d41430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d43e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4c9d1f0 con 0x55e8d4d41a00 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4c3a4b0 con 0x55e8d4d384d0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61f641740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4c991b0 con 0x55e8d4d38e80 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60f7fe640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d43e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d41430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d43e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46440/0 (socket says 172.21.15.87:46440) 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60ffff640 1 -- 172.21.15.87:0/3978514928 learned_addr learned my addr 172.21.15.87:0/3978514928 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60ffff640 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc616d9d640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc60f7fe640 1 -- 172.21.15.87:0/1594992644 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc6100d0900 con 0x7fc618074cd0 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61659c640 1 -- 172.21.15.87:0/3978514928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 msgr2=0x55e8d4d388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61659c640 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.550+0000 7fc61659c640 1 -- 172.21.15.87:0/3978514928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 msgr2=0x55e8d4d43e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc61659c640 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d43e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc61659c640 1 -- 172.21.15.87:0/3978514928 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8d4c2c1b0 con 0x55e8d4d38e80 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc61659c640 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d41430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc608000c00 tx=0x7fc608004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7729b7a54b0a88d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc616d9d640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 0x7fc6040a61e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc6100ce140 tx=0x7fc610087040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc60cff9640 1 -- 172.21.15.87:0/3978514928 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608045020 con 0x55e8d4d38e80 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc60cff9640 1 -- 172.21.15.87:0/3978514928 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc60802cc90 con 0x55e8d4d38e80 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.551+0000 7fc60cff9640 1 -- 172.21.15.87:0/3978514928 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6080046c0 con 0x55e8d4d38e80 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- 172.21.15.87:0/3978514928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 msgr2=0x55e8d4d41430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d41430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc608000c00 tx=0x7fc608004040 comp rx=0 tx=0).stop 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- 172.21.15.87:0/3978514928 shutdown_connections 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d43e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d41430 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- 172.21.15.87:0/3978514928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- 172.21.15.87:0/3978514928 >> 172.21.15.87:0/3978514928 conn(0x55e8d4d32ba0 msgr2=0x55e8d4d32f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- 172.21.15.87:0/3978514928 shutdown_connections 2024-04-22T13:44:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- 172.21.15.87:0/3978514928 wait complete. 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- start start 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 Processor -- start 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d384d0 0x55e8d4d41300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4d44910 con 0x55e8d4d41a00 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4c56500 con 0x55e8d4d38e80 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e8d4d44a90 con 0x55e8d4d384d0 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.552+0000 7fc61f641740 10 client.0 _ll_register_callbacks cb 0x55e8d4c24b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46444/0 (socket says 172.21.15.87:46444) 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 -- 172.21.15.87:0/3688369489 learned_addr learned my addr 172.21.15.87:0/3688369489 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d384d0 0x55e8d4d41300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc61659c640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 -- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d384d0 msgr2=0x55e8d4d41300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d384d0 0x55e8d4d41300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 -- 172.21.15.87:0/3688369489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 msgr2=0x55e8d4d3e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 -- 172.21.15.87:0/3688369489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8d49af3f0 con 0x55e8d4d41a00 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc60ffff640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fc600012ee0 tx=0x7fc600014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9595a624d2ea95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc5fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600012050 con 0x55e8d4d41a00 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc6000121f0 con 0x55e8d4d41a00 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc60802cc90 con 0x55e8d4d41a00 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc600011440 con 0x55e8d4d41a00 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc61f641740 10 client.19383 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e8d4a797d0 con 0x55e8d4d41a00 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.553+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc60802cc90 con 0x55e8d4d41a00 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc600020410 con 0x55e8d4d41a00 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 10 client.19383 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 10 client.19383 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 20 client.19383 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4d51bf0 0x55e8d4d54050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 4 client.19383 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8d4c3c930 con 0x55e8d4d51bf0 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61659c640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4d51bf0 0x55e8d4d54050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc61659c640 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4d51bf0 0x55e8d4d54050 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c1ccdabdd94e365 server_cookie=801c309ff509fb8e in_seq=0 out_seq=0 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.554+0000 7fc5fdffb640 10 client.19383 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc5fdffb640 1 -- 172.21.15.87:0/3688369489 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e8d4c3c930 con 0x55e8d4d51bf0 2024-04-22T13:44:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc5fdffb640 10 client.19383 handle_command_reply: tid=0 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 client.19383 shutdown 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 msgr2=0x55e8d4d3ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fc600012ee0 tx=0x7fc600014040 comp rx=0 tx=0).stop 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 shutdown_connections 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4d51bf0 0x55e8d4d54050 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3ec10 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc5fdffb640 0 client.19383 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 --2- 172.21.15.87:0/3688369489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d384d0 0x55e8d4d41300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 >> 172.21.15.87:0/3688369489 conn(0x55e8d4d32ba0 msgr2=0x55e8d4d348f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 shutdown_connections 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 1 -- 172.21.15.87:0/3688369489 wait complete. 2024-04-22T13:44:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.555+0000 7fc61f641740 20 client.19383 trim_cache size 0 max 16384 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 Processor -- start 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 -- start start 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d31a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d41a00 0x55e8d4d40dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181b2510 con 0x55e8d4d384d0 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181afde0 con 0x55e8d4d38e80 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.568+0000 7fc61f641740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181adc50 con 0x55e8d4d41a00 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46454/0 (socket says 172.21.15.87:46454) 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 -- 172.21.15.87:0/44001755 learned_addr learned my addr 172.21.15.87:0/44001755 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc61659c640 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d31a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 -- 172.21.15.87:0/44001755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d41a00 msgr2=0x55e8d4d40dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d41a00 0x55e8d4d40dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 -- 172.21.15.87:0/44001755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 msgr2=0x55e8d4d31a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d31a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 -- 172.21.15.87:0/44001755 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8d4d44850 con 0x55e8d4d384d0 2024-04-22T13:44:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc60ffff640 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc600012890 tx=0x7fc60000de30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cfe43f6fe84722 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc5fffff640 1 -- 172.21.15.87:0/44001755 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60002d070 con 0x55e8d4d384d0 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc5fffff640 1 -- 172.21.15.87:0/44001755 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6000125a0 con 0x55e8d4d384d0 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.569+0000 7fc5fffff640 1 -- 172.21.15.87:0/44001755 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000298d0 con 0x55e8d4d384d0 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- 172.21.15.87:0/44001755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 msgr2=0x55e8d4d4d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc600012890 tx=0x7fc60000de30 comp rx=0 tx=0).stop 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- 172.21.15.87:0/44001755 shutdown_connections 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d41a00 0x55e8d4d40dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d38e80 0x55e8d4d31a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- 172.21.15.87:0/44001755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d384d0 0x55e8d4d4d7d0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- 172.21.15.87:0/44001755 >> 172.21.15.87:0/44001755 conn(0x55e8d4d32ba0 msgr2=0x55e8d4d31370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- 172.21.15.87:0/44001755 shutdown_connections 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- 172.21.15.87:0/44001755 wait complete. 2024-04-22T13:44:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- start start 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 Processor -- start 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d49ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181ae2a0 con 0x55e8d4d41a00 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181af710 con 0x55e8d4d384d0 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181ae970 con 0x55e8d4d38e80 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc61f641740 10 client.0 _ll_register_callbacks cb 0x7fc600012af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc60ffff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46456/0 (socket says 172.21.15.87:46456) 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.570+0000 7fc60ffff640 1 -- 172.21.15.87:0/1471987104 learned_addr learned my addr 172.21.15.87:0/1471987104 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d49ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 -- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 msgr2=0x55e8d4d3af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 -- 172.21.15.87:0/1471987104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 msgr2=0x55e8d4d49ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d49ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 -- 172.21.15.87:0/1471987104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8d4d3b9c0 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc60ffff640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc600020780 tx=0x7fc600005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e36ff09e0873f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60002d040 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc60000d660 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6000125a0 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc600046050 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 10 client.19389 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e8d4db7e50 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6000125a0 con 0x55e8d4d41a00 2024-04-22T13:44:18.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc600031490 con 0x55e8d4d41a00 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 10 client.19389 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 10 client.19389 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 20 client.19389 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4db8460 0x55e8d4dba880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 4 client.19389 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.571+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 --> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8d4caac10 con 0x55e8d4db8460 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.572+0000 7fc61659c640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4db8460 0x55e8d4dba880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.572+0000 7fc61659c640 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4db8460 0x55e8d4dba880 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed6c57b412363bb4 server_cookie=cde4b7ec0e3e51e7 in_seq=0 out_seq=0 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.572+0000 7fc5fdffb640 10 client.19389 ms_handle_connect on v2:172.21.15.132:6838/2904643536 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc5fdffb640 1 -- 172.21.15.87:0/1471987104 <== mds.1 v2:172.21.15.132:6838/2904643536 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8d4caac10 con 0x55e8d4db8460 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc5fdffb640 10 client.19389 handle_command_reply: tid=0 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 client.19389 shutdown 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 msgr2=0x55e8d4d3b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc600020780 tx=0x7fc600005740 comp rx=0 tx=0).stop 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 shutdown_connections 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:6838/2904643536,v1:172.21.15.132:6839/2904643536] conn(0x55e8d4db8460 0x55e8d4dba880 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e8d4d41a00 0x55e8d4d3b480 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc5fdffb640 0 client.19389 ms_handle_reset on v2:172.21.15.132:6838/2904643536 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e8d4d38e80 0x55e8d4d3af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 --2- 172.21.15.87:0/1471987104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8d4d384d0 0x55e8d4d49ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 >> 172.21.15.87:0/1471987104 conn(0x55e8d4d32ba0 msgr2=0x55e8d4d39b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 shutdown_connections 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 1 -- 172.21.15.87:0/1471987104 wait complete. 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61f641740 20 client.19389 trim_cache size 0 max 16384 2024-04-22T13:44:18.577 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 msgr2=0x7fc6040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 0x7fc6040a61e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc6100ce140 tx=0x7fc610087040 comp rx=0 tx=0).stop 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 msgr2=0x7fc6181a6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fc61000eaa0 tx=0x7fc61000ef60 comp rx=0 tx=0).stop 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 shutdown_connections 2024-04-22T13:44:18.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc618074cd0 0x7fc6181a6a40 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6181072e0 0x7fc618074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 --2- 172.21.15.87:0/1594992644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc618102210 0x7fc618074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 >> 172.21.15.87:0/1594992644 conn(0x7fc6180fdab0 msgr2=0x7fc6180fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 shutdown_connections 2024-04-22T13:44:18.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.573+0000 7fc61ca4e640 1 -- 172.21.15.87:0/1594992644 wait complete. 2024-04-22T13:44:18.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.684+0000 7fc680ceb640 1 -- 172.21.15.87:0/2073407196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 msgr2=0x7fc67c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.684+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2073407196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c10dd30 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc674009900 tx=0x7fc67402bc30 comp rx=0 tx=0).stop 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- 172.21.15.87:0/2073407196 shutdown_connections 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2073407196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c10dd30 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2073407196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101a40 0x7fc67c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2073407196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c101090 0x7fc67c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- 172.21.15.87:0/2073407196 >> 172.21.15.87:0/2073407196 conn(0x7fc67c077f50 msgr2=0x7fc67c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- 172.21.15.87:0/2073407196 shutdown_connections 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- 172.21.15.87:0/2073407196 wait complete. 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 Processor -- start 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- start start 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101090 0x7fc67c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c19be50 0x7fc67c19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1110b0 con 0x7fc67c106160 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c110f30 con 0x7fc67c19be50 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.685+0000 7fc680ceb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c111230 con 0x7fc67c101090 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46474/0 (socket says 172.21.15.87:46474) 2024-04-22T13:44:18.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 -- 172.21.15.87:0/2110960258 learned_addr learned my addr 172.21.15.87:0/2110960258 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101090 0x7fc67c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 -- 172.21.15.87:0/2110960258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101090 msgr2=0x7fc67c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c19be50 0x7fc67c19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101090 0x7fc67c197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 -- 172.21.15.87:0/2110960258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c19be50 msgr2=0x7fc67c19c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c19be50 0x7fc67c19c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 -- 172.21.15.87:0/2110960258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67c19c940 con 0x7fc67c106160 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c970 tx=0x7fc67000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc66b7fe640 1 -- 172.21.15.87:0/2110960258 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670014070 con 0x7fc67c106160 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc66b7fe640 1 -- 172.21.15.87:0/2110960258 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc670004440 con 0x7fc67c106160 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67c19cc30 con 0x7fc67c106160 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc66b7fe640 1 -- 172.21.15.87:0/2110960258 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670003e50 con 0x7fc67c106160 2024-04-22T13:44:18.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.686+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67c1a4420 con 0x7fc67c106160 2024-04-22T13:44:18.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc66b7fe640 1 -- 172.21.15.87:0/2110960258 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc670017070 con 0x7fc67c106160 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc6839b8740 1 Processor -- start 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc6839b8740 1 -- start start 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.689+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f18a5bf0 0x5645f18a6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc6839b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5645f18031f0 con 0x5645f189e340 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc6839b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5645f17a04b0 con 0x5645f18a5bf0 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc6839b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5645f17ff1b0 con 0x5645f189ecf0 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46478/0 (socket says 172.21.15.87:46478) 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 -- 172.21.15.87:0/1782105700 learned_addr learned my addr 172.21.15.87:0/1782105700 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc679d9b640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f18a5bf0 0x5645f18a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67a59c640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 -- 172.21.15.87:0/1782105700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 msgr2=0x5645f18a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 -- 172.21.15.87:0/1782105700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f18a5bf0 msgr2=0x5645f18a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f18a5bf0 0x5645f18a6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 -- 172.21.15.87:0/1782105700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f17921b0 con 0x5645f189e340 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67a59c640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc66b7fe640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 0x7fc6640a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 0x7fc6640a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc674005ab0 tx=0x7fc674005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff219327ee05d966 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc66b7fe640 1 -- 172.21.15.87:0/2110960258 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc6700d0680 con 0x7fc67c106160 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc668ff9640 1 -- 172.21.15.87:0/1782105700 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674043020 con 0x5645f189e340 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc668ff9640 1 -- 172.21.15.87:0/1782105700 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc67402c7e0 con 0x5645f189e340 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.690+0000 7fc668ff9640 1 -- 172.21.15.87:0/1782105700 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674005490 con 0x5645f189e340 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- 172.21.15.87:0/1782105700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 msgr2=0x5645f189e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 0x7fc6640a6380 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5645f189d7a0 tx=0x7fc66c009420 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc674005ab0 tx=0x7fc674005e80 comp rx=0 tx=0).stop 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- 172.21.15.87:0/1782105700 shutdown_connections 2024-04-22T13:44:18.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f18a5bf0 0x5645f18a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1782105700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f189e720 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- 172.21.15.87:0/1782105700 >> 172.21.15.87:0/1782105700 conn(0x5645f1898ba0 msgr2=0x5645f1898f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- 172.21.15.87:0/1782105700 shutdown_connections 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- 172.21.15.87:0/1782105700 wait complete. 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.691+0000 7fc6839b8740 1 -- start start 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 Processor -- start 2024-04-22T13:44:18.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 0x5645f18ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x5645f18acb30 con 0x5645f18a5bf0 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5645f17bc500 con 0x5645f189e340 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5645f18accb0 con 0x5645f189ecf0 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 10 client.0 _ll_register_callbacks cb 0x5645f178ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46494/0 (socket says 172.21.15.87:46494) 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 -- 172.21.15.87:0/215764767 learned_addr learned my addr 172.21.15.87:0/215764767 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67a59c640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 0x5645f18ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc679d9b640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 -- 172.21.15.87:0/215764767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 msgr2=0x5645f18a9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 -- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 msgr2=0x5645f18ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 0x5645f18ac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 -- 172.21.15.87:0/215764767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645f15153f0 con 0x5645f18a5bf0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67a59c640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 0x5645f18ac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fc674009900 tx=0x7fc6740056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7dbc6287c03315 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67404c070 con 0x5645f18a5bf0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc6740035f0 con 0x5645f18a5bf0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.692+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc67402c7e0 con 0x5645f18a5bf0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc674058050 con 0x5645f18a5bf0 2024-04-22T13:44:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 10 client.19401 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5645f15df7d0 con 0x5645f18a5bf0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc67402c7e0 con 0x5645f18a5bf0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc674004ae0 con 0x5645f18a5bf0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 10 client.19401 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 10 client.19401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 20 client.19401 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f18b9be0 0x5645f18bbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 4 client.19401 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645f17a2930 con 0x5645f18b9be0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.693+0000 7fc67a59c640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f18b9be0 0x5645f18bbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.694+0000 7fc661ffb640 10 client.19401 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.694+0000 7fc67a59c640 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f18b9be0 0x5645f18bbfd0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a5f8c81a48ecc1 server_cookie=b6f2110fc0b6e779 in_seq=0 out_seq=0 2024-04-22T13:44:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.694+0000 7fc661ffb640 1 -- 172.21.15.87:0/215764767 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645f17a2930 con 0x5645f18b9be0 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.694+0000 7fc661ffb640 10 client.19401 handle_command_reply: tid=0 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.694+0000 7fc6839b8740 1 client.19401 shutdown 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 msgr2=0x5645f18a98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fc674009900 tx=0x7fc6740056a0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 shutdown_connections 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f18b9be0 0x5645f18bbfd0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f18a5bf0 0x5645f18a98d0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f189ecf0 0x5645f18a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 --2- 172.21.15.87:0/215764767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189e340 0x5645f18ac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc661ffb640 0 client.19401 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 >> 172.21.15.87:0/215764767 conn(0x5645f1898ba0 msgr2=0x5645f189a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 shutdown_connections 2024-04-22T13:44:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 1 -- 172.21.15.87:0/215764767 wait complete. 2024-04-22T13:44:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.695+0000 7fc6839b8740 20 client.19401 trim_cache size 0 max 16384 2024-04-22T13:44:18.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 Processor -- start 2024-04-22T13:44:18.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 -- start start 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ae0e0 con 0x5645f189e340 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ab9b0 con 0x5645f189ecf0 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc6839b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1a9820 con 0x5645f18a5bf0 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46500/0 (socket says 172.21.15.87:46500) 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 -- 172.21.15.87:0/1522344214 learned_addr learned my addr 172.21.15.87:0/1522344214 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 -- 172.21.15.87:0/1522344214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 msgr2=0x5645f18a8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18a8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 -- 172.21.15.87:0/1522344214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 msgr2=0x5645f18b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc679d9b640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 -- 172.21.15.87:0/1522344214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f18a9eb0 con 0x5645f189e340 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc679d9b640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.708+0000 7fc67a59c640 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fc66c005fd0 tx=0x7fc66c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b1e745fcc81c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc663fff640 1 -- 172.21.15.87:0/1522344214 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02b030 con 0x5645f189e340 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc663fff640 1 -- 172.21.15.87:0/1522344214 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c0273f0 con 0x5645f189e340 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc663fff640 1 -- 172.21.15.87:0/1522344214 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c005620 con 0x5645f189e340 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 -- 172.21.15.87:0/1522344214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 msgr2=0x5645f18a6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fc66c005fd0 tx=0x7fc66c013000 comp rx=0 tx=0).stop 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 -- 172.21.15.87:0/1522344214 shutdown_connections 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18a8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 --2- 172.21.15.87:0/1522344214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18a6fe0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 -- 172.21.15.87:0/1522344214 >> 172.21.15.87:0/1522344214 conn(0x5645f1898ba0 msgr2=0x5645f1897370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 -- 172.21.15.87:0/1522344214 shutdown_connections 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.709+0000 7fc6839b8740 1 -- 172.21.15.87:0/1522344214 wait complete. 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 -- start start 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 Processor -- start 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1a9e70 con 0x5645f189e340 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1ab2e0 con 0x5645f189ecf0 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1aa540 con 0x5645f18a5bf0 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc6839b8740 10 client.0 _ll_register_callbacks cb 0x7fc674005230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46512/0 (socket says 172.21.15.87:46512) 2024-04-22T13:44:18.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 -- 172.21.15.87:0/2051937330 learned_addr learned my addr 172.21.15.87:0/2051937330 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67ad9d640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 -- 172.21.15.87:0/2051937330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 msgr2=0x5645f18aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 -- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 msgr2=0x5645f18b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 -- 172.21.15.87:0/2051937330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645f1892fe0 con 0x5645f189e340 2024-04-22T13:44:18.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc67a59c640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0275c0 tx=0x7fc66c0275f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d9cdfe07624040 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.710+0000 7fc661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.87:3300/0 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02b050 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mon.0 v2:172.21.15.87:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc66c004c40 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66c0273f0 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc66c0052a0 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 10 client.19407 fetch_fsmap learned FSMap version 23 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5645f191f7c0 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mon.0 v2:172.21.15.87:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c0273f0 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mon.0 v2:172.21.15.87:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc66c030980 con 0x5645f189e340 2024-04-22T13:44:18.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 10 client.19407 fetch_fsmap finished waiting for FSMap version 23 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 10 client.19407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 20 client.19407 populate_metadata read hostname 'smithi087' 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f191fe50 0x5645f1922270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 4 client.19407 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 --> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645f1810c10 con 0x5645f191fe50 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.711+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f191fe50 0x5645f1922270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc661ffb640 10 client.19407 ms_handle_connect on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc679d9b640 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f191fe50 0x5645f1922270 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b0dcefe22ead0f1 server_cookie=f631fbdc3e5871e9 in_seq=0 out_seq=0 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc661ffb640 1 -- 172.21.15.87:0/2051937330 <== mds.0 v2:172.21.15.120:6838/2692231328 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645f1810c10 con 0x5645f191fe50 2024-04-22T13:44:18.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc661ffb640 10 client.19407 handle_command_reply: tid=0 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 client.19407 shutdown 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 msgr2=0x5645f18b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0275c0 tx=0x7fc66c0275f0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 shutdown_connections 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:6838/2692231328,v1:172.21.15.120:6839/2692231328] conn(0x5645f191fe50 0x5645f1922270 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645f18a5bf0 0x5645f18aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5645f189ecf0 0x5645f18b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 --2- 172.21.15.87:0/2051937330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x5645f189e340 0x5645f18b1f70 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc661ffb640 0 client.19407 ms_handle_reset on v2:172.21.15.120:6838/2692231328 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 >> 172.21.15.87:0/2051937330 conn(0x5645f1898ba0 msgr2=0x5645f1897c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 shutdown_connections 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 1 -- 172.21.15.87:0/2051937330 wait complete. 2024-04-22T13:44:18.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc6839b8740 20 client.19407 trim_cache size 0 max 16384 2024-04-22T13:44:18.718 INFO:teuthology.orchestra.run.smithi087.stdout:[] 2024-04-22T13:44:18.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 msgr2=0x7fc6640a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 0x7fc6640a6380 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5645f189d7a0 tx=0x7fc66c009420 comp rx=0 tx=0).stop 2024-04-22T13:44:18.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 msgr2=0x7fc67c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c970 tx=0x7fc67000ce30 comp rx=0 tx=0).stop 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.712+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 shutdown_connections 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc6640a3ec0 0x7fc6640a6380 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67c19be50 0x7fc67c19c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc67c106160 0x7fc67c197ad0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 --2- 172.21.15.87:0/2110960258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c101090 0x7fc67c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 >> 172.21.15.87:0/2110960258 conn(0x7fc67c077f50 msgr2=0x7fc67c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 shutdown_connections 2024-04-22T13:44:18.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.713+0000 7fc680ceb640 1 -- 172.21.15.87:0/2110960258 wait complete. 2024-04-22T13:44:18.720 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_d69f13d827952bb07bb9b7973bca14f97d764d6f/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-04-22T13:44:18.721 INFO:tasks.fwd_scrub:done joining 2024-04-22T13:44:18.721 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T13:44:18.734 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:44:18.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.836+0000 7f3c0cc31640 1 -- 172.21.15.87:0/2132307019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08069aa0 msgr2=0x7f3c0810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.836+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/2132307019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08069aa0 0x7f3c0810ede0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009ca0 tx=0x7f3c0002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 -- 172.21.15.87:0/2132307019 shutdown_connections 2024-04-22T13:44:18.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/2132307019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08069aa0 0x7f3c0810ede0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/2132307019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 0x7f3c08069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/2132307019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08106f50 0x7f3c08107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 -- 172.21.15.87:0/2132307019 >> 172.21.15.87:0/2132307019 conn(0x7f3c08077e30 msgr2=0x7f3c08078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 -- 172.21.15.87:0/2132307019 shutdown_connections 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 -- 172.21.15.87:0/2132307019 wait complete. 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 Processor -- start 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 -- start start 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.837+0000 7f3c0cc31640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 0x7f3c08198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08069aa0 0x7f3c08198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3c08112150 con 0x7f3c08106f50 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c08111fd0 con 0x7f3c08069aa0 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081122d0 con 0x7f3c080690e0 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46530/0 (socket says 172.21.15.87:46530) 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 -- 172.21.15.87:0/3553313919 learned_addr learned my addr 172.21.15.87:0/3553313919 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0659c640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 0x7f3c08198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c05d9b640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08069aa0 0x7f3c08198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 -- 172.21.15.87:0/3553313919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 msgr2=0x7f3c08198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 0x7f3c08198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 -- 172.21.15.87:0/3553313919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08069aa0 msgr2=0x7f3c08198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08069aa0 0x7f3c08198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 -- 172.21.15.87:0/3553313919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0819d5d0 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c06d9d640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f3c0002c700 tx=0x7f3c000346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00041020 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c00032960 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0819d860 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00039040 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.838+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c081a5150 con 0x7f3c08106f50 2024-04-22T13:44:18.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.839+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c08068de0 con 0x7f3c08106f50 2024-04-22T13:44:18.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.840+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3c00044020 con 0x7f3c08106f50 2024-04-22T13:44:18.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.841+0000 7f3bf77fe640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 0x7f3bf00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:18.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.841+0000 7f3c0659c640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 0x7f3bf00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:18.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.841+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f3c000f2880 con 0x7f3c08106f50 2024-04-22T13:44:18.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.841+0000 7f3c0659c640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 0x7f3bf00a61e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800bbe0 tx=0x7f3bf800b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:18.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:18.844+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c0004b020 con 0x7f3c08106f50 2024-04-22T13:44:18.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:18 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/486181695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:18.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:18 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/486181695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:18.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:18 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/486181695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:19.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.024+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c0810a4a0 con 0x7f3c08106f50 2024-04-22T13:44:19.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.026+0000 7f3bf77fe640 1 -- 172.21.15.87:0/3553313919 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23047 (secure 0 0 0) 0x7f3c000bf190 con 0x7f3c08106f50 2024-04-22T13:44:19.022 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:19.023 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":103,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:44:14.582679+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:44:19.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 msgr2=0x7f3bf00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 0x7f3bf00a61e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800bbe0 tx=0x7f3bf800b390 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 msgr2=0x7f3c0819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f3c0002c700 tx=0x7f3c000346d0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 shutdown_connections 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3bf00a3d20 0x7f3bf00a61e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c08106f50 0x7f3c0819cea0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c08069aa0 0x7f3c08198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 --2- 172.21.15.87:0/3553313919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c080690e0 0x7f3c08198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 >> 172.21.15.87:0/3553313919 conn(0x7f3c08077e30 msgr2=0x7f3c080fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 shutdown_connections 2024-04-22T13:44:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.028+0000 7f3c0cc31640 1 -- 172.21.15.87:0/3553313919 wait complete. 2024-04-22T13:44:19.036 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:44:19.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:44:19.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 -- 172.21.15.87:0/3972445281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0107030 msgr2=0x7fcbf0107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 --2- 172.21.15.87:0/3972445281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0107030 0x7fcbf0107410 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009930 tx=0x7fcbec02bc30 comp rx=0 tx=0).stop 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 -- 172.21.15.87:0/3972445281 shutdown_connections 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 --2- 172.21.15.87:0/3972445281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0101b40 0x7fcbf010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 --2- 172.21.15.87:0/3972445281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 0x7fcbf0101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 --2- 172.21.15.87:0/3972445281 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0107030 0x7fcbf0107410 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 -- 172.21.15.87:0/3972445281 >> 172.21.15.87:0/3972445281 conn(0x7fcbf0076c50 msgr2=0x7fcbf0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 -- 172.21.15.87:0/3972445281 shutdown_connections 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.141+0000 7fcbf8352640 1 -- 172.21.15.87:0/3972445281 wait complete. 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 Processor -- start 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 -- start start 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 0x7fcbf010e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 0x7fcbf010f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf0112220 con 0x7fcbf0101b40 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf01120a0 con 0x7fcbf01011a0 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf8352640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf01123a0 con 0x7fcbf0107030 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46550/0 (socket says 172.21.15.87:46550) 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 -- 172.21.15.87:0/2925858748 learned_addr learned my addr 172.21.15.87:0/2925858748 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf68ef640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 0x7fcbf010f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf60ee640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 0x7fcbf010e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 -- 172.21.15.87:0/2925858748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 msgr2=0x7fcbf010f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 0x7fcbf010f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 -- 172.21.15.87:0/2925858748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 msgr2=0x7fcbf010e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 0x7fcbf010e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 -- 172.21.15.87:0/2925858748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf01a6ca0 con 0x7fcbf0101b40 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf68ef640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 0x7fcbf010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:19.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbf58ed640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000d950 tx=0x7fcbe000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.142+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0017030 con 0x7fcbf0101b40 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.143+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe0004020 con 0x7fcbf0101b40 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.143+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf01a6f90 con 0x7fcbf0101b40 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.143+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00047a0 con 0x7fcbf0101b40 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.143+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf01a7450 con 0x7fcbf0101b40 2024-04-22T13:44:19.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.143+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf0102590 con 0x7fcbf0101b40 2024-04-22T13:44:19.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.145+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcbe0009a30 con 0x7fcbf0101b40 2024-04-22T13:44:19.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.146+0000 7fcbe77fe640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 0x7fcbdc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.146+0000 7fcbf60ee640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 0x7fcbdc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.146+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fcbe00d0920 con 0x7fcbf0101b40 2024-04-22T13:44:19.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.146+0000 7fcbf60ee640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 0x7fcbdc0a61e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcbec036040 tx=0x7fcbec000bc0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.149+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbe009d000 con 0x7fcbf0101b40 2024-04-22T13:44:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.329+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcbf0108d90 con 0x7fcbf0101b40 2024-04-22T13:44:19.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.330+0000 7fcbe77fe640 1 -- 172.21.15.87:0/2925858748 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7fcbe009c900 con 0x7fcbf0101b40 2024-04-22T13:44:19.326 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:44:19.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 msgr2=0x7fcbdc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 0x7fcbdc0a61e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcbec036040 tx=0x7fcbec000bc0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 msgr2=0x7fcbf010ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000d950 tx=0x7fcbe000de10 comp rx=0 tx=0).stop 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 shutdown_connections 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fcbdc0a3d20 0x7fcbdc0a61e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbf0107030 0x7fcbf010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbf0101b40 0x7fcbf010ece0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 --2- 172.21.15.87:0/2925858748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf01011a0 0x7fcbf010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 >> 172.21.15.87:0/2925858748 conn(0x7fcbf0076c50 msgr2=0x7fcbf00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 shutdown_connections 2024-04-22T13:44:19.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.332+0000 7fcbf8352640 1 -- 172.21.15.87:0/2925858748 wait complete. 2024-04-22T13:44:19.339 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:44:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 -- 172.21.15.87:0/1075393300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81072b0 msgr2=0x7fe9c810eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 --2- 172.21.15.87:0/1075393300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81072b0 0x7fe9c810eec0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009ca0 tx=0x7fe9c002c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 -- 172.21.15.87:0/1075393300 shutdown_connections 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 --2- 172.21.15.87:0/1075393300 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81072b0 0x7fe9c810eec0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 --2- 172.21.15.87:0/1075393300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81014a0 0x7fe9c8106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 --2- 172.21.15.87:0/1075393300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 0x7fe9c8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 -- 172.21.15.87:0/1075393300 >> 172.21.15.87:0/1075393300 conn(0x7fe9c80760e0 msgr2=0x7fe9c80764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 -- 172.21.15.87:0/1075393300 shutdown_connections 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.445+0000 7fe9cd967640 1 -- 172.21.15.87:0/1075393300 wait complete. 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 Processor -- start 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 -- start start 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 0x7fe9c8101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81072b0 0x7fe9c8101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c8112230 con 0x7fe9c81024e0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c81120b0 con 0x7fe9c8100af0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9cd967640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c81123b0 con 0x7fe9c81072b0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46570/0 (socket says 172.21.15.87:46570) 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 -- 172.21.15.87:0/905528107 learned_addr learned my addr 172.21.15.87:0/905528107 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9bffff640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81072b0 0x7fe9c8101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c6ffd640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 0x7fe9c8101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 -- 172.21.15.87:0/905528107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81072b0 msgr2=0x7fe9c8101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81072b0 0x7fe9c8101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 -- 172.21.15.87:0/905528107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 msgr2=0x7fe9c8101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 0x7fe9c8101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.446+0000 7fe9c77fe640 1 -- 172.21.15.87:0/905528107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c81a7060 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9c77fe640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0005e30 tx=0x7fe9c0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0039040 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9c0044070 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c81a72f0 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0003e60 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.447+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c81a77e0 con 0x7fe9c81024e0 2024-04-22T13:44:19.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.448+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c80691c0 con 0x7fe9c81024e0 2024-04-22T13:44:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.449+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe9c00324c0 con 0x7fe9c81024e0 2024-04-22T13:44:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.449+0000 7fe9cc965640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 0x7fe9b40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.450+0000 7fe9c6ffd640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 0x7fe9b40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.450+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fe9c00f3360 con 0x7fe9c81024e0 2024-04-22T13:44:19.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.450+0000 7fe9c6ffd640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 0x7fe9b40a6210 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800bbe0 tx=0x7fe9b800b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.453+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9c00bfb50 con 0x7fe9c81024e0 2024-04-22T13:44:19.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.631+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9c810a580 con 0x7fe9c81024e0 2024-04-22T13:44:19.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.631+0000 7fe9cc965640 1 -- 172.21.15.87:0/905528107 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7fe9c00bf450 con 0x7fe9c81024e0 2024-04-22T13:44:19.628 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 msgr2=0x7fe9b40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 0x7fe9b40a6210 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800bbe0 tx=0x7fe9b800b390 comp rx=0 tx=0).stop 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 msgr2=0x7fe9c81a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0005e30 tx=0x7fe9c0034760 comp rx=0 tx=0).stop 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 shutdown_connections 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe9b40a3d50 0x7fe9b40a6210 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe9c81024e0 0x7fe9c81a6960 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9c81072b0 0x7fe9c8101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 --2- 172.21.15.87:0/905528107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c8100af0 0x7fe9c8101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 >> 172.21.15.87:0/905528107 conn(0x7fe9c80760e0 msgr2=0x7fe9c80fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 shutdown_connections 2024-04-22T13:44:19.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.633+0000 7fe9cd967640 1 -- 172.21.15.87:0/905528107 wait complete. 2024-04-22T13:44:19.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:44:19.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- 172.21.15.87:0/421527596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3258103af0 msgr2=0x7f325810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 --2- 172.21.15.87:0/421527596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3258103af0 0x7f325810bfd0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f324c009900 tx=0x7f324c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:44:19.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- 172.21.15.87:0/421527596 shutdown_connections 2024-04-22T13:44:19.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 --2- 172.21.15.87:0/421527596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f325810c510 0x7f325810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 --2- 172.21.15.87:0/421527596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3258103af0 0x7f325810bfd0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 --2- 172.21.15.87:0/421527596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 0x7f32581035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- 172.21.15.87:0/421527596 >> 172.21.15.87:0/421527596 conn(0x7f3258077f50 msgr2=0x7f3258078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- 172.21.15.87:0/421527596 shutdown_connections 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- 172.21.15.87:0/421527596 wait complete. 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 Processor -- start 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.747+0000 7f325e359640 1 -- start start 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 0x7f3258198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 0x7f3258198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f32581121a0 con 0x7f325810c510 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3258112020 con 0x7f32581031d0 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3258112320 con 0x7f3258103af0 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33996/0 (socket says 172.21.15.87:33996) 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 -- 172.21.15.87:0/2650150479 learned_addr learned my addr 172.21.15.87:0/2650150479 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f32577fe640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 0x7f3258198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f3257fff640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 0x7f3258198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 -- 172.21.15.87:0/2650150479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 msgr2=0x7f3258198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 0x7f3258198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 -- 172.21.15.87:0/2650150479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 msgr2=0x7f3258198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 0x7f3258198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 -- 172.21.15.87:0/2650150479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325819d710 con 0x7f325810c510 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f32577fe640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 0x7f3258198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325c8f6640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f325000eaa0 tx=0x7f325000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325000cdb0 con 0x7f325810c510 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3250005020 con 0x7f325810c510 2024-04-22T13:44:19.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500116d0 con 0x7f325810c510 2024-04-22T13:44:19.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325819da00 con 0x7f325810c510 2024-04-22T13:44:19.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.748+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32581a5290 con 0x7f325810c510 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.749+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3258068d70 con 0x7f325810c510 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.751+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f32500051d0 con 0x7f325810c510 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.751+0000 7f32557fa640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 0x7f32400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.751+0000 7f3257fff640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 0x7f32400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.752+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f32500d09d0 con 0x7f325810c510 2024-04-22T13:44:19.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.752+0000 7f3257fff640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 0x7f32400a61e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3248009b60 tx=0x7f3248005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:19.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.756+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3250016030 con 0x7f325810c510 2024-04-22T13:44:19.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:19 smithi120 ceph-mon[31101]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:19.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:19 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3553313919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:19.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:19 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2925858748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:19.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.935+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f325802d5c0 con 0x7f325810c510 2024-04-22T13:44:19.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.935+0000 7f32557fa640 1 -- 172.21.15.87:0/2650150479 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f325009d0b0 con 0x7f325810c510 2024-04-22T13:44:19.931 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 msgr2=0x7f32400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 0x7f32400a61e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3248009b60 tx=0x7f3248005f90 comp rx=0 tx=0).stop 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 msgr2=0x7f325819cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f325000eaa0 tx=0x7f325000ef60 comp rx=0 tx=0).stop 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 shutdown_connections 2024-04-22T13:44:19.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f32400a3d20 0x7f32400a61e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f325810c510 0x7f325819cfe0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3258103af0 0x7f3258198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 --2- 172.21.15.87:0/2650150479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32581031d0 0x7f3258198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 >> 172.21.15.87:0/2650150479 conn(0x7f3258077f50 msgr2=0x7f32580fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 shutdown_connections 2024-04-22T13:44:19.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:19.937+0000 7f325e359640 1 -- 172.21.15.87:0/2650150479 wait complete. 2024-04-22T13:44:19.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:19 smithi087 ceph-mon[27999]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:19.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:19 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3553313919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:19.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:19 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2925858748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:19.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T13:44:19.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:19 smithi132 ceph-mon[31531]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:19.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:19 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3553313919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:19.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:19 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2925858748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 -- 172.21.15.87:0/3810384110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc103b00 msgr2=0x7fe1bc10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 --2- 172.21.15.87:0/3810384110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc103b00 0x7fe1bc10bfe0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009900 tx=0x7fe1b002bd60 comp rx=0 tx=0).stop 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 -- 172.21.15.87:0/3810384110 shutdown_connections 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 --2- 172.21.15.87:0/3810384110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc10c520 0x7fe1bc10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 --2- 172.21.15.87:0/3810384110 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc103b00 0x7fe1bc10bfe0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 --2- 172.21.15.87:0/3810384110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 0x7fe1bc1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 -- 172.21.15.87:0/3810384110 >> 172.21.15.87:0/3810384110 conn(0x7fe1bc077f50 msgr2=0x7fe1bc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 -- 172.21.15.87:0/3810384110 shutdown_connections 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.046+0000 7fe1c1431640 1 -- 172.21.15.87:0/3810384110 wait complete. 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 Processor -- start 2024-04-22T13:44:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 -- start start 2024-04-22T13:44:20.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 0x7fe1bc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:20.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc103b00 0x7fe1bc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:20.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc112190 con 0x7fe1bc10c520 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc112010 con 0x7fe1bc103b00 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1c1431640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc112310 con 0x7fe1bc1031e0 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1bb7fe640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34008/0 (socket says 172.21.15.87:34008) 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1bb7fe640 1 -- 172.21.15.87:0/120519075 learned_addr learned my addr 172.21.15.87:0/120519075 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.047+0000 7fe1ba7fc640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc103b00 0x7fe1bc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1baffd640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 0x7fe1bc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 -- 172.21.15.87:0/120519075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 msgr2=0x7fe1bc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 0x7fe1bc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 -- 172.21.15.87:0/120519075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc103b00 msgr2=0x7fe1bc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc103b00 0x7fe1bc198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 -- 172.21.15.87:0/120519075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc19d700 con 0x7fe1bc10c520 2024-04-22T13:44:20.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1bb7fe640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400d750 tx=0x7fe1b400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4017070 con 0x7fe1bc10c520 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b40054d0 con 0x7fe1bc10c520 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc19d9f0 con 0x7fe1bc10c520 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4004c60 con 0x7fe1bc10c520 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.048+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc1a5280 con 0x7fe1bc10c520 2024-04-22T13:44:20.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.049+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1bc068d70 con 0x7fe1bc10c520 2024-04-22T13:44:20.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.050+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe1b4009bd0 con 0x7fe1bc10c520 2024-04-22T13:44:20.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.051+0000 7fe1a3fff640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 0x7fe1a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:20.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.051+0000 7fe1baffd640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 0x7fe1a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:20.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.051+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fe1b40d00f0 con 0x7fe1bc10c520 2024-04-22T13:44:20.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.051+0000 7fe1baffd640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 0x7fe1a40a61e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac005c40 tx=0x7fe1ac00a780 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:20.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.055+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1b409c7d0 con 0x7fe1bc10c520 2024-04-22T13:44:20.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.280+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fe1bc1076a0 con 0x7fe1bc10c520 2024-04-22T13:44:20.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.288+0000 7fe1a3fff640 1 -- 172.21.15.87:0/120519075 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746061 (secure 0 0 0) 0x7fe1b409c0d0 con 0x7fe1bc10c520 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 26, 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "active_gid": 15303, 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "active_name": "x", 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "active_addrs": { 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:20.295 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6800", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2766661417 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v1", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2766661417 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "active_addr": "172.21.15.87:6801/2766661417", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "active_change": "2024-04-22T13:31:52.016639+0000", 2024-04-22T13:44:20.296 INFO:teuthology.orchestra.run.smithi087.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "available": true, 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "standbys": [ 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "gid": 24818, 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "y", 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_features": 4540701547738038271, 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "available_modules": [ 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "alerts", 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.297 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.298 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.299 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.300 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.301 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.302 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_destination": { 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_destination", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.303 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email address to send alerts to", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_from_name": { 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_from_name", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.304 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "Ceph", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email From: name", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_host": { 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_host", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.305 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP server", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_password": { 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_password", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.306 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Password to authenticate with", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_port": { 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_port", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "465", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.307 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP port", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_sender": { 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_sender", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.308 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP envelope sender", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_ssl": { 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_ssl", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.309 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_user": { 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_user", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.310 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "User to authenticate as", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "balancer", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.311 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "active": { 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "active", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_time": { 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_time", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0000", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.313 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_weekday": { 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_weekday", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.314 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_max_iterations": { 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_max_iterations", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "25", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "250", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_metrics": { 2024-04-22T13:44:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_metrics", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_step": { 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_step", 2024-04-22T13:44:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.5", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0.001", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "0.999", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "aggressiveness of optimization", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "end_time": { 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_time", 2024-04-22T13:44:20.317 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2359", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "end_weekday": { 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_weekday", 2024-04-22T13:44:20.318 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.319 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.320 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.321 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.322 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.323 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "min_score": { 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_score", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.324 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "upmap", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.325 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-compat", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "none", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "read", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap-read" 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Balancer mode", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_ids": { 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_ids", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.326 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.327 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_deviation": { 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_deviation", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.328 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_optimizations": { 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_optimizations", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.329 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm", 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.330 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_down_multiplier": { 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_down_multiplier", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.331 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_refresh_rate": { 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_refresh_rate", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "20", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_starting_port": { 2024-04-22T13:44:20.332 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_starting_port", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4721", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_ptrace": { 2024-04-22T13:44:20.333 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_ptrace", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_interval": { 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_interval", 2024-04-22T13:44:20.334 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_memory_target_ratio": { 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T13:44:20.335 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.7", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm_log_destination": { 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm_log_destination", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.336 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "file", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "file,syslog", 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: "syslog" 2024-04-22T13:44:20.337 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "cgroups_split": { 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cgroups_split", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.338 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "config_checks_enabled": { 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_checks_enabled", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.339 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "config_dashboard": { 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_dashboard", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T13:44:20.340 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_alertmanager": { 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_alertmanager", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.341 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_base": { 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_base", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Container image name, without the tag", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.342 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_elasticsearch": { 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_elasticsearch", 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.343 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "elasticsearch container image", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_grafana": { 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_grafana", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.344 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_haproxy": { 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_haproxy", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "HAproxy container image", 2024-04-22T13:44:20.345 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_agent": { 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_agent", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger agent container image", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_collector": { 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_collector", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger collector container image", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.347 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_query": { 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_query", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger query container image", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_keepalived": { 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_keepalived", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Keepalived container image", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_loki": { 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_loki", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Loki container image", 2024-04-22T13:44:20.350 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_node_exporter": { 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_node_exporter", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.351 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_nvmeof": { 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_nvmeof", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Nvme-of container image", 2024-04-22T13:44:20.352 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_prometheus": { 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_prometheus", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.353 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_promtail": { 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_promtail", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Promtail container image", 2024-04-22T13:44:20.354 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_samba": { 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_samba", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Samba/SMB container image", 2024-04-22T13:44:20.355 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_snmp_gateway": { 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_snmp_gateway", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SNMP Gateway container image", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.356 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "container_init": { 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_init", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T13:44:20.357 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "daemon_cache_timeout": { 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "daemon_cache_timeout", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T13:44:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "default_cephadm_command_timeout": { 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "900", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.359 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "default_registry": { 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_registry", 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io", 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.360 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "device_cache_timeout": { 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_cache_timeout", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1800", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.361 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache device inventory", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "device_enhanced_scan": { 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_enhanced_scan", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.362 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "facts_cache_timeout": { 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "facts_cache_timeout", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.363 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache host facts data", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "grafana_dashboards_path": { 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "grafana_dashboards_path", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.364 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "host_check_interval": { 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "host_check_interval", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.365 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to perform a host check", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "hw_monitoring": { 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hw_monitoring", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.366 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "inventory_list_all": { 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "inventory_list_all", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.367 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.368 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "log_refresh_metadata": { 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_refresh_metadata", 2024-04-22T13:44:20.369 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.370 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.371 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.372 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.374 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "*", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.375 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "max_count_per_host": { 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_count_per_host", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.376 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of daemons per service per host", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd_draining_count": { 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_osd_draining_count", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.377 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "migration_current": { 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "migration_current", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "internal - do not modify", 2024-04-22T13:44:20.378 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "root", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm-package", 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "root" 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.379 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "oob_default_addr": { 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "oob_default_addr", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "169.254.1.1", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T13:44:20.380 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "prometheus_alerts_path": { 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus_alerts_path", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.381 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_insecure": { 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_insecure", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T13:44:20.382 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_password": { 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_password", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T13:44:20.383 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_url": { 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_url", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T13:44:20.384 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_username": { 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_username", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.385 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "secure_monitoring_stack": { 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "secure_monitoring_stack", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.386 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "service_discovery_port": { 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "service_discovery_port", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8765", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "cephadm service discovery port", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.387 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "ssh_config_file": { 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssh_config_file", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.388 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "use_agent": { 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_agent", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.389 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "use_repo_digest": { 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_repo_digest", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.390 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_failed_host_check": { 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_failed_host_check", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.391 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_daemons": { 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_daemons", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.392 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_hosts": { 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_hosts", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crash", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.394 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.395 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.396 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.397 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.398 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "retain_interval": { 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retain_interval", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.399 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "31536000", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_recent_interval": { 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_recent_interval", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1209600", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.400 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dashboard", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T13:44:20.401 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.402 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.403 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_ENABLED": { 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.404 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.406 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.407 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.408 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.409 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.410 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.411 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.412 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.413 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.414 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.415 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.416 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_URL": { 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_URL", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.417 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.418 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.419 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.420 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.421 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.422 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "[]", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.423 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "{}", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.424 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.425 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.426 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T13:44:20.427 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T13:44:20.428 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.429 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.430 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.431 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.432 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.433 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T13:44:20.434 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.435 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.436 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "45", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.437 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.438 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.440 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.441 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.442 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.443 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.445 INFO:teuthology.orchestra.run.smithi087.stdout: "cross_origin_url": { 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cross_origin_url", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.446 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "crt_file": { 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crt_file", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.447 INFO:teuthology.orchestra.run.smithi087.stdout: "debug": { 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "debug", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable/disable debug options", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "jwt_token_ttl": { 2024-04-22T13:44:20.448 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "jwt_token_ttl", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "28800", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.449 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.450 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.451 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.453 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.454 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "motd": { 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "motd", 2024-04-22T13:44:20.458 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "The message of the day", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect_resolve_ip_addr": { 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T13:44:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:20.460 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8080", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.462 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl_server_port": { 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl_server_port", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8443", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "redirect", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect" 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.465 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.466 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "url_prefix": { 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url_prefix", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.467 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "devicehealth", 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_monitoring": { 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_monitoring", 2024-04-22T13:44:20.468 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "monitor device health metrics", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.469 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.470 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.471 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.472 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.473 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "mark_out_threshold": { 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mark_out_threshold", 2024-04-22T13:44:20.474 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2419200", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": { 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_name", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "device_health_metrics", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.476 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "retention_period": { 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retention_period", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15552000", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain device health metrics", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.477 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_frequency": { 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_frequency", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "self_heal": { 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "self_heal", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.479 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.480 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_threshold": { 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_threshold", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7257600", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.481 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "diskprediction_local", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.482 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.483 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.485 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.486 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "predict_interval": { 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predict_interval", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "predictor_model": { 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predictor_model", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "prophetstor", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.488 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.489 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "influx", 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": false, 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "influxdb python module not found", 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "batch_size": { 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "batch_size", 2024-04-22T13:44:20.490 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5000", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "database": { 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "database", 2024-04-22T13:44:20.491 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "ceph", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "hostname": { 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hostname", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.492 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server hostname", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.493 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "30", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "5", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.494 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.495 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.496 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.497 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.498 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "password": { 2024-04-22T13:44:20.499 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "password", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "password of InfluxDB server user", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "port": { 2024-04-22T13:44:20.500 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "port", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8086", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server port", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:20.501 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "false", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "threads": { 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threads", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "32", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.503 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "username": { 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "username", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "username of InfluxDB server user", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.504 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "verify_ssl": { 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "verify_ssl", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "true", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "insights", 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.506 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.508 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.509 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.510 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "iostat", 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.511 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.512 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.513 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.514 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.515 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "k8sevents", 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_event_retention_days": { 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ceph_event_retention_days", 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.517 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "config_check_secs": { 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_check_secs", 2024-04-22T13:44:20.518 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "10", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.519 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.520 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.521 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.522 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.523 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.524 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "localpool", 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "failure_domain": { 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "failure_domain", 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.525 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "host", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "failure domain for any created local pool", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.526 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.527 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.528 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.529 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": { 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_size", 2024-04-22T13:44:20.531 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default min_size for any created local pool", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.532 INFO:teuthology.orchestra.run.smithi087.stdout: "num_rep": { 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "num_rep", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default replica count for any created local pool", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": { 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_num", 2024-04-22T13:44:20.533 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "128", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "prefix": { 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prefix", 2024-04-22T13:44:20.534 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name prefix for any created local pool", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "subtree": { 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "subtree", 2024-04-22T13:44:20.535 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "rack", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mds_autoscaler", 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.537 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.538 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.539 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.540 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.541 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirroring", 2024-04-22T13:44:20.542 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.543 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.544 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.545 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.546 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.547 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "nfs", 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.549 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.550 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.551 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "fail_fs": { 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "fail_fs", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.555 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.556 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.557 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.558 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.559 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.560 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator": { 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "rook", 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "test_orchestrator" 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.561 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Orchestrator backend", 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_perf_query", 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.562 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.563 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.564 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.565 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.566 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.567 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_support", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.568 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.569 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.570 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.571 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.572 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_autoscaler", 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.573 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.574 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.575 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.576 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.577 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.578 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.579 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "threshold": { 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threshold", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1.0", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "scaling threshold", 2024-04-22T13:44:20.580 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "progress", 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_pg_recovery_event": { 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_pg_recovery_event", 2024-04-22T13:44:20.581 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:20.582 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.583 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.584 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.585 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.586 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.587 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "max_completed_events": { 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_completed_events", 2024-04-22T13:44:20.588 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "50", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "number of past completed events to remember", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.589 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long the module is going to sleep", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "cache": { 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cache", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.591 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "exclude_perf_counters": { 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "exclude_perf_counters", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T13:44:20.592 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.593 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.594 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.595 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.596 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.597 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools": { 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.598 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "300", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.599 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_interval": { 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_interval", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15.0", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.601 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "9283", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.602 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "stale_cache_strategy": { 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stale_cache_strategy", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "log", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.603 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "default", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.604 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "default", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "error" 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.605 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.606 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_support", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.607 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.608 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.609 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.610 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.611 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_snap_create": { 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_snap_create", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.612 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "mirror_snapshot_schedule": { 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.613 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "trash_purge_schedule": { 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "trash_purge_schedule", 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.614 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "restful", 2024-04-22T13:44:20.615 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_auth": { 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_auth", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.616 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.617 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.618 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.619 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.620 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.621 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.622 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:20.623 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:20.624 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rgw", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.626 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.627 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.629 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.630 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rook", 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.631 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.632 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.633 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.634 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.635 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.636 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "storage_class": { 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "storage_class", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "local", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T13:44:20.637 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "selftest", 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.638 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.639 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.643 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "roption1": { 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption1", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "roption2": { 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption2", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "xyz", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.645 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption1": { 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption1", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.646 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption2": { 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption2", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.647 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption3": { 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption3", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption4": { 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption4", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption5": { 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption5", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.650 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption6": { 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption6", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.651 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption7": { 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption7", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "42", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.652 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "testkey": { 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testkey", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "testlkey": { 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testlkey", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "testnewline": { 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testnewline", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.655 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snap_schedule", 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_m_granularity": { 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_m_granularity", 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "dump_on_update": { 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dump_on_update", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.657 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "dump database to debug log on update", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.659 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.660 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.661 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.662 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stats", 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.663 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.664 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.665 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.666 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.667 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "status", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.669 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.670 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.671 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.672 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.673 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telegraf", 2024-04-22T13:44:20.674 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "address": { 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "address", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.675 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.676 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.677 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.678 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.679 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.680 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telemetry", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_basic": { 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_basic", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.682 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_crash": { 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_crash", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.683 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_device": { 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_device", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.684 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_ident": { 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_ident", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.685 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_perf": { 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_perf", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.686 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "contact": { 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "contact", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.687 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "description": { 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "description", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.688 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "device_url": { 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_url", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.690 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.691 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "24", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "8", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "last_opt_revision": { 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "last_opt_revision", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.692 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard": { 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.693 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard_description": { 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard_description", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.694 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.695 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.698 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.699 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "organization": { 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "organization", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.700 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "proxy": { 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "proxy", 2024-04-22T13:44:20.701 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.702 INFO:teuthology.orchestra.run.smithi087.stdout: "url": { 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.703 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "test_orchestrator", 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.704 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.706 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.707 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.708 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "volumes", 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.709 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.710 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.711 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.712 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.714 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_clones": { 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_clones", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.715 INFO:teuthology.orchestra.run.smithi087.stdout: "periodic_async_work": { 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "periodic_async_work", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Periodically check for async work", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_delay": { 2024-04-22T13:44:20.716 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_delay", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_no_wait": { 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.718 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix", 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "discovery_interval": { 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "discovery_interval", 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.719 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "100", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "identifier": { 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "identifier", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.720 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.721 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.722 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.724 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.725 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.726 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_host": { 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_host", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.727 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_port": { 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_port", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10051", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.728 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_sender": { 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_sender", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.729 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "gid": 24832, 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "z", 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_features": 4540701547738038271, 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "available_modules": [ 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "alerts", 2024-04-22T13:44:20.730 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.732 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.733 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.734 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.735 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.736 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_destination": { 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_destination", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email address to send alerts to", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.737 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_from_name": { 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_from_name", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "Ceph", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email From: name", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_host": { 2024-04-22T13:44:20.738 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_host", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP server", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_password": { 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_password", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.739 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Password to authenticate with", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_port": { 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_port", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.740 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "465", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP port", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_sender": { 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_sender", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.741 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP envelope sender", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_ssl": { 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_ssl", 2024-04-22T13:44:20.742 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_user": { 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_user", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.743 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "User to authenticate as", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "balancer", 2024-04-22T13:44:20.744 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "active": { 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "active", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.745 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_time": { 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_time", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0000", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T13:44:20.746 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_weekday": { 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_weekday", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T13:44:20.747 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_max_iterations": { 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_max_iterations", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "25", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "250", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T13:44:20.748 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_metrics": { 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_metrics", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T13:44:20.749 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_step": { 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_step", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.5", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0.001", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "0.999", 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.750 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "aggressiveness of optimization", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "end_time": { 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_time", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2359", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.751 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "end_weekday": { 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_weekday", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T13:44:20.752 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.753 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.754 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.755 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.756 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.757 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "min_score": { 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_score", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.758 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "upmap", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-compat", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "none", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "read", 2024-04-22T13:44:20.759 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap-read" 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Balancer mode", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_ids": { 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_ids", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.760 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.761 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_deviation": { 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_deviation", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.762 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_optimizations": { 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_optimizations", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.763 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm", 2024-04-22T13:44:20.764 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_down_multiplier": { 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_down_multiplier", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.765 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_refresh_rate": { 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_refresh_rate", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "20", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T13:44:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_starting_port": { 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_starting_port", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4721", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.767 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_ptrace": { 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_ptrace", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.768 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_interval": { 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_interval", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.769 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_memory_target_ratio": { 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.7", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm_log_destination": { 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm_log_destination", 2024-04-22T13:44:20.771 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "file", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "file,syslog", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "syslog" 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T13:44:20.772 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "cgroups_split": { 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cgroups_split", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.773 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "config_checks_enabled": { 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_checks_enabled", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.774 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "config_dashboard": { 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_dashboard", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.775 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_alertmanager": { 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_alertmanager", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T13:44:20.776 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_base": { 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_base", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T13:44:20.777 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Container image name, without the tag", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_elasticsearch": { 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_elasticsearch", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T13:44:20.778 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "elasticsearch container image", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_grafana": { 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_grafana", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.779 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_haproxy": { 2024-04-22T13:44:20.780 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_haproxy", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "HAproxy container image", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.781 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_agent": { 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_agent", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.782 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger agent container image", 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_collector": { 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_collector", 2024-04-22T13:44:20.783 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger collector container image", 2024-04-22T13:44:20.784 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_query": { 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_query", 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.785 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger query container image", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.786 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_keepalived": { 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_keepalived", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Keepalived container image", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.787 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_loki": { 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_loki", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.788 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Loki container image", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_node_exporter": { 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_node_exporter", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.789 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_nvmeof": { 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_nvmeof", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.790 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Nvme-of container image", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_prometheus": { 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_prometheus", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.791 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_promtail": { 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_promtail", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T13:44:20.792 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Promtail container image", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_samba": { 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_samba", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T13:44:20.793 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Samba/SMB container image", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_snmp_gateway": { 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_snmp_gateway", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.794 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SNMP Gateway container image", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "container_init": { 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_init", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.795 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "daemon_cache_timeout": { 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "daemon_cache_timeout", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.796 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "default_cephadm_command_timeout": { 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.797 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "900", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "default_registry": { 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_registry", 2024-04-22T13:44:20.798 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "device_cache_timeout": { 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_cache_timeout", 2024-04-22T13:44:20.799 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1800", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache device inventory", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "device_enhanced_scan": { 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_enhanced_scan", 2024-04-22T13:44:20.800 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "facts_cache_timeout": { 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "facts_cache_timeout", 2024-04-22T13:44:20.801 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache host facts data", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "grafana_dashboards_path": { 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "grafana_dashboards_path", 2024-04-22T13:44:20.802 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: "host_check_interval": { 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "host_check_interval", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to perform a host check", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "hw_monitoring": { 2024-04-22T13:44:20.804 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hw_monitoring", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "inventory_list_all": { 2024-04-22T13:44:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "inventory_list_all", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.806 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.807 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "log_refresh_metadata": { 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_refresh_metadata", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.808 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.809 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.811 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.812 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T13:44:20.813 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "*", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.814 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "max_count_per_host": { 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_count_per_host", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.815 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of daemons per service per host", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd_draining_count": { 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_osd_draining_count", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.816 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "migration_current": { 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "migration_current", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.817 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "internal - do not modify", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.818 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "root", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm-package", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "root" 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "oob_default_addr": { 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "oob_default_addr", 2024-04-22T13:44:20.819 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "169.254.1.1", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "prometheus_alerts_path": { 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus_alerts_path", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_insecure": { 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_insecure", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.821 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_password": { 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_password", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.822 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_url": { 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_url", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.823 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_username": { 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_username", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.824 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "secure_monitoring_stack": { 2024-04-22T13:44:20.825 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "secure_monitoring_stack", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.826 INFO:teuthology.orchestra.run.smithi087.stdout: "service_discovery_port": { 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "service_discovery_port", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8765", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "cephadm service discovery port", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "ssh_config_file": { 2024-04-22T13:44:20.827 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssh_config_file", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "use_agent": { 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_agent", 2024-04-22T13:44:20.828 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "use_repo_digest": { 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_repo_digest", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.829 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_failed_host_check": { 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_failed_host_check", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.830 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_daemons": { 2024-04-22T13:44:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_daemons", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.832 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_hosts": { 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_hosts", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.833 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crash", 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.834 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.835 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.836 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.837 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.838 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.839 INFO:teuthology.orchestra.run.smithi087.stdout: "retain_interval": { 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retain_interval", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "31536000", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.840 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_recent_interval": { 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_recent_interval", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1209600", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.841 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dashboard", 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.842 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.843 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.844 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_ENABLED": { 2024-04-22T13:44:20.845 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.846 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.847 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.848 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.849 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.850 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.851 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.852 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.853 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.855 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T13:44:20.856 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T13:44:20.857 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.858 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_URL": { 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_URL", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.860 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.861 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.862 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.863 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.865 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T13:44:20.866 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "[]", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T13:44:20.867 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.869 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.869 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.869 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "{}", 2024-04-22T13:44:20.869 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.869 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.870 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.871 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.872 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.873 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.875 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.876 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.877 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.878 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.879 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.880 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.882 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "45", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.884 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.885 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.886 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.887 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.888 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.889 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.890 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.891 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cross_origin_url": { 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cross_origin_url", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.892 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "crt_file": { 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crt_file", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.893 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "debug": { 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "debug", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.894 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable/disable debug options", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "jwt_token_ttl": { 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "jwt_token_ttl", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.895 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "28800", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.896 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.897 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.898 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.899 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.900 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.901 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "motd": { 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "motd", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.902 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "The message of the day", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect_resolve_ip_addr": { 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T13:44:20.903 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.904 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.905 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8080", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.906 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl_server_port": { 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl_server_port", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.907 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8443", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "redirect", 2024-04-22T13:44:20.908 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect" 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.909 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "url_prefix": { 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url_prefix", 2024-04-22T13:44:20.910 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "devicehealth", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_monitoring": { 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_monitoring", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "monitor device health metrics", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.913 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.914 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.915 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.916 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.917 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "mark_out_threshold": { 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mark_out_threshold", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2419200", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.918 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": { 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_name", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "device_health_metrics", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.919 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "retention_period": { 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retention_period", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15552000", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.920 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain device health metrics", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_frequency": { 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_frequency", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.921 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "self_heal": { 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "self_heal", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.922 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T13:44:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_threshold": { 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_threshold", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7257600", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.924 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "diskprediction_local", 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.925 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.926 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.927 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.928 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.929 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "predict_interval": { 2024-04-22T13:44:20.930 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predict_interval", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "predictor_model": { 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predictor_model", 2024-04-22T13:44:20.931 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "prophetstor", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:20.932 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "influx", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": false, 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "influxdb python module not found", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "batch_size": { 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "batch_size", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5000", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "database": { 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "database", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "ceph", 2024-04-22T13:44:20.935 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "hostname": { 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hostname", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.936 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server hostname", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.937 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "30", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "5", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.939 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.940 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.941 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.942 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "password": { 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "password", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.943 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "password of InfluxDB server user", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "port": { 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "port", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8086", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.944 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server port", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "false", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.945 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "threads": { 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threads", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.946 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "32", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "username": { 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "username", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.947 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "username of InfluxDB server user", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "verify_ssl": { 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "verify_ssl", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.948 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "true", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "insights", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.949 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.953 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.954 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "iostat", 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.955 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.956 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.957 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.958 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.959 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.960 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.961 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "k8sevents", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_event_retention_days": { 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ceph_event_retention_days", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.962 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "config_check_secs": { 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_check_secs", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "10", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.963 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.964 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.965 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.966 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.967 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.968 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "localpool", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "failure_domain": { 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "failure_domain", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.969 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "host", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "failure domain for any created local pool", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.970 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.971 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.972 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.973 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.974 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.975 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": { 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_size", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.976 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default min_size for any created local pool", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "num_rep": { 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "num_rep", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.977 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default replica count for any created local pool", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": { 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_num", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "128", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.978 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "prefix": { 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prefix", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.979 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name prefix for any created local pool", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "subtree": { 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "subtree", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "rack", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.980 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mds_autoscaler", 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.981 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.982 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.983 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.984 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.985 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.986 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.987 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirroring", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.988 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.989 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.990 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.991 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.992 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "nfs", 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:20.993 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.994 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.995 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.996 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.997 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:20.998 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "fail_fs": { 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "fail_fs", 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:20.999 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.000 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.001 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.002 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.003 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.004 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.005 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator": { 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "rook", 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "test_orchestrator" 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.006 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Orchestrator backend", 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_perf_query", 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.007 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.008 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.009 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.010 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.011 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.012 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_support", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.014 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.015 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.017 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.018 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_autoscaler", 2024-04-22T13:44:21.019 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.020 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.021 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.022 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.023 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.024 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.025 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "threshold": { 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threshold", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.026 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1.0", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "scaling threshold", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "progress", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.027 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_pg_recovery_event": { 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_pg_recovery_event", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.028 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.029 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.030 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.031 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.032 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.033 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.034 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "max_completed_events": { 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_completed_events", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "50", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.035 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "number of past completed events to remember", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:21.036 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long the module is going to sleep", 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.037 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "cache": { 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cache", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.038 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "exclude_perf_counters": { 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "exclude_perf_counters", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.039 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.040 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.041 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.042 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.043 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.044 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.045 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools": { 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.046 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "300", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.047 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_interval": { 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_interval", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15.0", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.048 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T13:44:21.049 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "9283", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T13:44:21.050 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "stale_cache_strategy": { 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stale_cache_strategy", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "log", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.051 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "default", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "default", 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: "error" 2024-04-22T13:44:21.052 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.054 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_support", 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.055 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.056 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.057 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.058 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.059 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.060 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.061 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.062 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.063 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.064 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_snap_create": { 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_snap_create", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.065 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.066 INFO:teuthology.orchestra.run.smithi087.stdout: "mirror_snapshot_schedule": { 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "trash_purge_schedule": { 2024-04-22T13:44:21.067 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "trash_purge_schedule", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.068 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "restful", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_auth": { 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_auth", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.069 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:21.070 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:21.073 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.073 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.074 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.075 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.076 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.077 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.078 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.079 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.080 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.081 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rgw", 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.082 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.083 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.084 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.085 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.086 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rook", 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.088 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.089 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.090 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.091 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.092 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "storage_class": { 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "storage_class", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.093 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "local", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "selftest", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.095 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.096 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.097 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.098 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.099 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "roption1": { 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption1", 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.100 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "roption2": { 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption2", 2024-04-22T13:44:21.101 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "xyz", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption1": { 2024-04-22T13:44:21.102 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption1", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption2": { 2024-04-22T13:44:21.103 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption2", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption3": { 2024-04-22T13:44:21.104 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption3", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption4": { 2024-04-22T13:44:21.105 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption4", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption5": { 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption5", 2024-04-22T13:44:21.106 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption6": { 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption6", 2024-04-22T13:44:21.107 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption7": { 2024-04-22T13:44:21.108 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption7", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "42", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "testkey": { 2024-04-22T13:44:21.109 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testkey", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: "testlkey": { 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testlkey", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.111 INFO:teuthology.orchestra.run.smithi087.stdout: "testnewline": { 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testnewline", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snap_schedule", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_m_granularity": { 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_m_granularity", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.113 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "dump_on_update": { 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dump_on_update", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.114 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "dump database to debug log on update", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.115 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.116 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.117 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.118 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.119 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stats", 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.120 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.121 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.122 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.123 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.124 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "status", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.125 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.126 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.127 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.128 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.129 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.130 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telegraf", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "address": { 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "address", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.131 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.132 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.133 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.134 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.135 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.136 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.137 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telemetry", 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_basic": { 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_basic", 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.138 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_crash": { 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_crash", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.139 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_device": { 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_device", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.140 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_ident": { 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_ident", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.141 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_perf": { 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_perf", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.142 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "contact": { 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "contact", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.143 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "description": { 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "description", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.144 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "device_url": { 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_url", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.145 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.146 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.147 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "24", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "8", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "last_opt_revision": { 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "last_opt_revision", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.148 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard": { 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.149 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard_description": { 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard_description", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.150 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.151 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.152 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.153 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.154 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.155 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "organization": { 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "organization", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.156 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "proxy": { 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "proxy", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.157 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "url": { 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.158 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "test_orchestrator", 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.159 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.160 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.161 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.162 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.163 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.164 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "volumes", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.165 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.166 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.167 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.169 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_clones": { 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_clones", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.170 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "periodic_async_work": { 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "periodic_async_work", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.171 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Periodically check for async work", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_delay": { 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_delay", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.172 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_no_wait": { 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T13:44:21.173 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix", 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "discovery_interval": { 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "discovery_interval", 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.174 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "100", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "identifier": { 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "identifier", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.175 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.178 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.179 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.180 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.181 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_host": { 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_host", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.183 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_port": { 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_port", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10051", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.184 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_sender": { 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_sender", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.185 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "modules": [ 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm", 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "dashboard", 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "iostat", 2024-04-22T13:44:21.186 INFO:teuthology.orchestra.run.smithi087.stdout: "nfs", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "restful", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_schedule" 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "available_modules": [ 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "alerts", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.187 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.188 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.189 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.190 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.191 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.192 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_destination": { 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_destination", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.193 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email address to send alerts to", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_from_name": { 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_from_name", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "Ceph", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.194 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Email From: name", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_host": { 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_host", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.195 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP server", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_password": { 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_password", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Password to authenticate with", 2024-04-22T13:44:21.196 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_port": { 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_port", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "465", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP port", 2024-04-22T13:44:21.197 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_sender": { 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_sender", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.198 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SMTP envelope sender", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_ssl": { 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_ssl", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T13:44:21.199 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "smtp_user": { 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "smtp_user", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "User to authenticate as", 2024-04-22T13:44:21.200 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "balancer", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "active": { 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "active", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.201 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_time": { 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_time", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.202 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0000", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "begin_weekday": { 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "begin_weekday", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.203 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_max_iterations": { 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_max_iterations", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.204 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "25", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "250", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_metrics": { 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_metrics", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.205 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_compat_step": { 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crush_compat_step", 2024-04-22T13:44:21.206 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.5", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0.001", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "0.999", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "aggressiveness of optimization", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "end_time": { 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_time", 2024-04-22T13:44:21.207 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2359", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "end_weekday": { 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "end_weekday", 2024-04-22T13:44:21.208 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "0", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "6", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.209 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.210 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.212 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.213 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.214 INFO:teuthology.orchestra.run.smithi087.stdout: "min_score": { 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_score", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.215 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "upmap", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-compat", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "none", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "read", 2024-04-22T13:44:21.216 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap-read" 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Balancer mode", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_ids": { 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_ids", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.217 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.218 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_deviation": { 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_deviation", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:21.219 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "upmap_max_optimizations": { 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "upmap_max_optimizations", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.220 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.221 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_down_multiplier": { 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_down_multiplier", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.222 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_refresh_rate": { 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_refresh_rate", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "20", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.223 INFO:teuthology.orchestra.run.smithi087.stdout: "agent_starting_port": { 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "agent_starting_port", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4721", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_ptrace": { 2024-04-22T13:44:21.224 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_ptrace", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_interval": { 2024-04-22T13:44:21.225 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_interval", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.226 INFO:teuthology.orchestra.run.smithi087.stdout: "autotune_memory_target_ratio": { 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0.7", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.227 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm_log_destination": { 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cephadm_log_destination", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "file", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "file,syslog", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "syslog" 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.228 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "cgroups_split": { 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cgroups_split", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.229 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "config_checks_enabled": { 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_checks_enabled", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T13:44:21.230 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "config_dashboard": { 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_dashboard", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.231 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_alertmanager": { 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_alertmanager", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:21.232 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_base": { 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_base", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.233 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Container image name, without the tag", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_elasticsearch": { 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_elasticsearch", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "elasticsearch container image", 2024-04-22T13:44:21.234 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_grafana": { 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_grafana", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:21.235 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_haproxy": { 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_haproxy", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "HAproxy container image", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.236 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_agent": { 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_agent", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.237 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger agent container image", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_collector": { 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_collector", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.238 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger collector container image", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_jaeger_query": { 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_jaeger_query", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.239 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Jaeger query container image", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_keepalived": { 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_keepalived", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Keepalived container image", 2024-04-22T13:44:21.240 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_loki": { 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_loki", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.241 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Loki container image", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_node_exporter": { 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_node_exporter", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.242 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_nvmeof": { 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_nvmeof", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.243 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Nvme-of container image", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_prometheus": { 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_prometheus", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Prometheus container image", 2024-04-22T13:44:21.244 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_promtail": { 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_promtail", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Promtail container image", 2024-04-22T13:44:21.245 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_samba": { 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_samba", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.246 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Samba/SMB container image", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image_snmp_gateway": { 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_image_snmp_gateway", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.247 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "SNMP Gateway container image", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "container_init": { 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "container_init", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.248 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "daemon_cache_timeout": { 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "daemon_cache_timeout", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "default_cephadm_command_timeout": { 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "900", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.250 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "default_registry": { 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "default_registry", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "docker.io", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.251 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "device_cache_timeout": { 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_cache_timeout", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1800", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.252 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache device inventory", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "device_enhanced_scan": { 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_enhanced_scan", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.253 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "facts_cache_timeout": { 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "facts_cache_timeout", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.254 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "seconds to cache host facts data", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "grafana_dashboards_path": { 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "grafana_dashboards_path", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T13:44:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "host_check_interval": { 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "host_check_interval", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to perform a host check", 2024-04-22T13:44:21.256 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "hw_monitoring": { 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hw_monitoring", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T13:44:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "inventory_list_all": { 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "inventory_list_all", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T13:44:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "log_refresh_metadata": { 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_refresh_metadata", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.262 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.263 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.264 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.265 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "*", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T13:44:21.266 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "max_count_per_host": { 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_count_per_host", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of daemons per service per host", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.267 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd_draining_count": { 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_osd_draining_count", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.268 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "migration_current": { 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "migration_current", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "internal - do not modify", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.269 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "mode": { 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mode", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "root", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm-package", 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "root" 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.270 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "oob_default_addr": { 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "oob_default_addr", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "169.254.1.1", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.271 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "prometheus_alerts_path": { 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus_alerts_path", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.272 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_insecure": { 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_insecure", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.273 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_password": { 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_password", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.274 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_url": { 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_url", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.275 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "registry_username": { 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "registry_username", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.276 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "secure_monitoring_stack": { 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "secure_monitoring_stack", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.277 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "service_discovery_port": { 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "service_discovery_port", 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.278 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8765", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "cephadm service discovery port", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "ssh_config_file": { 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssh_config_file", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "use_agent": { 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_agent", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.280 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "use_repo_digest": { 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "use_repo_digest", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.281 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_failed_host_check": { 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_failed_host_check", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.282 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_daemons": { 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_daemons", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.283 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_on_stray_hosts": { 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_on_stray_hosts", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.284 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crash", 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.285 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.286 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.287 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.288 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.289 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.290 INFO:teuthology.orchestra.run.smithi087.stdout: "retain_interval": { 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retain_interval", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "31536000", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.291 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_recent_interval": { 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_recent_interval", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1209600", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.292 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dashboard", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.293 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.294 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.295 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_ENABLED": { 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.296 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.297 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.298 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.299 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.300 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.301 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.302 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.303 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.304 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.305 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.306 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.307 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.308 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_URL": { 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_URL", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.309 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:21.310 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.311 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.312 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.313 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.314 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "[]", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.315 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "{}", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.316 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.317 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.318 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.319 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.320 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.321 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.322 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.323 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.324 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.325 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.326 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.327 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.328 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.329 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "45", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.330 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.331 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "admin", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.332 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.333 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.334 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.335 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.336 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.337 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.338 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "cross_origin_url": { 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cross_origin_url", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.339 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "crt_file": { 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "crt_file", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.340 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "debug": { 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "debug", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.341 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Enable/disable debug options", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "jwt_token_ttl": { 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "jwt_token_ttl", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "28800", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.342 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.344 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.345 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.346 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.347 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "motd": { 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "motd", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "The message of the day", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.349 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect_resolve_ip_addr": { 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8080", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.353 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl_server_port": { 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl_server_port", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8443", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "redirect", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: "redirect" 2024-04-22T13:44:21.355 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:21.356 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "url_prefix": { 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url_prefix", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.357 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "devicehealth", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.358 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_monitoring": { 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_monitoring", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "monitor device health metrics", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.359 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.360 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.361 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.362 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.363 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.364 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "mark_out_threshold": { 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mark_out_threshold", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "2419200", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": { 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pool_name", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "device_health_metrics", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.366 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "retention_period": { 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "retention_period", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15552000", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.367 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long to retain device health metrics", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_frequency": { 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_frequency", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.368 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "self_heal": { 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "self_heal", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.369 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.370 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "warn_threshold": { 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "warn_threshold", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7257600", 2024-04-22T13:44:21.371 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "diskprediction_local", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.372 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.373 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.374 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.375 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.376 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "predict_interval": { 2024-04-22T13:44:21.377 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predict_interval", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "86400", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.378 INFO:teuthology.orchestra.run.smithi087.stdout: "predictor_model": { 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "predictor_model", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "prophetstor", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.379 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "600", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.380 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "influx", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": false, 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "influxdb python module not found", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "batch_size": { 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "batch_size", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5000", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.381 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "database": { 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "database", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "ceph", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.382 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "hostname": { 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "hostname", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.383 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server hostname", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "30", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "5", 2024-04-22T13:44:21.384 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.385 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.386 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.388 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "password": { 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "password", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.390 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "password of InfluxDB server user", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "port": { 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "port", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.391 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "8086", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "InfluxDB server port", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "ssl": { 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ssl", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.392 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "false", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "threads": { 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threads", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.393 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "32", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "username": { 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "username", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.394 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "username of InfluxDB server user", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "verify_ssl": { 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "verify_ssl", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.395 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "true", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "insights", 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.396 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.397 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.398 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.399 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.400 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.401 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "iostat", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.402 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.403 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.404 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.405 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.406 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "k8sevents", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_event_retention_days": { 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "ceph_event_retention_days", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.407 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "7", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "config_check_secs": { 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "config_check_secs", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.408 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "10", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.411 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.412 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.413 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "localpool", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "failure_domain": { 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "failure_domain", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "host", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "failure domain for any created local pool", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.415 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.416 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.417 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.418 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.419 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": { 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "min_size", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default min_size for any created local pool", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T13:44:21.420 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "num_rep": { 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "num_rep", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default replica count for any created local pool", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.421 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": { 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_num", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "128", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.422 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "prefix": { 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prefix", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.423 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "name prefix for any created local pool", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "subtree": { 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "subtree", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "rack", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T13:44:21.424 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mds_autoscaler", 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.425 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.426 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.427 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.428 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.429 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.430 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirroring", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.431 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.432 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.433 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.434 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.435 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "nfs", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.436 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.437 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.438 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.439 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.440 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "fail_fs": { 2024-04-22T13:44:21.441 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "fail_fs", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.442 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.445 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.446 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.447 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.448 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.449 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator": { 2024-04-22T13:44:21.450 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "orchestrator", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "rook", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "test_orchestrator" 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Orchestrator backend", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.451 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_perf_query", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.452 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.456 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "osd_support", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.457 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.458 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.459 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.462 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "pg_autoscaler", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.463 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.464 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.465 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.466 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.467 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.468 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "threshold": { 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "threshold", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "3.0", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1.0", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "scaling threshold", 2024-04-22T13:44:21.469 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "progress", 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_pg_recovery_event": { 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_pg_recovery_event", 2024-04-22T13:44:21.470 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:21.471 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.472 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.473 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.474 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.475 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.477 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "max_completed_events": { 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_completed_events", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "50", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "number of past completed events to remember", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.478 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "sleep_interval": { 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "sleep_interval", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "5", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "how long the module is going to sleep", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.479 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "prometheus", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "cache": { 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "cache", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.480 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "exclude_perf_counters": { 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "exclude_perf_counters", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.481 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.482 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.483 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.484 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.485 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.486 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools": { 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.487 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "300", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.488 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "scrape_interval": { 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "scrape_interval", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15.0", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "::", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.490 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "9283", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.491 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "stale_cache_strategy": { 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stale_cache_strategy", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "log", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.492 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_behaviour": { 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_behaviour", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "default", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.493 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "default", 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "error" 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "standby_error_status_code": { 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "standby_error_status_code", 2024-04-22T13:44:21.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "500", 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "400", 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "599", 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.495 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_support", 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.496 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.497 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.498 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.499 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.500 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_snap_create": { 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_snap_create", 2024-04-22T13:44:21.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "mirror_snapshot_schedule": { 2024-04-22T13:44:21.502 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "trash_purge_schedule": { 2024-04-22T13:44:21.503 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "trash_purge_schedule", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "restful", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_auth": { 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enable_auth", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.505 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "key_file": { 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "key_file", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.506 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.507 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.508 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.509 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.510 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.511 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "server_addr": { 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_addr", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.512 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "server_port": { 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "server_port", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.513 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rgw", 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.514 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.515 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.516 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.517 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.518 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rook", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.519 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.520 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.521 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.522 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: "storage_class": { 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "storage_class", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "local", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "selftest", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.526 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.527 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.528 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.529 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.530 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "roption1": { 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption1", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.531 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "roption2": { 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "roption2", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "xyz", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.532 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption1": { 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption1", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.533 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption2": { 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption2", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.534 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption3": { 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption3", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "float", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.535 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption4": { 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption4", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.536 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption5": { 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption5", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.537 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption6": { 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption6", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.538 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "rwoption7": { 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rwoption7", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.539 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "1", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "42", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "testkey": { 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testkey", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.540 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "testlkey": { 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testlkey", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.541 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "testnewline": { 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "testnewline", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.542 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snap_schedule", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.543 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "allow_m_granularity": { 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "allow_m_granularity", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.544 INFO:teuthology.orchestra.run.smithi087.stdout: "dump_on_update": { 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "dump_on_update", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "dump database to debug log on update", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.545 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.546 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.547 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.548 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.549 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "stats", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.551 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.552 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.553 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.554 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.555 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "status", 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.556 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.557 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.558 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.559 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.560 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telegraf", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.561 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "address": { 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "address", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.562 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "15", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.563 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.564 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.565 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.566 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.567 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "telemetry", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_basic": { 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_basic", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.569 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_crash": { 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_crash", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.570 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_device": { 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_device", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.571 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_ident": { 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_ident", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.572 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "channel_perf": { 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "channel_perf", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.573 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "contact": { 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "contact", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.574 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "description": { 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "description", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.575 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "device_url": { 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "device_url", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.576 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "enabled": { 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "enabled", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.577 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.578 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "24", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "8", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "last_opt_revision": { 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "last_opt_revision", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.579 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "1", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard": { 2024-04-22T13:44:21.580 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "leaderboard_description": { 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "leaderboard_description", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.583 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.584 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.585 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.586 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "organization": { 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "organization", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.587 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "proxy": { 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "proxy", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "url": { 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "url", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T13:44:21.589 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "test_orchestrator", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.590 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.591 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.592 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.593 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.594 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.595 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "volumes", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.596 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.597 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.598 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.599 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.600 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "max_concurrent_clones": { 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "max_concurrent_clones", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "4", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T13:44:21.601 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "periodic_async_work": { 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "periodic_async_work", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Periodically check for async work", 2024-04-22T13:44:21.602 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_delay": { 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_delay", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "0", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.603 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "snapshot_clone_no_wait": { 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "True", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.604 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix", 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "can_run": true, 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "error_string": "", 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "module_options": { 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "discovery_interval": { 2024-04-22T13:44:21.605 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "discovery_interval", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "uint", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "100", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.606 INFO:teuthology.orchestra.run.smithi087.stdout: "identifier": { 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "identifier", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "interval": { 2024-04-22T13:44:21.607 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "interval", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "secs", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "60", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "log_level": { 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_level", 2024-04-22T13:44:21.608 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.609 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster": { 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.610 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_cluster_level": { 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_cluster_level", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "info", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [ 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "critical", 2024-04-22T13:44:21.611 INFO:teuthology.orchestra.run.smithi087.stdout: "debug", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "error", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "info", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "warning" 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": { 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "log_to_file", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "bool", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "False", 2024-04-22T13:44:21.612 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_host": { 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_host", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_port": { 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_port", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "int", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "10051", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "zabbix_sender": { 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "zabbix_sender", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "str", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "level": "advanced", 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 0, 2024-04-22T13:44:21.615 INFO:teuthology.orchestra.run.smithi087.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "min": "", 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "max": "", 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "enum_allowed": [], 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "desc": "", 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "long_desc": "", 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "tags": [], 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "see_also": [] 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "services": { 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "dashboard": "https://172.21.15.87:8443/" 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "always_on_modules": { 2024-04-22T13:44:21.616 INFO:teuthology.orchestra.run.smithi087.stdout: "octopus": [ 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "balancer", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "crash", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "devicehealth", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscaler", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "progress", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_support", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "status", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "telemetry", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "volumes" 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "pacific": [ 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "balancer", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "crash", 2024-04-22T13:44:21.617 INFO:teuthology.orchestra.run.smithi087.stdout: "devicehealth", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscaler", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "progress", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_support", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "status", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "telemetry", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "volumes" 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "quincy": [ 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "balancer", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "crash", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "devicehealth", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscaler", 2024-04-22T13:44:21.618 INFO:teuthology.orchestra.run.smithi087.stdout: "progress", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_support", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "status", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "telemetry", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "volumes" 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "reef": [ 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "balancer", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "crash", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "devicehealth", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscaler", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "progress", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_support", 2024-04-22T13:44:21.619 INFO:teuthology.orchestra.run.smithi087.stdout: "status", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "telemetry", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "volumes" 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "squid": [ 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "balancer", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "crash", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "devicehealth", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "orchestrator", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscaler", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "progress", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_support", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "status", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "telemetry", 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: "volumes" 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "last_failure_osd_epoch": 88, 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "active_clients": [ 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "devicehealth", 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3884705799 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.621 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "libcephsqlite", 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2798365384 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "rbd_support", 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2673501834 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snap_schedule", 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 1789473296 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "snap_schedule", 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 74812648 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "volumes", 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-04-22T13:44:21.624 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:0", 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 1382572189 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-04-22T13:44:21.625 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:21.626 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:20 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/905528107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.627 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:20 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2650150479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.627 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:20 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/120519075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T13:44:21.627 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:20 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/905528107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.627 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:20 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2650150479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.627 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:20 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/120519075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 msgr2=0x7fe1a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 0x7fe1a40a61e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac005c40 tx=0x7fe1ac00a780 comp rx=0 tx=0).stop 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 msgr2=0x7fe1bc19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400d750 tx=0x7fe1b400dc10 comp rx=0 tx=0).stop 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 shutdown_connections 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe1a40a3d20 0x7fe1a40a61e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe1bc10c520 0x7fe1bc19cfd0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc103b00 0x7fe1bc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 --2- 172.21.15.87:0/120519075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1bc1031e0 0x7fe1bc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.293+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 >> 172.21.15.87:0/120519075 conn(0x7fe1bc077f50 msgr2=0x7fe1bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.294+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 shutdown_connections 2024-04-22T13:44:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:20.294+0000 7fe1c1431640 1 -- 172.21.15.87:0/120519075 wait complete. 2024-04-22T13:44:21.635 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T13:44:21.636 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T13:44:21.636 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:20 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/905528107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.636 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:20 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2650150479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:21.637 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:20 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/120519075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T13:44:21.882 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:44:21.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:21 smithi120 ceph-mon[31101]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:21.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:21 smithi132 ceph-mon[31531]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:21.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:21 smithi087 ceph-mon[27999]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:23.670 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:23.670 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 248, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 237, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 1597008, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 145984, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.052961609, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.015758940, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.038960319, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.006020732, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 236, 2024-04-22T13:44:23.671 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.014350865, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000060808 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 1597008, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 145984 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 2, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 182, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 320, 2024-04-22T13:44:23.672 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 188144, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.015328677, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.010988042, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.001684961, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 179, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.017897424, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000099985 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:23.673 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 188144 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 4, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 4, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 2992, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 448, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.000582353, 2024-04-22T13:44:23.674 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.000048845, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.000536125, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 3, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000694025, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000231341 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T13:44:23.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "cct": { 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "total_workers": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "unhealthy_workers": 0 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "mempool": { 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:23.676 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:23.677 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:23.678 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_bytes": 148955, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_items": 30, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:23.679 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_items": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_items": 2, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_items": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_items": 0 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "val": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 104857600, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "get_started": 0, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "get": 250, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "get_sum": 1562004, 2024-04-22T13:44:23.680 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_success": 250, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "take": 0, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "take_sum": 0, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "put": 250, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "put_sum": 1562004, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "wait": { 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:23.681 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:23.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:23 smithi120 ceph-mon[31101]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:23.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:23 smithi132 ceph-mon[31531]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:23.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:23 smithi087 ceph-mon[27999]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:44:24.360 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-22T13:44:24.361 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-22T13:44:24.361 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-22T13:44:24.361 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-22T13:44:24.361 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-22T13:44:24.361 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T13:44:24.361 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T13:44:24.611 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:44:25.810 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:25 smithi087 ceph-mon[27999]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:25.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:25 smithi120 ceph-mon[31101]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:25.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:25 smithi132 ceph-mon[31531]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:26.374 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 0, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 2, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 0, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 0, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.001055680, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.000205881, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:26.375 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:26.376 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 259, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 246, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 1602528, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 150608, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.060624703, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.018215953, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.039694747, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.010960086, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 246, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.031781840, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000129194 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.377 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 1602528, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 150608 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 2, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 186, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 320, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 188848, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:26.378 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.016073206, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.011696797, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.001641687, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 183, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.014759730, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000080654 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:26.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 188848 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "cct": { 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "total_workers": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "unhealthy_workers": 0 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "mempool": { 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:26.380 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:26.381 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:26.382 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_bytes": 148967, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_items": 29, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:26.383 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_items": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_items": 2, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_items": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_items": 0 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "val": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 104857600, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "get_started": 0, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "get": 257, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "get_sum": 1563551, 2024-04-22T13:44:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_success": 257, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "take": 0, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "take_sum": 0, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "put": 257, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "put_sum": 1563551, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "wait": { 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:26.385 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:27.121 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T13:44:27.122 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T13:44:27.122 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-22T13:44:27.122 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-22T13:44:27.122 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-22T13:44:27.122 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T13:44:27.122 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T13:44:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:44:27.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:27 smithi087 ceph-mon[27999]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:27.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:27 smithi120 ceph-mon[31101]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:27.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:27 smithi132 ceph-mon[31531]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 254, 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 241, 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 1600544, 2024-04-22T13:44:29.125 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 147712, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.057369637, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.016737238, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.041407934, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.006158717, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 241, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.031085336, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000128984 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:29.126 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 1600544, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 147712 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 2, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 183, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 320, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 188320, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.015468614, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.011020996, 2024-04-22T13:44:29.127 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.001846071, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 180, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.037335328, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000207418 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 188320 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.128 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 0, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 2, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 0, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 0, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.000403243, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.000332602, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.129 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "cct": { 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "total_workers": 0, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "unhealthy_workers": 0 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mempool": { 2024-04-22T13:44:29.130 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:29.131 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:29.132 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_bytes": 148977, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_items": 30, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bytes": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_items": 0, 2024-04-22T13:44:29.133 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_items": 2, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_items": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:29.134 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_items": 0 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "val": 0, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 104857600, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "get_started": 0, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "get": 252, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "get_sum": 1562264, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_success": 252, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "take": 0, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "take_sum": 0, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "put": 252, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "put_sum": 1562264, 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "wait": { 2024-04-22T13:44:29.135 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:29.136 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:29.136 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:29.136 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:29.136 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:29.136 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:29.804 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:29 smithi087 ceph-mon[27999]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:29.830 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-22T13:44:29.831 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-22T13:44:29.831 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-22T13:44:29.831 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-22T13:44:29.831 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-22T13:44:29.831 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T13:44:29.831 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T13:44:29.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:29 smithi120 ceph-mon[31101]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:29.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:29 smithi132 ceph-mon[31531]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:30.069 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T13:44:31.786 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-04-22T13:44:31.786 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:31.786 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 261, 2024-04-22T13:44:31.786 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 248, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 1603072, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 151888, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.054071795, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.019036824, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.034809951, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.007499053, 2024-04-22T13:44:31.787 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 248, 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.025800699, 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000104035 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:31.788 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 1603072, 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 151888 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 2, 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 187, 2024-04-22T13:44:31.789 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 320, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 189024, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.016689233, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.011802102, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.001907787, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:31.790 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 184, 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.037127489, 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000201779 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:31.791 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 189024 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_messages": 0, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages": 2, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_bytes": 0, 2024-04-22T13:44:31.792 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_bytes": 0, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_total_time": 0.000383276, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_recv_time": 0.000320811, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:31.793 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:31.794 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "cct": { 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "total_workers": 0, 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "unhealthy_workers": 0 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.795 INFO:teuthology.orchestra.run.smithi087.stdout: "mempool": { 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:31.797 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_bytes": 148977, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_anon_items": 30, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bytes": 0, 2024-04-22T13:44:31.798 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "pgmap_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_co_items": 2, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_1_items": 0, 2024-04-22T13:44:31.799 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "unittest_2_items": 0 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "val": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "max": 104857600, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "get_started": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "get": 259, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "get_sum": 1563811, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "get_or_fail_success": 259, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "take": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "take_sum": 0, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "put": 259, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "put_sum": 1563811, 2024-04-22T13:44:31.800 INFO:teuthology.orchestra.run.smithi087.stdout: "wait": { 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout: "avgcount": 0, 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout: "sum": 0.000000000, 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout: "avgtime": 0.000000000 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-04-22T13:44:31.801 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-04-22T13:44:31.801 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:31 smithi087 ceph-mon[27999]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:31.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:31 smithi120 ceph-mon[31101]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:31.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:31 smithi132 ceph-mon[31531]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:32.538 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T13:44:32.538 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T13:44:32.538 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-22T13:44:32.539 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-22T13:44:32.539 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-22T13:44:32.539 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T13:44:32.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T13:44:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:44:33.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:33 smithi120 ceph-mon[31101]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:33.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:33 smithi132 ceph-mon[31531]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:33.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:33 smithi087 ceph-mon[27999]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 6996, 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 6066, 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 2108988, 2024-04-22T13:44:34.648 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 213683612, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 16, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 1.605040196, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.915501034, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.189199404, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.436543591, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6064, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 1.111665656, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000183322 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2469, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000512972, 2024-04-22T13:44:34.649 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000207 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 443984, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 124704 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 4245, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 4424, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1332732, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 104550257, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 319, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 8, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 1.111302834, 2024-04-22T13:44:34.650 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.537573080, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.199968007, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.344757190, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4425, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.747350110, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000168892 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2305, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000512712, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000222 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 42848, 2024-04-22T13:44:34.651 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 495760 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 156125, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 158532, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 58936425, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 140822076, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 26, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 6, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 9.644718441, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 4.426386245, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 11.792554859, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.275497643, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 158528, 2024-04-22T13:44:34.652 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 22.035581667, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000139001 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 247513, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.063568939, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000256 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 58473616, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 62882896 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-04-22T13:44:34.653 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2244, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 132.990139386, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.059264767 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2464, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.077306042, 2024-04-22T13:44:34.654 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000031374 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 319, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 319, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 319, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 49.095044725, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.153902961 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:34.655 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 113, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 118, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 38, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 80, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 63, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 3, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 4, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 2, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 485, 2024-04-22T13:44:34.656 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 472, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 641, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-04-22T13:44:34.657 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 6, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 641, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 229, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 22, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 207, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 412, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 4, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 133, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 3254, 2024-04-22T13:44:34.658 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 153037, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 152902, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 6, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 50 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 3, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 4, 2024-04-22T13:44:34.659 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 22, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 18, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 11, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 300, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 22, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 153574, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 153515, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 153515, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 59, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 1, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 247, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 246, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 246, 2024-04-22T13:44:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 1, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 1, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 1, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 495780410, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 496091138, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5145, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 1057.247704066, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.205490321 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-04-22T13:44:34.662 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 90, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 130, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 40, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 22, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 22, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 118, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 136, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 18, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 4, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 57, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 53, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 79904, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 231800 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-04-22T13:44:34.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 416, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 67, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 319, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 38, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 134, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 22, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.025983974, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001181089 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 145, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 6.893176894, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.047539150 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:34.664 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 67, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 6.994000768, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.104388071 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:34.665 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000385474, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000192737 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:34.666 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.667 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 13, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.010031350, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000771642 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 10, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 22.765788212, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 2.276578821 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.668 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 13, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 9.843655563, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.757204274 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000752002, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000376001 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.009032564, 2024-04-22T13:44:34.669 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001806512 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.003485710, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001742855 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.670 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.031093791, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.007773447 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 7, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.472071574, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.353153082 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.671 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000997492, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000997492 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.672 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.020275919, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002534489 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.673 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 18, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.024313438, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001350746 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.674 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 2, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 6, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 4, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 2, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 57, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 28, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 755, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:34.675 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:34.676 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:34.677 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 343877, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 138, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 163, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:34.678 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 351751, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 2826, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:34.679 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 9707, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 492420511, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 37, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 9707, 2024-04-22T13:44:34.680 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 9707, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 328.044400979, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.033794622 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 9707, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 10023 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 9707, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 686, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 9021, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-04-22T13:44:34.681 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 91, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 44, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 23, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 5146, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 166, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 3324, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 663, 2024-04-22T13:44:34.682 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 88, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-04-22T13:44:34.683 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 19, 2024-04-22T13:44:34.684 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 12, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 12, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 459, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 19 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-04-22T13:44:34.685 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 378 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 518, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 6, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 2578, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 18, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.686 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:34.687 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:34.687 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:34.687 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:34.687 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 167362, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 40764211, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 167362, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 167362, 2024-04-22T13:44:34.688 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 40764211, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:34.689 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 9707, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 492423193, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 9707, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 492423193, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.690 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 9707, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 9707, 2024-04-22T13:44:34.691 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 9707, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 9707, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 18, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 2682, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:34.692 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 18, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 1, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2682, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55655c4c2500": { 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:34.693 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 153574, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 491896834, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 153574, 2024-04-22T13:44:34.694 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 5145, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 491896834, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:34.695 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-04-22T13:44:35.472 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-22T13:44:35.473 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-22T13:44:35.473 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-22T13:44:35.473 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=13 2024-04-22T13:44:35.473 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=7 2024-04-22T13:44:35.473 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T13:44:35.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T13:44:35.716 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:44:35.755 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:35 smithi120 ceph-mon[31101]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:35.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:35 smithi132 ceph-mon[31531]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:35.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:35 smithi087 ceph-mon[27999]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 888, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 890, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 236082, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 282448, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.080766121, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.031841628, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.018654633, 2024-04-22T13:44:37.417 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.021762650, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 888, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.040658851, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000045786 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:37.418 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1045, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1031, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1824402, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 397761, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.122230607, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.035375067, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.053978213, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.036001450, 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:37.419 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1030, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.053122895, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000051575 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1636928, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 155440 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 885, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1068, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 298656, 2024-04-22T13:44:37.420 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 771970, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.117688409, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.038763138, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.028963063, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.037794578, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1066, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.073633805, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000069074 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.421 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 492448 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4215, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.131274607, 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000031144 2024-04-22T13:44:37.422 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-04-22T13:44:37.423 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 0, 2024-04-22T13:44:37.424 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-04-22T13:44:37.425 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:37.426 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-04-22T13:44:37.427 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-04-22T13:44:37.428 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-04-22T13:44:37.429 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 10, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-04-22T13:44:37.430 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4197187, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 4197046, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 10 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 3, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 3, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-04-22T13:44:37.431 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 1, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 1, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 52800, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215416 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-04-22T13:44:37.432 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.433 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.434 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.435 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-04-22T13:44:37.436 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.437 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.438 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.439 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.440 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.441 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.442 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-04-22T13:44:37.443 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 2, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 5, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 3, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 5, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 3996, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:37.444 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:37.445 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:37.446 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 189417, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 52, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 163, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 8512, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 13, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 2529, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 21, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 3, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 2529, 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-04-22T13:44:37.448 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2529, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 7.370807381, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002914514 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2529, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2530 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 2529, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 2529, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 1639, 2024-04-22T13:44:37.449 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 888, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 1, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-04-22T13:44:37.450 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-04-22T13:44:37.451 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 18, 2024-04-22T13:44:37.452 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 5, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 5, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-04-22T13:44:37.453 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:37.454 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2814, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 2088400, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2814, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2814, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2088400, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:37.455 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2529, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2888, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2529, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2888, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.456 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2529, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2529, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2529, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2529, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.457 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-04-22T13:44:37.458 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55daa19626e0": { 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:37.459 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:37.460 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-04-22T13:44:37.670 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:37 smithi120 ceph-mon[31101]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:37.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:37 smithi132 ceph-mon[31531]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:37.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:37 smithi087 ceph-mon[27999]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:38.241 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-22T13:44:38.241 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-22T13:44:38.241 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-22T13:44:38.241 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-22T13:44:38.241 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T13:44:38.241 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T13:44:38.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T13:44:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:44:39.911 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:39 smithi120 ceph-mon[31101]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:39.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:39 smithi132 ceph-mon[31531]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:39.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:39 smithi087 ceph-mon[27999]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:40.194 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-04-22T13:44:40.194 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:40.194 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 2550, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2552, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 724094, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 810907, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.196685680, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.061095393, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.047699134, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.073110283, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2550, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.077476055, 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000030382 2024-04-22T13:44:40.195 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 4, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 189, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 876, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 494314, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-04-22T13:44:40.196 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.011711454, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.007507138, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.001931645, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000052125, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 186, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.019497982, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000104827 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.197 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 493616 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 286, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 270, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1639644, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 154926, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 3, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.050437235, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.015950665, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.036298654, 2024-04-22T13:44:40.198 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.006609661, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 270, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.014775498, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000054724 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1639376, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 154608 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-04-22T13:44:40.199 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4295, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.105608763, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000024588 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.200 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 0, 2024-04-22T13:44:40.202 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-04-22T13:44:40.203 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T13:44:40.204 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.205 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:40.206 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-04-22T13:44:40.207 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 7, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-04-22T13:44:40.208 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4196347, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 4196206, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.209 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 7 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 3, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 3, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 1, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 1, 2024-04-22T13:44:40.210 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 49680, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215416 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-04-22T13:44:40.211 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.212 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.213 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.214 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-04-22T13:44:40.216 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.217 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.218 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.219 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.220 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.221 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-04-22T13:44:40.223 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.224 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 3, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 2, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 3, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 4646, 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:40.226 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:40.227 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:40.228 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 181308, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 48, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-04-22T13:44:40.229 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 163, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-04-22T13:44:40.230 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 8512, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 13, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-04-22T13:44:40.231 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 2553, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 21, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 3, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 2553, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2553, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.010947754, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000787680 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2553, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2554 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.232 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 2553, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 2553, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 1655, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 896, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-04-22T13:44:40.233 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 1, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:40.234 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-04-22T13:44:40.235 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-04-22T13:44:40.236 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 18, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 4, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 4, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-04-22T13:44:40.237 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.238 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2836, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 2091998, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2836, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2836, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2091998, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.239 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2553, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2048, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2553, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2048, 2024-04-22T13:44:40.240 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.241 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:40.242 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2553, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2553, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2553, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2553, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:40.243 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.244 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x5559834206e0": { 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-04-22T13:44:40.245 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:40.246 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-04-22T13:44:40.999 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-22T13:44:40.999 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-22T13:44:40.999 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-22T13:44:40.999 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-22T13:44:40.999 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T13:44:41.000 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T13:44:41.000 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T13:44:41.235 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T13:44:41.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:41 smithi120 ceph-mon[31101]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:41.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:41 smithi132 ceph-mon[31531]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:41.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:41 smithi087 ceph-mon[27999]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:42.918 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-04-22T13:44:42.918 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:42.918 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 0, 2024-04-22T13:44:42.918 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 0, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 0, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.001068516, 2024-04-22T13:44:42.919 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T13:44:42.920 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.000299139, 2024-04-22T13:44:42.920 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:42.920 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:42.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:42.920 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:42.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:42.922 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.922 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:42.922 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:42.922 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.922 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 260, 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 249, 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1600896, 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 153360, 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:42.923 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.051997094, 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.014796650, 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.039383133, 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.006786113, 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:42.924 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 248, 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.013476894, 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000054342 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:42.925 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:42.926 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:42.926 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.926 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1600896, 2024-04-22T13:44:42.926 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 153360 2024-04-22T13:44:42.926 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.927 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:42.927 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 6, 2024-04-22T13:44:42.927 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 192, 2024-04-22T13:44:42.927 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 3312, 2024-04-22T13:44:42.927 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 190000, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.013002581, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.008020140, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.002494395, 2024-04-22T13:44:42.928 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 190, 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.020149278, 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000106048 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.929 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:42.930 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:42.930 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:42.930 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:42.930 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.930 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 190000 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 0, 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-04-22T13:44:42.931 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:42.932 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:42.933 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:42.933 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:42.933 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:42.933 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:42.933 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:42.934 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:42.935 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:42.936 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:42.936 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:42.936 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:42.936 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:42.936 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-04-22T13:44:42.937 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 149047, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 31, 2024-04-22T13:44:42.938 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 2, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-04-22T13:44:42.941 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:42.942 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-04-22T13:44:42.942 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-04-22T13:44:42.942 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:42.942 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-04-22T13:44:42.942 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 262, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1564201, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 262, 2024-04-22T13:44:42.943 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 262, 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1564201, 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-04-22T13:44:42.944 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-04-22T13:44:42.945 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-04-22T13:44:42.945 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:42.945 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-04-22T13:44:42.945 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-04-22T13:44:43.516 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-22T13:44:43.516 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-22T13:44:43.516 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-22T13:44:43.517 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-22T13:44:43.517 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-22T13:44:43.517 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T13:44:43.517 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T13:44:43.769 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:44:43.890 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:43 smithi132 ceph-mon[31531]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:43.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:43 smithi120 ceph-mon[31101]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:43.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:43 smithi087 ceph-mon[27999]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:45.548 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-22T13:44:45.548 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 91, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 99, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 21944, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 492867, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 49, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 4, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.024256974, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.002185304, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.016384188, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000494168, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 98, 2024-04-22T13:44:45.549 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.015033032, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000153398 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 88, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000011249, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000127 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 9344, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 10224 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 1409, 2024-04-22T13:44:45.550 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 1348, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1934812, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 655389, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 7, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 6, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.144051157, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.052407418, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.097421556, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.012547542, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1347, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.061316372, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000045520 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:45.551 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2041, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000691535, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000338 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1649984, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 174432 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 52, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 293, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 13202, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 642406, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 20, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:45.552 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.028876707, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.012476884, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.013480930, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000130916, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 290, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.028022110, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000096627 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 118, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000026286, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000222 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.553 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 10368, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 521696 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 72, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.001847290, 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000025656 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.554 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000006468, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000003234 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 18, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 11, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 11, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.004252524, 2024-04-22T13:44:45.555 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000386593 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 7, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 11, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 14, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 14, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 14, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 0, 2024-04-22T13:44:45.556 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 2, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 2, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 2, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 48, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 43, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 2, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 1, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-04-22T13:44:45.557 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 22, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 10738141805, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 6, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T13:44:45.558 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 20, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 12, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 21, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 1 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:45.559 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 2, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-04-22T13:44:45.560 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_path": 11, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_inode": 55, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "evlrg": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 10, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 10, 2024-04-22T13:44:45.561 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 2, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 2, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segmjr": 2, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 4194304, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 4197187, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 0, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 9, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.014960964, 2024-04-22T13:44:45.562 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.001662329 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 0 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 17, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 17, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 0, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 15, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 16, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 1, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 14, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 14, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 0, 2024-04-22T13:44:45.563 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 2, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 9, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 7, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 53292, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 223608 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 22, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 0, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 18, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 33, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 56, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.564 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 7, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.003312383, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000473197 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.565 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 4, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000940141, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000235035 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.566 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.567 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.568 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.569 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.570 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.571 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-04-22T13:44:45.572 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.573 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.574 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-04-22T13:44:45.575 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 2, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 5, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 3, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 2, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 10, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 2, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 766, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:45.576 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:45.577 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:45.578 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 262471, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 85, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:45.579 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 163, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 68493, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 301, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-04-22T13:44:45.580 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 51, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 14007, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 28, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 51, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 51, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.233993618, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.004588110 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 51, 2024-04-22T13:44:45.581 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 79 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 51, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 51, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 10, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 9, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 7, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-04-22T13:44:45.582 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 6, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 20, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-04-22T13:44:45.583 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-04-22T13:44:45.584 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 103, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 18, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 8, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 8, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 25, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 0, 2024-04-22T13:44:45.585 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 2 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 1, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 66 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T13:44:45.586 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed_ops": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 1548, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 1808727, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 1548, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:45.587 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 1548, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1808727, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 51, 2024-04-22T13:44:45.588 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 14007, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 51, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 14007, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:45.589 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 51, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 51, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 51, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 51, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:45.590 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x56244b768500": { 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:45.591 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 10, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 2883, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 10, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 9, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 2883, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:45.592 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-22T13:44:45.918 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:45 smithi132 ceph-mon[31531]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:45.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:45 smithi120 ceph-mon[31101]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:45.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:45 smithi087 ceph-mon[27999]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:46.416 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-22T13:44:46.416 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-22T13:44:46.416 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-22T13:44:46.416 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-22T13:44:46.416 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T13:44:46.416 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T13:44:46.416 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T13:44:46.669 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:44:47.847 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:47 smithi132 ceph-mon[31531]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:47.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:47 smithi120 ceph-mon[31101]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:47.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:47 smithi087 ceph-mon[27999]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:48.478 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-22T13:44:48.478 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 848, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 847, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 54964114, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 311611, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 7, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.148999442, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.033095998, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.073333330, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.032469930, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 846, 2024-04-22T13:44:48.479 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.044343485, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000052415 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 208416, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 97584 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 1162, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 1275, 2024-04-22T13:44:48.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 102608165, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 658703, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 7, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 6, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.152094901, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.034028213, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.078991867, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.030738561, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1274, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.060409692, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000047417 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.481 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 290720 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 418, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 420, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 146940624, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 132931, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 6, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 5, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.119593404, 2024-04-22T13:44:48.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.009748185, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.091626437, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.014094360, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 418, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.020632375, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000049359 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T13:44:48.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 3464, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.104683156, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000030220 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.484 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 0, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 0, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.485 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 101, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 21, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 80, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 71, 2024-04-22T13:44:48.486 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 2, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-04-22T13:44:48.487 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 4, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 234, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 6, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T13:44:48.488 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 0, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 0 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-04-22T13:44:48.489 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:48.490 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_path": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_inode": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "evlrg": 0, 2024-04-22T13:44:48.491 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 78927, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 2, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segmjr": 39, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 495780410, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 496091138, 2024-04-22T13:44:48.492 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 496088060, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 78927 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 60, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 78, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 18, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 11, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 11, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 0, 2024-04-22T13:44:48.493 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 101, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 101, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 51364, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 215392 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 0, 2024-04-22T13:44:48.494 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 0, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.496 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.497 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.498 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-04-22T13:44:48.499 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.500 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.501 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.502 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.503 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.504 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.505 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 2, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 2, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-04-22T13:44:48.506 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 2080, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:48.507 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:48.508 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 240271, 2024-04-22T13:44:48.509 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 139, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 163, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:48.510 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 262138, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 2460, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 2248, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 26, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 4, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 2248, 2024-04-22T13:44:48.511 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2248, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 7.522781713, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.003346433 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2248, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 2249 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 2248, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 2248, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 0, 2024-04-22T13:44:48.512 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 1474, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 771, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:48.513 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 2, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-04-22T13:44:48.514 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-04-22T13:44:48.515 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 103, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 14, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 11, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 11, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 2, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 0 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-04-22T13:44:48.516 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 0 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed_ops": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 0, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.517 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 2424, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 304282814, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 2424, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2424, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 304282814, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.518 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 2248, 2024-04-22T13:44:48.519 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 303678478, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2248, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 303678478, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:48.520 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 2248, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 2248, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2248, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 2248, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:48.521 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.522 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x561f1c5726e0": { 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:48.523 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:48.524 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:48.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.524 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:48.524 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-22T13:44:49.365 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-22T13:44:49.366 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-22T13:44:49.366 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-22T13:44:49.366 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-22T13:44:49.366 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T13:44:49.366 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T13:44:49.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T13:44:49.609 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:44:49.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:49 smithi120 ceph-mon[31101]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:49.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:49 smithi132 ceph-mon[31531]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:49.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:49 smithi087 ceph-mon[27999]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 39, 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 221, 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 8867, 2024-04-22T13:44:51.444 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 434497, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 14, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 4, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.020322458, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.007353926, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.006306722, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000400678, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 215, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.052160974, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000242609 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.445 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 44, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000011543, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000262 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 2048, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 374480 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 931, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 1043, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1815769, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 506360, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 14, 2024-04-22T13:44:51.446 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 4, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.128594685, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.044250845, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.081888740, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.012583698, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1047, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.058038374, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000055433 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1292, 2024-04-22T13:44:51.447 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000442454, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000342 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1654160, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 161328 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 57, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 91, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 10023, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 610087, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 52, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 5, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.023095626, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.002963051, 2024-04-22T13:44:51.448 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.017854546, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000209614, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 88, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.010851605, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000123313 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 58, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000007797, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000134 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 2016, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 132128 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.449 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 48, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.001489932, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000031040 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-04-22T13:44:51.450 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000006463, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000003231 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 0, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 0, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.451 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 11, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 10, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 10, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 10, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 0, 2024-04-22T13:44:51.452 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 1, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 22, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 22, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-04-22T13:44:51.453 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 22, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 10738141805, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 6, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 0, 2024-04-22T13:44:51.454 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T13:44:51.455 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 0, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 0, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 0, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 0 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-04-22T13:44:51.456 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 0, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 0, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 0, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 1, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 0, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 0, 2024-04-22T13:44:51.457 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-04-22T13:44:51.458 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_path": 11, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_quiesce_inode": 11, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T13:44:51.459 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 0, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "evlrg": 0, 2024-04-22T13:44:51.460 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 7, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 7, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 2, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-04-22T13:44:51.461 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 2, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "segmjr": 2, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 4194304, 2024-04-22T13:44:51.462 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 4196347, 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 0, 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 6, 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.010068929, 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.001678154 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.463 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 0 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 13, 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 13, 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 0, 2024-04-22T13:44:51.464 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 11, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 12, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 1, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 10, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 10, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 0, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 0, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 0, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 0, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 55236, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 223608 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 0, 2024-04-22T13:44:51.465 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 0, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 0, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 18, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 11, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.466 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.467 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.468 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.469 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.470 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.471 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-04-22T13:44:51.472 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.473 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.474 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.475 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.476 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-04-22T13:44:51.477 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 1, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 3, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 2, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 1, 2024-04-22T13:44:51.478 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 766, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:51.479 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:51.480 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 238071, 2024-04-22T13:44:51.481 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 74, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 7840, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 163, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:51.482 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 50880, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 199, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 42, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 12345, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 33, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 42, 2024-04-22T13:44:51.483 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 42, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.277992435, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.006618867 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 42, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 64 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 42, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 0, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 42, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-04-22T13:44:51.484 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 10, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 6, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 7, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 6, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 0, 2024-04-22T13:44:51.485 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 20, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-04-22T13:44:51.486 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 103, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-04-22T13:44:51.487 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 18, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 9, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 14, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 5, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 14, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 0, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 1 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 1, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-04-22T13:44:51.488 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 22 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed_ops": 0, 2024-04-22T13:44:51.489 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 1023, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 1722723, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 1023, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:51.490 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 1023, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1722723, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:51.491 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 42, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 12345, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 42, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 12345, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.492 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 42, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 42, 2024-04-22T13:44:51.493 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 42, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 42, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-04-22T13:44:51.494 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x55d63fb18500": { 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:51.495 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 7, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 2043, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 7, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 6, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 2043, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.496 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:51.497 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-22T13:44:51.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:51 smithi132 ceph-mon[31531]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:51.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:51 smithi120 ceph-mon[31101]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:51.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:51 smithi087 ceph-mon[27999]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:52.419 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-22T13:44:52.420 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-22T13:44:52.420 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-22T13:44:52.420 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-22T13:44:52.420 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T13:44:52.420 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T13:44:52.421 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 176003e8-00ab-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T13:44:52.661 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T13:44:53.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:53 smithi132 ceph-mon[31531]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:53.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:53 smithi120 ceph-mon[31101]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:53.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:53 smithi087 ceph-mon[27999]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T13:44:54.478 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 264, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 253, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1601984, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 155920, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 4, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 0, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.052840066, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.015874252, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.038150078, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.007030542, 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:54.479 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 252, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.014495426, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000057521 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1601984, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 155920 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 2, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 194, 2024-04-22T13:44:54.480 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 320, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 190256, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 5, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 3, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.013309267, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.009202648, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.001403713, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 191, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.019067459, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000099829 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:54.481 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 190256 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 4, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 4, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 2992, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 448, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 2, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 1, 2024-04-22T13:44:54.482 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.000722168, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.000039333, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.000679070, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 3, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000682850, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000227616 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T13:44:54.483 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T13:44:54.484 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T13:44:54.485 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-04-22T13:44:54.486 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 149024, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 31, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-04-22T13:44:54.487 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 928, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 2, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 266, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 1564721, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-04-22T13:44:54.488 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 266, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 266, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1564721, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-22T13:44:54.489 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-22T13:44:55.221 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-22T13:44:55.221 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-22T13:44:55.221 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-22T13:44:55.221 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-22T13:44:55.221 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-22T13:44:55.222 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T13:44:55.234 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T13:44:55.234 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:44:55.235 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T13:44:55.256 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-04-22T13:44:55.256 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T13:44:55.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:44:55.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 -- 172.21.15.87:0/4101513569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 msgr2=0x7fe74c106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 --2- 172.21.15.87:0/4101513569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c106210 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fe744009cd0 tx=0x7fe74402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 -- 172.21.15.87:0/4101513569 shutdown_connections 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 --2- 172.21.15.87:0/4101513569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 0x7fe74c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 --2- 172.21.15.87:0/4101513569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 0x7fe74c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 --2- 172.21.15.87:0/4101513569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c106210 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 -- 172.21.15.87:0/4101513569 >> 172.21.15.87:0/4101513569 conn(0x7fe74c077e30 msgr2=0x7fe74c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 -- 172.21.15.87:0/4101513569 shutdown_connections 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.473+0000 7fe75350d640 1 -- 172.21.15.87:0/4101513569 wait complete. 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 Processor -- start 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 -- start start 2024-04-22T13:44:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 0x7fe74c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 0x7fe74c198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1110c0 con 0x7fe74c105e30 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c110f40 con 0x7fe74c101b60 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe75350d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c111240 con 0x7fe74c1011a0 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55000/0 (socket says 172.21.15.87:55000) 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 -- 172.21.15.87:0/1651681982 learned_addr learned my addr 172.21.15.87:0/1651681982 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe7512a9640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 0x7fe74c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe750aa8640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 0x7fe74c198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 -- 172.21.15.87:0/1651681982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 msgr2=0x7fe74c1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 0x7fe74c1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 -- 172.21.15.87:0/1651681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 msgr2=0x7fe74c198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 0x7fe74c198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.474+0000 7fe751aaa640 1 -- 172.21.15.87:0/1651681982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c19d6d0 con 0x7fe74c105e30 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe751aaa640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e970 tx=0x7fe73c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c00ccc0 con 0x7fe74c105e30 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c19d9c0 con 0x7fe74c105e30 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c00ce60 con 0x7fe74c105e30 2024-04-22T13:44:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c010430 con 0x7fe74c105e30 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.475+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1a5250 con 0x7fe74c105e30 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.476+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c102570 con 0x7fe74c105e30 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.477+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe73c004fb0 con 0x7fe74c105e30 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.477+0000 7fe74a7fc640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 0x7fe7340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.478+0000 7fe7512a9640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 0x7fe7340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.478+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fe73c0d0dd0 con 0x7fe74c105e30 2024-04-22T13:44:55.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.478+0000 7fe7512a9640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 0x7fe7340a61e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe744004af0 tx=0x7fe744005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:55.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.481+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe73c053ff0 con 0x7fe74c105e30 2024-04-22T13:44:55.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.661+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe74c068de0 con 0x7fe74c105e30 2024-04-22T13:44:55.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.663+0000 7fe74a7fc640 1 -- 172.21.15.87:0/1651681982 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23047 (secure 0 0 0) 0x7fe74c068de0 con 0x7fe74c105e30 2024-04-22T13:44:55.678 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:55.678 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":103,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:44:14.582679+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:44:55.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 msgr2=0x7fe7340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 0x7fe7340a61e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe744004af0 tx=0x7fe744005c50 comp rx=0 tx=0).stop 2024-04-22T13:44:55.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 msgr2=0x7fe74c19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e970 tx=0x7fe73c00ed70 comp rx=0 tx=0).stop 2024-04-22T13:44:55.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 shutdown_connections 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fe7340a3d20 0x7fe7340a61e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe74c105e30 0x7fe74c19cfa0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe74c101b60 0x7fe74c198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 --2- 172.21.15.87:0/1651681982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe74c1011a0 0x7fe74c1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 >> 172.21.15.87:0/1651681982 conn(0x7fe74c077e30 msgr2=0x7fe74c0694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 shutdown_connections 2024-04-22T13:44:55.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.665+0000 7fe75350d640 1 -- 172.21.15.87:0/1651681982 wait complete. 2024-04-22T13:44:55.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T13:25:00.762949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T13:29:44.513144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T13:29:45.638980+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T13:44:55.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T13:44:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.786+0000 7ff03f4c7640 1 -- 172.21.15.87:0/3264899914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff038103af0 msgr2=0x7ff03810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.786+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/3264899914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff038103af0 0x7ff03810bfd0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7ff028009900 tx=0x7ff02802bd60 comp rx=0 tx=0).stop 2024-04-22T13:44:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- 172.21.15.87:0/3264899914 shutdown_connections 2024-04-22T13:44:55.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/3264899914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff03810c510 0x7ff03810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/3264899914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff038103af0 0x7ff03810bfd0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/3264899914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff0381035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- 172.21.15.87:0/3264899914 >> 172.21.15.87:0/3264899914 conn(0x7ff038077f50 msgr2=0x7ff038078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- 172.21.15.87:0/3264899914 shutdown_connections 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- 172.21.15.87:0/3264899914 wait complete. 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 Processor -- start 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- start start 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff038198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff038103af0 0x7ff038198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff038112180 con 0x7ff03810c510 2024-04-22T13:44:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff038112000 con 0x7ff0381031d0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.787+0000 7ff03f4c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff038112300 con 0x7ff038103af0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55018/0 (socket says 172.21.15.87:55018) 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 -- 172.21.15.87:0/2974785425 learned_addr learned my addr 172.21.15.87:0/2974785425 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03ca62640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff038103af0 0x7ff038198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03d263640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff038198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 -- 172.21.15.87:0/2974785425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff038103af0 msgr2=0x7ff038198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff038103af0 0x7ff038198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 -- 172.21.15.87:0/2974785425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 msgr2=0x7ff038198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff038198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 -- 172.21.15.87:0/2974785425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03819d730 con 0x7ff03810c510 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03d263640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff038198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03da64640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7ff03000eaa0 tx=0x7ff03000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03000cdb0 con 0x7ff03810c510 2024-04-22T13:44:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff030005020 con 0x7ff03810c510 2024-04-22T13:44:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03819da20 con 0x7ff03810c510 2024-04-22T13:44:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0300116d0 con 0x7ff03810c510 2024-04-22T13:44:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.788+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0381a52b0 con 0x7ff03810c510 2024-04-22T13:44:55.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.789+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff038068d70 con 0x7ff03810c510 2024-04-22T13:44:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.790+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff0300051d0 con 0x7ff03810c510 2024-04-22T13:44:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.791+0000 7ff02e7fc640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 0x7ff0240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.791+0000 7ff03d263640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 0x7ff0240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.791+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7ff0300d0cd0 con 0x7ff03810c510 2024-04-22T13:44:55.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.791+0000 7ff03d263640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 0x7ff0240a61e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0340058c0 tx=0x7ff034005850 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:55.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.794+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff030016030 con 0x7ff03810c510 2024-04-22T13:44:55.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:55 smithi120 ceph-mon[31101]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:55.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:55 smithi132 ceph-mon[31531]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:55.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.969+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff03802d5c0 con 0x7ff03810c510 2024-04-22T13:44:55.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.970+0000 7ff02e7fc640 1 -- 172.21.15.87:0/2974785425 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7ff03009d3b0 con 0x7ff03810c510 2024-04-22T13:44:55.966 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-04-22T13:44:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 msgr2=0x7ff0240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 0x7ff0240a61e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0340058c0 tx=0x7ff034005850 comp rx=0 tx=0).stop 2024-04-22T13:44:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 msgr2=0x7ff03819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7ff03000eaa0 tx=0x7ff03000ef60 comp rx=0 tx=0).stop 2024-04-22T13:44:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 shutdown_connections 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff0240a3d20 0x7ff0240a61e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff03810c510 0x7ff03819d000 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff038103af0 0x7ff038198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 --2- 172.21.15.87:0/2974785425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0381031d0 0x7ff038198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 >> 172.21.15.87:0/2974785425 conn(0x7ff038077f50 msgr2=0x7ff0380fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 shutdown_connections 2024-04-22T13:44:55.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:55.972+0000 7ff03f4c7640 1 -- 172.21.15.87:0/2974785425 wait complete. 2024-04-22T13:44:55.978 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:55 smithi087 ceph-mon[27999]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T13:44:55.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T13:44:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- 172.21.15.87:0/995845282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c50102b40 msgr2=0x7f2c50106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 --2- 172.21.15.87:0/995845282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c50102b40 0x7f2c50106b80 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009900 tx=0x7f2c4c02bc30 comp rx=0 tx=0).stop 2024-04-22T13:44:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- 172.21.15.87:0/995845282 shutdown_connections 2024-04-22T13:44:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 --2- 172.21.15.87:0/995845282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c5010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 --2- 172.21.15.87:0/995845282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c50102b40 0x7f2c50106b80 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 --2- 172.21.15.87:0/995845282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 0x7f2c50102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- 172.21.15.87:0/995845282 >> 172.21.15.87:0/995845282 conn(0x7f2c500793a0 msgr2=0x7f2c500797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- 172.21.15.87:0/995845282 shutdown_connections 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- 172.21.15.87:0/995845282 wait complete. 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 Processor -- start 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.083+0000 7f2c57625640 1 -- start start 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 0x7f2c50198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c50198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f2c501121a0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50112020 con 0x7f2c50102190 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2c50112320 con 0x7f2c50107260 2024-04-22T13:44:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55030/0 (socket says 172.21.15.87:55030) 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 -- 172.21.15.87:0/3545072632 learned_addr learned my addr 172.21.15.87:0/3545072632 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c553c1640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 0x7f2c50198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c54bc0640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c50198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 -- 172.21.15.87:0/3545072632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 msgr2=0x7f2c50198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c50198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 -- 172.21.15.87:0/3545072632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 msgr2=0x7f2c50198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 0x7f2c50198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 -- 172.21.15.87:0/3545072632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5019dab0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c54bc0640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c50198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c55bc2640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800eb10 tx=0x7f2c4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004b20 con 0x7f2c5019cfc0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c48004cc0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5019dda0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48010670 con 0x7f2c5019cfc0 2024-04-22T13:44:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.084+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c501a5590 con 0x7f2c5019cfc0 2024-04-22T13:44:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.085+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c500691c0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.086+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2c48005020 con 0x7f2c5019cfc0 2024-04-22T13:44:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.087+0000 7f2c427fc640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 0x7f2c3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.087+0000 7f2c553c1640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 0x7f2c3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.087+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f2c480d0df0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.087+0000 7f2c553c1640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 0x7f2c3c0a61e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005e10 tx=0x7f2c44005da0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.092+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c4809d4d0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.267+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2c50102b40 con 0x7f2c5019cfc0 2024-04-22T13:44:56.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.268+0000 7f2c427fc640 1 -- 172.21.15.87:0/3545072632 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f2c4809cdd0 con 0x7f2c5019cfc0 2024-04-22T13:44:56.264 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.269+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 msgr2=0x7f2c3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.269+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 0x7f2c3c0a61e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005e10 tx=0x7f2c44005da0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.269+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 msgr2=0x7f2c5019d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.269+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800eb10 tx=0x7f2c4800c2d0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 shutdown_connections 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f2c3c0a3d20 0x7f2c3c0a61e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c5019cfc0 0x7f2c5019d470 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c50107260 0x7f2c50198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 --2- 172.21.15.87:0/3545072632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c50102190 0x7f2c50198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 >> 172.21.15.87:0/3545072632 conn(0x7f2c500793a0 msgr2=0x7f2c500ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 shutdown_connections 2024-04-22T13:44:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.270+0000 7f2c57625640 1 -- 172.21.15.87:0/3545072632 wait complete. 2024-04-22T13:44:56.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.381+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3414094548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294107250 msgr2=0x7fd29410ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.381+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3414094548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294107250 0x7fd29410ee20 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009ca0 tx=0x7fd28c02c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3414094548 shutdown_connections 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3414094548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294107250 0x7fd29410ee20 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3414094548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294102b30 0x7fd294106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3414094548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294102180 0x7fd294102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3414094548 >> 172.21.15.87:0/3414094548 conn(0x7fd2940fd9c0 msgr2=0x7fd2940ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3414094548 shutdown_connections 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3414094548 wait complete. 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 Processor -- start 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- start start 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 0x7fd29419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 0x7fd294198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fd294112290 con 0x7fd294102180 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd294112110 con 0x7fd294107250 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd29bd4a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd294112410 con 0x7fd294198670 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.382+0000 7fd299ae6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55040/0 (socket says 172.21.15.87:55040) 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 -- 172.21.15.87:0/3986075780 learned_addr learned my addr 172.21.15.87:0/3986075780 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:56.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd29a2e7640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 0x7fd294198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd2992e5640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 0x7fd29419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 -- 172.21.15.87:0/3986075780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 msgr2=0x7fd294198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 0x7fd294198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 -- 172.21.15.87:0/3986075780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 msgr2=0x7fd29419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 0x7fd29419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 -- 172.21.15.87:0/3986075780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd294199190 con 0x7fd294102180 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd2992e5640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 0x7fd29419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd29a2e7640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 0x7fd294198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd299ae6640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd29000b4a0 tx=0x7fd29000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290014070 con 0x7fd294102180 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd294199480 con 0x7fd294102180 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd290010ac0 con 0x7fd294102180 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290017c30 con 0x7fd294102180 2024-04-22T13:44:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.383+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2941a5680 con 0x7fd294102180 2024-04-22T13:44:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.384+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd294068d70 con 0x7fd294102180 2024-04-22T13:44:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.385+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd290017070 con 0x7fd294102180 2024-04-22T13:44:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.386+0000 7fd28affd640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 0x7fd2800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.386+0000 7fd2992e5640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 0x7fd2800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.386+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fd2900cfea0 con 0x7fd294102180 2024-04-22T13:44:56.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.386+0000 7fd2992e5640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 0x7fd2800a61e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2940fdee0 tx=0x7fd284008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.389+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd29009c690 con 0x7fd294102180 2024-04-22T13:44:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.567+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd294102b30 con 0x7fd294102180 2024-04-22T13:44:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.567+0000 7fd28affd640 1 -- 172.21.15.87:0/3986075780 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7fd29009bf90 con 0x7fd294102180 2024-04-22T13:44:56.563 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 64 2024-04-22T13:44:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 msgr2=0x7fd2800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 0x7fd2800a61e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2940fdee0 tx=0x7fd284008040 comp rx=0 tx=0).stop 2024-04-22T13:44:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 msgr2=0x7fd29419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd29000b4a0 tx=0x7fd29000b960 comp rx=0 tx=0).stop 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 shutdown_connections 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fd2800a3d20 0x7fd2800a61e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd294198670 0x7fd294198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd294107250 0x7fd29419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 --2- 172.21.15.87:0/3986075780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd294102180 0x7fd29419e470 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 >> 172.21.15.87:0/3986075780 conn(0x7fd2940fd9c0 msgr2=0x7fd2940fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 shutdown_connections 2024-04-22T13:44:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.569+0000 7fd29bd4a640 1 -- 172.21.15.87:0/3986075780 wait complete. 2024-04-22T13:44:56.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T13:44:56.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 -- 172.21.15.87:0/575899121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794102190 msgr2=0x7f1794102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 --2- 172.21.15.87:0/575899121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794102190 0x7f1794102570 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f1784009ca0 tx=0x7f178402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 -- 172.21.15.87:0/575899121 shutdown_connections 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 --2- 172.21.15.87:0/575899121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794107260 0x7f179410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 --2- 172.21.15.87:0/575899121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1794102b40 0x7f1794106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 --2- 172.21.15.87:0/575899121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794102190 0x7f1794102570 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 -- 172.21.15.87:0/575899121 >> 172.21.15.87:0/575899121 conn(0x7f17940793a0 msgr2=0x7f17940797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 -- 172.21.15.87:0/575899121 shutdown_connections 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 -- 172.21.15.87:0/575899121 wait complete. 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.680+0000 7f1799e14640 1 Processor -- start 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 -- start start 2024-04-22T13:44:56.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 0x7f179419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 0x7f1794198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f17941121d0 con 0x7f1794107260 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1794112050 con 0x7f17941985d0 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1799e14640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1794112350 con 0x7f1794102190 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55048/0 (socket says 172.21.15.87:55048) 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 -- 172.21.15.87:0/127625780 learned_addr learned my addr 172.21.15.87:0/127625780 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f17937fe640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 0x7f179419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1793fff640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 0x7f1794198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 -- 172.21.15.87:0/127625780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 msgr2=0x7f179419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 0x7f179419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 -- 172.21.15.87:0/127625780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 msgr2=0x7f1794198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 0x7f1794198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 -- 172.21.15.87:0/127625780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17941990f0 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1793fff640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 0x7f1794198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f17937fe640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 0x7f179419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.681+0000 7f1792ffd640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1788005fd0 tx=0x7f178800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.682+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788017070 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.682+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17941993e0 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.682+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1788009c30 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.682+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788003d30 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.682+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17941a5630 con 0x7f1794107260 2024-04-22T13:44:56.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.683+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17940691c0 con 0x7f1794107260 2024-04-22T13:44:56.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.684+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1788003ed0 con 0x7f1794107260 2024-04-22T13:44:56.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.684+0000 7f1790ff9640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 0x7f177c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:56.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.684+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f17880d04c0 con 0x7f1794107260 2024-04-22T13:44:56.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.684+0000 7f17937fe640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 0x7f177c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:56.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.685+0000 7f17937fe640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 0x7f177c0a61e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1784009a60 tx=0x7f1784005d80 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:56.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.689+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1788020030 con 0x7f1794107260 2024-04-22T13:44:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.877+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1794102b40 con 0x7f1794107260 2024-04-22T13:44:56.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.877+0000 7f1790ff9640 1 -- 172.21.15.87:0/127625780 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) ==== 81+0+937 (secure 0 0 0) 0x7f1788009de0 con 0x7f1794107260 2024-04-22T13:44:56.873 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/1019204087 2024-04-23T13:22:36.668187+0000 2024-04-22T13:44:56.873 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/837668825 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.873 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/3462524867 2024-04-23T13:22:36.668187+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/3207002517 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6801/1421536657 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/2538396844 2024-04-23T13:22:36.668187+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6801/35245616 2024-04-23T13:22:36.668187+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6800/4093217616 2024-04-23T13:22:10.831107+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/4211868253 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/343208246 2024-04-23T13:22:10.831107+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6801/4093217616 2024-04-23T13:22:10.831107+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/194272199 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/73776732 2024-04-23T13:22:10.831107+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:0/1882698965 2024-04-23T13:22:10.831107+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6800/1421536657 2024-04-23T13:31:52.016434+0000 2024-04-22T13:44:56.874 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87:6800/35245616 2024-04-23T13:22:36.668187+0000 2024-04-22T13:44:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:listed 16 entries 2024-04-22T13:44:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 msgr2=0x7f177c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 0x7f177c0a61e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1784009a60 tx=0x7f1784005d80 comp rx=0 tx=0).stop 2024-04-22T13:44:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 msgr2=0x7f179419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:56.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f1788005fd0 tx=0x7f178800dde0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 shutdown_connections 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f177c0a3d20 0x7f177c0a61e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17941985d0 0x7f1794198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1794107260 0x7f179419e960 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 --2- 172.21.15.87:0/127625780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1794102190 0x7f179419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 >> 172.21.15.87:0/127625780 conn(0x7f17940793a0 msgr2=0x7f1794100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 shutdown_connections 2024-04-22T13:44:56.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:56.879+0000 7f1799e14640 1 -- 172.21.15.87:0/127625780 wait complete. 2024-04-22T13:44:56.885 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/1651681982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:56.885 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2974785425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.885 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3545072632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.885 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:56 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3986075780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.886 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T13:44:56.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:44:56.886 DEBUG:teuthology.orchestra.run.smithi087:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T13:44:56.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/1651681982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:56.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2974785425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3545072632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:56 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3986075780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.924 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:44:56.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:44:56.924 DEBUG:teuthology.orchestra.run.smithi087:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T13:44:56.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/1651681982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:56.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2974785425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3545072632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:56 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3986075780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T13:44:56.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:44:57.000 DEBUG:teuthology.orchestra.run.smithi087:> ip netns list 2024-04-22T13:44:57.057 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T13:44:57.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:44:57.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T13:44:57.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T13:44:57.123 DEBUG:teuthology.orchestra.run.smithi087:> sudo ip link delete ceph-brx 2024-04-22T13:44:57.224 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T13:44:57.235 INFO:tasks.cephadm:Teardown begin 2024-04-22T13:44:57.236 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T13:44:57.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:44:57.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- 172.21.15.87:0/3513185581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c069aa0 msgr2=0x7fbf9c10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3513185581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c10ede0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009ca0 tx=0x7fbf9402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- 172.21.15.87:0/3513185581 shutdown_connections 2024-04-22T13:44:57.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3513185581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c10ede0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3513185581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c0690e0 0x7fbf9c069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3513185581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c106f50 0x7fbf9c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- 172.21.15.87:0/3513185581 >> 172.21.15.87:0/3513185581 conn(0x7fbf9c077e30 msgr2=0x7fbf9c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- 172.21.15.87:0/3513185581 shutdown_connections 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- 172.21.15.87:0/3513185581 wait complete. 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 Processor -- start 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.339+0000 7fbfa272b640 1 -- start start 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c0690e0 0x7fbf9c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1121f0 con 0x7fbf9c106f50 2024-04-22T13:44:57.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c112070 con 0x7fbf9c0690e0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c112370 con 0x7fbf9c069aa0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55076/0 (socket says 172.21.15.87:55076) 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 -- 172.21.15.87:0/3282149959 learned_addr learned my addr 172.21.15.87:0/3282149959 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbf9bfff640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c0690e0 0x7fbf9c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbf9b7fe640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 -- 172.21.15.87:0/3282149959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c069aa0 msgr2=0x7fbf9c198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 -- 172.21.15.87:0/3282149959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c0690e0 msgr2=0x7fbf9c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c0690e0 0x7fbf9c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 -- 172.21.15.87:0/3282149959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c19d660 con 0x7fbf9c106f50 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa0cc8640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fbf9402c700 tx=0x7fbf94035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94039040 con 0x7fbf9c106f50 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c19d8f0 con 0x7fbf9c106f50 2024-04-22T13:44:57.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf94044070 con 0x7fbf9c106f50 2024-04-22T13:44:57.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf940042a0 con 0x7fbf9c106f50 2024-04-22T13:44:57.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.340+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c1a51e0 con 0x7fbf9c106f50 2024-04-22T13:44:57.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.341+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9c068de0 con 0x7fbf9c106f50 2024-04-22T13:44:57.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.343+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf94004440 con 0x7fbf9c106f50 2024-04-22T13:44:57.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.343+0000 7fbf997fa640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 0x7fbf840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.343+0000 7fbf9bfff640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 0x7fbf840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.343+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fbf940f33b0 con 0x7fbf9c106f50 2024-04-22T13:44:57.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.344+0000 7fbf9bfff640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 0x7fbf840a61e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009890 tx=0x7fbf8c008040 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:57.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.347+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf940bfc60 con 0x7fbf9c106f50 2024-04-22T13:44:57.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.561+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf9c10a4a0 con 0x7fbf9c106f50 2024-04-22T13:44:57.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.562+0000 7fbf997fa640 1 -- 172.21.15.87:0/3282149959 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7fbf940bf560 con 0x7fbf9c106f50 2024-04-22T13:44:57.558 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:57.558 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:44:57.559 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:44:57.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 msgr2=0x7fbf840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 0x7fbf840a61e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009890 tx=0x7fbf8c008040 comp rx=0 tx=0).stop 2024-04-22T13:44:57.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 msgr2=0x7fbf9c19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fbf9402c700 tx=0x7fbf94035d60 comp rx=0 tx=0).stop 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 shutdown_connections 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fbf840a3d20 0x7fbf840a61e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbf9c106f50 0x7fbf9c19cf30 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf9c069aa0 0x7fbf9c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 --2- 172.21.15.87:0/3282149959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9c0690e0 0x7fbf9c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 >> 172.21.15.87:0/3282149959 conn(0x7fbf9c077e30 msgr2=0x7fbf9c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 shutdown_connections 2024-04-22T13:44:57.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.564+0000 7fbfa272b640 1 -- 172.21.15.87:0/3282149959 wait complete. 2024-04-22T13:44:57.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T13:44:57.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 -- 172.21.15.87:0/885008858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc108030 msgr2=0x7f3ddc108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 --2- 172.21.15.87:0/885008858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc108030 0x7f3ddc108410 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009900 tx=0x7f3dc402bcd0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 -- 172.21.15.87:0/885008858 shutdown_connections 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 --2- 172.21.15.87:0/885008858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc06b040 0x7f3ddc10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 --2- 172.21.15.87:0/885008858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06a680 0x7f3ddc06ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 --2- 172.21.15.87:0/885008858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc108030 0x7f3ddc108410 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 -- 172.21.15.87:0/885008858 >> 172.21.15.87:0/885008858 conn(0x7f3ddc077e30 msgr2=0x7f3ddc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 -- 172.21.15.87:0/885008858 shutdown_connections 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.674+0000 7f3de29b6640 1 -- 172.21.15.87:0/885008858 wait complete. 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 Processor -- start 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 -- start start 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06b040 0x7f3ddc1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 0x7f3ddc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc112150 con 0x7f3ddc06a680 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc111fd0 con 0x7f3ddc06b040 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de29b6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc1122d0 con 0x7f3ddc108030 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55092/0 (socket says 172.21.15.87:55092) 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 -- 172.21.15.87:0/2110739923 learned_addr learned my addr 172.21.15.87:0/2110739923 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddb7fe640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06b040 0x7f3ddc1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de0f53640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 0x7f3ddc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 -- 172.21.15.87:0/2110739923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 msgr2=0x7f3ddc19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 0x7f3ddc19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 -- 172.21.15.87:0/2110739923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06b040 msgr2=0x7f3ddc1a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06b040 0x7f3ddc1a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3ddbfff640 1 -- 172.21.15.87:0/2110739923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc19afb0 con 0x7f3ddc06a680 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.675+0000 7f3de0f53640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 0x7f3ddc19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3ddbfff640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4005940 tx=0x7f3dc402d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4043020 con 0x7f3ddc06a680 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ddc19b240 con 0x7f3ddc06a680 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dc402dc90 con 0x7f3ddc06a680 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4003ab0 con 0x7f3ddc06a680 2024-04-22T13:44:57.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.676+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ddc1a75b0 con 0x7f3ddc06a680 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.677+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ddc103680 con 0x7f3ddc06a680 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.678+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3dc4046020 con 0x7f3ddc06a680 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.678+0000 7f3dd97fa640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 0x7f3dc80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.678+0000 7f3ddb7fe640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 0x7f3dc80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.678+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7f3dc40f27f0 con 0x7f3ddc06a680 2024-04-22T13:44:57.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.679+0000 7f3ddb7fe640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 0x7f3dc80a61e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc19bf40 tx=0x7f3dcc00b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:57.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.683+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3dc404d020 con 0x7f3ddc06a680 2024-04-22T13:44:57.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.896+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ddc1a7780 con 0x7f3ddc06a680 2024-04-22T13:44:57.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.897+0000 7f3dd97fa640 1 -- 172.21.15.87:0/2110739923 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11869 (secure 0 0 0) 0x7f3dc40bf100 con 0x7f3ddc06a680 2024-04-22T13:44:57.893 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:57.893 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6835/915813894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6834","nonce":915813894},{"type":"v1","addr":"172.21.15.87:6835","nonce":915813894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6837/1472471745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6836","nonce":1472471745},{"type":"v1","addr":"172.21.15.87:6837","nonce":1472471745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6839/2616185101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6838","nonce":2616185101},{"type":"v1","addr":"172.21.15.87:6839","nonce":2616185101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.87:6841/982852061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6840","nonce":982852061},{"type":"v1","addr":"172.21.15.87:6841","nonce":982852061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24383,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/820427937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":820427937},{"type":"v1","addr":"172.21.15.120:6833","nonce":820427937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/1335303979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":1335303979},{"type":"v1","addr":"172.21.15.132:6833","nonce":1335303979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T13:29:46.666137+0000","modified":"2024-04-22T13:35:24.206458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24407":{"gid":24407,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1829745193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1829745193},{"type":"v1","addr":"172.21.15.120:6835","nonce":1829745193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6837/3391616456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3391616456},{"type":"v1","addr":"172.21.15.120:6837","nonce":3391616456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.132:6837/453938062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":453938062},{"type":"v1","addr":"172.21.15.132:6837","nonce":453938062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.132:6839/2904643536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2904643536},{"type":"v1","addr":"172.21.15.132:6839","nonce":2904643536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6839/2692231328","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":2692231328},{"type":"v1","addr":"172.21.15.120:6839","nonce":2692231328}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25897":{"gid":25897,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/666537819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":666537819},{"type":"v1","addr":"172.21.15.132:6835","nonce":666537819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24460,24475,24482]},"id":1}]} 2024-04-22T13:44:57.894 INFO:teuthology.orchestra.run.smithi087.stderr:dumped fsmap epoch 23 2024-04-22T13:44:57.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 msgr2=0x7f3dc80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 0x7f3dc80a61e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc19bf40 tx=0x7f3dcc00b390 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 msgr2=0x7f3ddc1a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4005940 tx=0x7f3dc402d890 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 shutdown_connections 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f3dc80a3d20 0x7f3dc80a61e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ddc108030 0x7f3ddc19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ddc06b040 0x7f3ddc1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 --2- 172.21.15.87:0/2110739923 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ddc06a680 0x7f3ddc1a0630 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 >> 172.21.15.87:0/2110739923 conn(0x7f3ddc077e30 msgr2=0x7f3ddc10d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 shutdown_connections 2024-04-22T13:44:57.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:57.899+0000 7f3de29b6640 1 -- 172.21.15.87:0/2110739923 wait complete. 2024-04-22T13:44:57.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:57 smithi087 ceph-mon[27999]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:57.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:57 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/127625780' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T13:44:57.905 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:57 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3282149959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:57.906 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T13:44:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:57 smithi120 ceph-mon[31101]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:57 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/127625780' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T13:44:57.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:57 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3282149959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:57.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:57 smithi132 ceph-mon[31531]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T13:44:57.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:57 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/127625780' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T13:44:57.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:57 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3282149959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 -- 172.21.15.87:0/577287871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc918107270 msgr2=0x7fc91810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 --2- 172.21.15.87:0/577287871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc918107270 0x7fc91810ee40 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fc908009900 tx=0x7fc90802bc30 comp rx=0 tx=0).stop 2024-04-22T13:44:58.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 -- 172.21.15.87:0/577287871 shutdown_connections 2024-04-22T13:44:58.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 --2- 172.21.15.87:0/577287871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc918107270 0x7fc91810ee40 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 --2- 172.21.15.87:0/577287871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918102b50 0x7fc918106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 --2- 172.21.15.87:0/577287871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9181021a0 0x7fc918102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 -- 172.21.15.87:0/577287871 >> 172.21.15.87:0/577287871 conn(0x7fc9180793a0 msgr2=0x7fc9180797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 -- 172.21.15.87:0/577287871 shutdown_connections 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 -- 172.21.15.87:0/577287871 wait complete. 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.008+0000 7fc91d006640 1 Processor -- start 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 -- start start 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918107270 0x7fc918198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc91819cf90 0x7fc91819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fc9181121e0 con 0x7fc9181021a0 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc918112060 con 0x7fc918107270 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91d006640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc918112360 con 0x7fc91819cf90 2024-04-22T13:44:58.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55118/0 (socket says 172.21.15.87:55118) 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 -- 172.21.15.87:0/2550706500 learned_addr learned my addr 172.21.15.87:0/2550706500 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91659c640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918107270 0x7fc918198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc91759e640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc91819cf90 0x7fc91819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 -- 172.21.15.87:0/2550706500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc91819cf90 msgr2=0x7fc91819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc91819cf90 0x7fc91819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 -- 172.21.15.87:0/2550706500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918107270 msgr2=0x7fc918198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918107270 0x7fc918198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 -- 172.21.15.87:0/2550706500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91819da80 con 0x7fc9181021a0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.009+0000 7fc916d9d640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fc91000c970 tx=0x7fc91000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.010+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910007bb0 con 0x7fc9181021a0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.010+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc910007d50 con 0x7fc9181021a0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.010+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91819dd70 con 0x7fc9181021a0 2024-04-22T13:44:58.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.010+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91000f450 con 0x7fc9181021a0 2024-04-22T13:44:58.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.010+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9181a5510 con 0x7fc9181021a0 2024-04-22T13:44:58.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.011+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc918069230 con 0x7fc9181021a0 2024-04-22T13:44:58.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.012+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc910004f20 con 0x7fc9181021a0 2024-04-22T13:44:58.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.012+0000 7fc8fffff640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 0x7fc9000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.012+0000 7fc91659c640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 0x7fc9000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.013+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7fc9100d0490 con 0x7fc9181021a0 2024-04-22T13:44:58.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.013+0000 7fc91659c640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 0x7fc9000a61e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc918101820 tx=0x7fc90c009500 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.017+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc910014030 con 0x7fc9181021a0 2024-04-22T13:44:58.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.199+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc91810a560 con 0x7fc9181021a0 2024-04-22T13:44:58.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.200+0000 7fc8fffff640 1 -- 172.21.15.87:0/2550706500 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23047 (secure 0 0 0) 0x7fc91009c520 con 0x7fc9181021a0 2024-04-22T13:44:58.197 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-04-22T13:44:58.197 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":103,"fsid":"176003e8-00ab-11ef-bc93-c7b262605968","created":"2024-04-22T13:21:43.838276+0000","modified":"2024-04-22T13:44:14.582679+0000","last_up_change":"2024-04-22T13:28:52.803479+0000","last_in_change":"2024-04-22T13:28:32.270781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T13:25:00.762949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T13:29:44.513144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T13:29:45.638980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48d1248e-5bb0-4829-a035-88f1d8b049d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6803","nonce":4104861709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6805","nonce":4104861709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6809","nonce":4104861709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":4104861709},{"type":"v1","addr":"172.21.15.87:6807","nonce":4104861709}]},"public_addr":"172.21.15.87:6803/4104861709","cluster_addr":"172.21.15.87:6805/4104861709","heartbeat_back_addr":"172.21.15.87:6809/4104861709","heartbeat_front_addr":"172.21.15.87:6807/4104861709","state":["exists","up"]},{"osd":1,"uuid":"b96dba19-35ae-4b99-9a8d-43a623679f0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6801","nonce":3769829822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6803","nonce":3769829822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6807","nonce":3769829822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3769829822},{"type":"v1","addr":"172.21.15.120:6805","nonce":3769829822}]},"public_addr":"172.21.15.120:6801/3769829822","cluster_addr":"172.21.15.120:6803/3769829822","heartbeat_back_addr":"172.21.15.120:6807/3769829822","heartbeat_front_addr":"172.21.15.120:6805/3769829822","state":["exists","up"]},{"osd":2,"uuid":"02678f49-713f-40a2-8cc6-426c50c594a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6801","nonce":2448340418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6803","nonce":2448340418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6807","nonce":2448340418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":2448340418},{"type":"v1","addr":"172.21.15.132:6805","nonce":2448340418}]},"public_addr":"172.21.15.132:6801/2448340418","cluster_addr":"172.21.15.132:6803/2448340418","heartbeat_back_addr":"172.21.15.132:6807/2448340418","heartbeat_front_addr":"172.21.15.132:6805/2448340418","state":["exists","up"]},{"osd":3,"uuid":"cc5b62ee-b979-43d0-8f66-1c4183042bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6811","nonce":2474923327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6813","nonce":2474923327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6817","nonce":2474923327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2474923327},{"type":"v1","addr":"172.21.15.87:6815","nonce":2474923327}]},"public_addr":"172.21.15.87:6811/2474923327","cluster_addr":"172.21.15.87:6813/2474923327","heartbeat_back_addr":"172.21.15.87:6817/2474923327","heartbeat_front_addr":"172.21.15.87:6815/2474923327","state":["exists","up"]},{"osd":4,"uuid":"dd7d7e47-595c-4f2b-846c-0030654a9954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6809","nonce":172450130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6811","nonce":172450130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6815","nonce":172450130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":172450130},{"type":"v1","addr":"172.21.15.120:6813","nonce":172450130}]},"public_addr":"172.21.15.120:6809/172450130","cluster_addr":"172.21.15.120:6811/172450130","heartbeat_back_addr":"172.21.15.120:6815/172450130","heartbeat_front_addr":"172.21.15.120:6813/172450130","state":["exists","up"]},{"osd":5,"uuid":"1c76c783-6c30-48d4-b20b-c4ef76b03f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6809","nonce":1166176702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6811","nonce":1166176702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6815","nonce":1166176702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1166176702},{"type":"v1","addr":"172.21.15.132:6813","nonce":1166176702}]},"public_addr":"172.21.15.132:6809/1166176702","cluster_addr":"172.21.15.132:6811/1166176702","heartbeat_back_addr":"172.21.15.132:6815/1166176702","heartbeat_front_addr":"172.21.15.132:6813/1166176702","state":["exists","up"]},{"osd":6,"uuid":"9d20c5d7-b08d-4a48-8694-1d2faf0a1651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6819","nonce":3192057229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6821","nonce":3192057229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6825","nonce":3192057229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":3192057229},{"type":"v1","addr":"172.21.15.87:6823","nonce":3192057229}]},"public_addr":"172.21.15.87:6819/3192057229","cluster_addr":"172.21.15.87:6821/3192057229","heartbeat_back_addr":"172.21.15.87:6825/3192057229","heartbeat_front_addr":"172.21.15.87:6823/3192057229","state":["exists","up"]},{"osd":7,"uuid":"086e7b21-8587-47e5-a1c3-4cb514d43975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6817","nonce":1140009169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6819","nonce":1140009169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6823","nonce":1140009169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1140009169},{"type":"v1","addr":"172.21.15.120:6821","nonce":1140009169}]},"public_addr":"172.21.15.120:6817/1140009169","cluster_addr":"172.21.15.120:6819/1140009169","heartbeat_back_addr":"172.21.15.120:6823/1140009169","heartbeat_front_addr":"172.21.15.120:6821/1140009169","state":["exists","up"]},{"osd":8,"uuid":"0d2acda1-2a71-452c-acc3-5fbaf491f53c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6817","nonce":1871032509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6819","nonce":1871032509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6823","nonce":1871032509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1871032509},{"type":"v1","addr":"172.21.15.132:6821","nonce":1871032509}]},"public_addr":"172.21.15.132:6817/1871032509","cluster_addr":"172.21.15.132:6819/1871032509","heartbeat_back_addr":"172.21.15.132:6823/1871032509","heartbeat_front_addr":"172.21.15.132:6821/1871032509","state":["exists","up"]},{"osd":9,"uuid":"07f9c4fd-b4f0-48db-b5de-05dd6c55b890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6827","nonce":2036314844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6828","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6829","nonce":2036314844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6832","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6833","nonce":2036314844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6830","nonce":2036314844},{"type":"v1","addr":"172.21.15.87:6831","nonce":2036314844}]},"public_addr":"172.21.15.87:6827/2036314844","cluster_addr":"172.21.15.87:6829/2036314844","heartbeat_back_addr":"172.21.15.87:6833/2036314844","heartbeat_front_addr":"172.21.15.87:6831/2036314844","state":["exists","up"]},{"osd":10,"uuid":"a596e14d-82e2-4b4c-8af0-21f73df9b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6825","nonce":2160182398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6827","nonce":2160182398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6831","nonce":2160182398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":2160182398},{"type":"v1","addr":"172.21.15.120:6829","nonce":2160182398}]},"public_addr":"172.21.15.120:6825/2160182398","cluster_addr":"172.21.15.120:6827/2160182398","heartbeat_back_addr":"172.21.15.120:6831/2160182398","heartbeat_front_addr":"172.21.15.120:6829/2160182398","state":["exists","up"]},{"osd":11,"uuid":"c8cbc34d-da2e-48c0-a354-f30062b39ade","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6825","nonce":3735146249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6827","nonce":3735146249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6831","nonce":3735146249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3735146249},{"type":"v1","addr":"172.21.15.132:6829","nonce":3735146249}]},"public_addr":"172.21.15.132:6825/3735146249","cluster_addr":"172.21.15.132:6827/3735146249","heartbeat_back_addr":"172.21.15.132:6831/3735146249","heartbeat_front_addr":"172.21.15.132:6829/3735146249","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:11.110917+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:34.426471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:24:58.254454+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:23.166572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:25:48.424696+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:13.465153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:26:39.684516+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:05.507003+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:31.661495+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:27:58.698566+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:24.618675+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T13:28:51.001092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/1019204087":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/837668825":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/3462524867":"2024-04-23T13:22:36.668187+0000","172.21.15.87:0/3207002517":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6801/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/2538396844":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6801/35245616":"2024-04-23T13:22:36.668187+0000","172.21.15.87:6800/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/4211868253":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/343208246":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6801/4093217616":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/194272199":"2024-04-23T13:31:52.016434+0000","172.21.15.87:0/73776732":"2024-04-23T13:22:10.831107+0000","172.21.15.87:0/1882698965":"2024-04-23T13:22:10.831107+0000","172.21.15.87:6800/1421536657":"2024-04-23T13:31:52.016434+0000","172.21.15.87:6800/35245616":"2024-04-23T13:22:36.668187+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T13:44:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 msgr2=0x7fc9000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 0x7fc9000a61e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc918101820 tx=0x7fc90c009500 comp rx=0 tx=0).stop 2024-04-22T13:44:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 msgr2=0x7fc918198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fc91000c970 tx=0x7fc91000ce30 comp rx=0 tx=0).stop 2024-04-22T13:44:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 shutdown_connections 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fc9000a3d20 0x7fc9000a61e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc91819cf90 0x7fc91819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc918107270 0x7fc918198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 --2- 172.21.15.87:0/2550706500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9181021a0 0x7fc918198720 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 >> 172.21.15.87:0/2550706500 conn(0x7fc9180793a0 msgr2=0x7fc9180ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 shutdown_connections 2024-04-22T13:44:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.203+0000 7fc91d006640 1 -- 172.21.15.87:0/2550706500 wait complete. 2024-04-22T13:44:58.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T13:44:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 -- 172.21.15.87:0/791147452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c1010a0 msgr2=0x7ff52c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 --2- 172.21.15.87:0/791147452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c1010a0 0x7ff52c101480 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff524009ca0 tx=0x7ff52402c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 -- 172.21.15.87:0/791147452 shutdown_connections 2024-04-22T13:44:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 --2- 172.21.15.87:0/791147452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c106170 0x7ff52c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 --2- 172.21.15.87:0/791147452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c101a50 0x7ff52c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 --2- 172.21.15.87:0/791147452 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c1010a0 0x7ff52c101480 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 -- 172.21.15.87:0/791147452 >> 172.21.15.87:0/791147452 conn(0x7ff52c077eb0 msgr2=0x7ff52c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 -- 172.21.15.87:0/791147452 shutdown_connections 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.313+0000 7ff534077640 1 -- 172.21.15.87:0/791147452 wait complete. 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 Processor -- start 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 -- start start 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 0x7ff52c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c198590 0x7ff52c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1110b0 con 0x7ff52c106170 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c110f30 con 0x7ff52c198590 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff534077640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c111230 con 0x7ff52c1010a0 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55138/0 (socket says 172.21.15.87:55138) 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 -- 172.21.15.87:0/3900230521 learned_addr learned my addr 172.21.15.87:0/3900230521 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff532614640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c198590 0x7ff52c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531e13640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 0x7ff52c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 -- 172.21.15.87:0/3900230521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 msgr2=0x7ff52c19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 0x7ff52c19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 -- 172.21.15.87:0/3900230521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c198590 msgr2=0x7ff52c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c198590 0x7ff52c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 -- 172.21.15.87:0/3900230521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c1990b0 con 0x7ff52c106170 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531e13640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 0x7ff52c19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.314+0000 7ff531612640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00d770 tx=0x7ff51c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.315+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c017070 con 0x7ff52c106170 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.315+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff51c009c30 con 0x7ff52c106170 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.315+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c00f620 con 0x7ff52c106170 2024-04-22T13:44:58.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.315+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c1993a0 con 0x7ff52c106170 2024-04-22T13:44:58.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.315+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a5640 con 0x7ff52c106170 2024-04-22T13:44:58.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.316+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c0691c0 con 0x7ff52c106170 2024-04-22T13:44:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.317+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff51c01a070 con 0x7ff52c106170 2024-04-22T13:44:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.317+0000 7ff522ffd640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 0x7ff5180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.317+0000 7ff531e13640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 0x7ff5180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.317+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8455+0+0 (secure 0 0 0) 0x7ff51c0d02b0 con 0x7ff52c106170 2024-04-22T13:44:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.318+0000 7ff531e13640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 0x7ff5180a6260 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff524000f80 tx=0x7ff524005c50 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.322+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff51c009de0 con 0x7ff52c106170 2024-04-22T13:44:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.542+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff52c101a80 con 0x7ff52c106170 2024-04-22T13:44:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.544+0000 7ff522ffd640 1 -- 172.21.15.87:0/3900230521 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7ff51c09c990 con 0x7ff52c106170 2024-04-22T13:44:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 msgr2=0x7ff5180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 0x7ff5180a6260 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff524000f80 tx=0x7ff524005c50 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 msgr2=0x7ff52c19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00d770 tx=0x7ff51c00dc30 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 shutdown_connections 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7ff5180a3da0 0x7ff5180a6260 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff52c198590 0x7ff52c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff52c106170 0x7ff52c19e920 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 --2- 172.21.15.87:0/3900230521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff52c1010a0 0x7ff52c19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.547+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 >> 172.21.15.87:0/3900230521 conn(0x7ff52c077eb0 msgr2=0x7ff52c104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.548+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 shutdown_connections 2024-04-22T13:44:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.548+0000 7ff534077640 1 -- 172.21.15.87:0/3900230521 wait complete. 2024-04-22T13:44:58.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T13:44:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 -- 172.21.15.87:0/1632594409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510106150 msgr2=0x7f251010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 --2- 172.21.15.87:0/1632594409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510106150 0x7f251010dd20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2508009ca0 tx=0x7f250802c0f0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 -- 172.21.15.87:0/1632594409 shutdown_connections 2024-04-22T13:44:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 --2- 172.21.15.87:0/1632594409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510106150 0x7f251010dd20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 --2- 172.21.15.87:0/1632594409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2510101a30 0x7f2510105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 --2- 172.21.15.87:0/1632594409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510101080 0x7f2510101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 -- 172.21.15.87:0/1632594409 >> 172.21.15.87:0/1632594409 conn(0x7f2510078000 msgr2=0x7f2510078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 -- 172.21.15.87:0/1632594409 shutdown_connections 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.663+0000 7f251800f640 1 -- 172.21.15.87:0/1632594409 wait complete. 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 Processor -- start 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 -- start start 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510106150 0x7f2510199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f251019a020 0x7f25101a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f25101110c0 con 0x7f2510101080 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2510110f40 con 0x7f251019a020 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f251800f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2510111240 con 0x7f2510106150 2024-04-22T13:44:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55166/0 (socket says 172.21.15.87:55166) 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 -- 172.21.15.87:0/4016227438 learned_addr learned my addr 172.21.15.87:0/4016227438 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f25165ac640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f251019a020 0x7f25101a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f25155aa640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510106150 0x7f2510199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 -- 172.21.15.87:0/4016227438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510106150 msgr2=0x7f2510199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510106150 0x7f2510199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 -- 172.21.15.87:0/4016227438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f251019a020 msgr2=0x7f25101a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f251019a020 0x7f25101a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.664+0000 7f2515dab640 1 -- 172.21.15.87:0/4016227438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25101a4da0 con 0x7f2510101080 2024-04-22T13:44:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.665+0000 7f2515dab640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f250c00e9f0 tx=0x7f250c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.667+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c00cd80 con 0x7f2510101080 2024-04-22T13:44:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.667+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f250c0054d0 con 0x7f2510101080 2024-04-22T13:44:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.668+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25101a5090 con 0x7f2510101080 2024-04-22T13:44:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.668+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c010690 con 0x7f2510101080 2024-04-22T13:44:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.668+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25101a55e0 con 0x7f2510101080 2024-04-22T13:44:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.669+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2510068d70 con 0x7f2510101080 2024-04-22T13:44:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.669+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f250c004e90 con 0x7f2510101080 2024-04-22T13:44:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.670+0000 7f2506ffd640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 0x7f24fc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.670+0000 7f25155aa640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 0x7f24fc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.670+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8669+0+0 (secure 0 0 0) 0x7f250c0d0c30 con 0x7f2510101080 2024-04-22T13:44:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.670+0000 7f25155aa640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 0x7f24fc0a6260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2510069be0 tx=0x7f2500005f90 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:58.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.675+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f250c09d240 con 0x7f2510101080 2024-04-22T13:44:58.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:58.890+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f25101093e0 con 0x7f2510101080 2024-04-22T13:44:58.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2110739923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/2550706500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mon[31101]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mds[61442]: -- 172.21.15.120:0/823021494 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555f522e3a40 con 0x555f522b7000 2024-04-22T13:44:58.921 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: starting mds.b at 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mds[60823]: -- 172.21.15.120:0/3480202512 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5567ab56da40 con 0x5567ab540c00 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: starting mds.e at 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-mds[61132]: -- 172.21.15.120:0/3072493589 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c710dda40 con 0x562c710b0800 2024-04-22T13:44:58.923 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:44:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: starting mds.h at 2024-04-22T13:44:58.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2110739923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/2550706500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mon[31531]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mds[62257]: -- 172.21.15.132:0/719442086 <== mon.0 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564112669a40 con 0x56411263c800 2024-04-22T13:44:58.964 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: starting mds.c at 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mds[61948]: -- 172.21.15.132:0/1601678303 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582e53e7a40 con 0x5582e53bb000 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: starting mds.f at 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: ignoring --setuser ceph since I am not root 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: ignoring --setgroup ceph since I am not root 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-mds[62562]: -- 172.21.15.132:0/2347212039 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564065bf5a40 con 0x564065bc9000 2024-04-22T13:44:58.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:44:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: starting mds.i at 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a[27995]: 2024-04-22T13:44:58.543+0000 7f12a8e08700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2110739923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/2550706500' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/3900230521' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T13:44:58.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T13:44:58.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:58 smithi087 ceph-mon[27999]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T13:44:59.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.546+0000 7f2506ffd640 1 -- 172.21.15.87:0/4016227438 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f250c09cb40 con 0x7f2510101080 2024-04-22T13:44:59.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 msgr2=0x7f24fc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 0x7f24fc0a6260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2510069be0 tx=0x7f2500005f90 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 msgr2=0x7f25101995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f250c00e9f0 tx=0x7f250c00edf0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 shutdown_connections 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f24fc0a3da0 0x7f24fc0a6260 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f251019a020 0x7f25101a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2510106150 0x7f2510199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 --2- 172.21.15.87:0/4016227438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2510101080 0x7f25101995a0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 >> 172.21.15.87:0/4016227438 conn(0x7f2510078000 msgr2=0x7f25100fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 shutdown_connections 2024-04-22T13:44:59.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.548+0000 7f251800f640 1 -- 172.21.15.87:0/4016227438 wait complete. 2024-04-22T13:44:59.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T13:44:59.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 -- 172.21.15.87:0/1457272249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 msgr2=0x7f867c10bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:59.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 --2- 172.21.15.87:0/1457272249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c10bff0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f866c009900 tx=0x7f866c02bd60 comp rx=0 tx=0).stop 2024-04-22T13:44:59.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 -- 172.21.15.87:0/1457272249 shutdown_connections 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 --2- 172.21.15.87:0/1457272249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c10c530 0x7f867c10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 --2- 172.21.15.87:0/1457272249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c10bff0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 --2- 172.21.15.87:0/1457272249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f867c1031f0 0x7f867c1035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 -- 172.21.15.87:0/1457272249 >> 172.21.15.87:0/1457272249 conn(0x7f867c0771d0 msgr2=0x7f867c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 -- 172.21.15.87:0/1457272249 shutdown_connections 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.664+0000 7f8681b75640 1 -- 172.21.15.87:0/1457272249 wait complete. 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 Processor -- start 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 -- start start 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c1031f0 0x7f867c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f867c10c530 0x7f867c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7f867c112220 con 0x7f867c103b10 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f867c1120a0 con 0x7f867c1031f0 2024-04-22T13:44:59.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f8681b75640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f867c1123a0 con 0x7f867c10c530 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47892/0 (socket says 172.21.15.87:47892) 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 -- 172.21.15.87:0/4074733955 learned_addr learned my addr 172.21.15.87:0/4074733955 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867b7fe640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c1031f0 0x7f867c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 -- 172.21.15.87:0/4074733955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f867c10c530 msgr2=0x7f867c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f867c10c530 0x7f867c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 -- 172.21.15.87:0/4074733955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c1031f0 msgr2=0x7f867c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c1031f0 0x7f867c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.665+0000 7f867affd640 1 -- 172.21.15.87:0/4074733955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c19d6d0 con 0x7f867c103b10 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f867affd640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f866c005940 tx=0x7f866c02d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c043020 con 0x7f867c103b10 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866c02dda0 con 0x7f867c103b10 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f867c19d960 con 0x7f867c103b10 2024-04-22T13:44:59.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c003bf0 con 0x7f867c103b10 2024-04-22T13:44:59.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.666+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c1a5250 con 0x7f867c103b10 2024-04-22T13:44:59.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.667+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f867c0691c0 con 0x7f867c103b10 2024-04-22T13:44:59.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.668+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f866c046020 con 0x7f867c103b10 2024-04-22T13:44:59.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.669+0000 7f8678ff9640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 0x7f86640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:44:59.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.669+0000 7f867b7fe640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 0x7f86640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:44:59.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.669+0000 7f867b7fe640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 0x7f86640a61e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8670005e10 tx=0x7f8670005da0 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:44:59.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.670+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8669+0+0 (secure 0 0 0) 0x7f866c0054f0 con 0x7f867c103b10 2024-04-22T13:44:59.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.676+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f866c04d020 con 0x7f867c103b10 2024-04-22T13:44:59.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:44:59.859+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f867c1a54b0 con 0x7f867c103b10 2024-04-22T13:44:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:44:59.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: Cluster is now healthy 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6834/3068444686,v1:172.21.15.120:6835/3068444686] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6835/1025305887,v1:172.21.15.132:6838/1025305887] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6834/1463480398,v1:172.21.15.132:6837/1463480398] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.132:6836/3254979385,v1:172.21.15.132:6839/3254979385] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6836/1974374045,v1:172.21.15.120:6837/1974374045] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: mds.? [v2:172.21.15.120:6838/3369043420,v1:172.21.15.120:6839/3369043420] up:boot 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: fsmap 12 up:standby 2024-04-22T13:44:59.921 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:44:59.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:44:59.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:44:59.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:44:59.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:44:59.922 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:44:59 smithi120 ceph-mon[31101]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:44:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:44:59.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: Cluster is now healthy 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6834/3068444686,v1:172.21.15.120:6835/3068444686] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6835/1025305887,v1:172.21.15.132:6838/1025305887] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6834/1463480398,v1:172.21.15.132:6837/1463480398] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.132:6836/3254979385,v1:172.21.15.132:6839/3254979385] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6836/1974374045,v1:172.21.15.120:6837/1974374045] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: mds.? [v2:172.21.15.120:6838/3369043420,v1:172.21.15.120:6839/3369043420] up:boot 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: fsmap 12 up:standby 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:44:59.964 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:44:59.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:44:59.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:44:59.965 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:44:59 smithi132 ceph-mon[31531]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: Cluster is now healthy 2024-04-22T13:44:59.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4016227438' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6834/3068444686,v1:172.21.15.120:6835/3068444686] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6835/1025305887,v1:172.21.15.132:6838/1025305887] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6834/1463480398,v1:172.21.15.132:6837/1463480398] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.132:6836/3254979385,v1:172.21.15.132:6839/3254979385] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6836/1974374045,v1:172.21.15.120:6837/1974374045] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: mds.? [v2:172.21.15.120:6838/3369043420,v1:172.21.15.120:6839/3369043420] up:boot 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: fsmap 12 up:standby 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T13:44:59.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:44:59 smithi087 ceph-mon[27999]: from='mgr.15303 172.21.15.87:0/1598818737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T13:45:00.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.672+0000 7f8678ff9640 1 -- 172.21.15.87:0/4074733955 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7f866c03c350 con 0x7f867c103b10 2024-04-22T13:45:00.669 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cephfs_metadata' removed 2024-04-22T13:45:00.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 msgr2=0x7f86640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:00.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 0x7f86640a61e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8670005e10 tx=0x7f8670005da0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 msgr2=0x7f867c198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f866c005940 tx=0x7f866c02d9a0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 shutdown_connections 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7f86640a3d20 0x7f86640a61e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f867c10c530 0x7f867c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f867c103b10 0x7f867c198c20 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 --2- 172.21.15.87:0/4074733955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f867c1031f0 0x7f867c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 >> 172.21.15.87:0/4074733955 conn(0x7f867c0771d0 msgr2=0x7f867c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 shutdown_connections 2024-04-22T13:45:00.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.675+0000 7f8681b75640 1 -- 172.21.15.87:0/4074733955 wait complete. 2024-04-22T13:45:00.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T13:45:00.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 -- 172.21.15.87:0/2690960751 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb37006ab10 msgr2=0x7fb37006af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 --2- 172.21.15.87:0/2690960751 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb37006ab10 0x7fb37006af70 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fb360009900 tx=0x7fb36002bc30 comp rx=0 tx=0).stop 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 -- 172.21.15.87:0/2690960751 shutdown_connections 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 --2- 172.21.15.87:0/2690960751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb370104980 0x7fb370104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 --2- 172.21.15.87:0/2690960751 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb37006ab10 0x7fb37006af70 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 --2- 172.21.15.87:0/2690960751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37006a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 -- 172.21.15.87:0/2690960751 >> 172.21.15.87:0/2690960751 conn(0x7fb370077eb0 msgr2=0x7fb3700782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 -- 172.21.15.87:0/2690960751 shutdown_connections 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.797+0000 7fb377cba640 1 -- 172.21.15.87:0/2690960751 wait complete. 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 Processor -- start 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 -- start start 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37006ab10 0x7fb3701a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:45:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 -- 0x7fb3701187a0 con 0x7fb370104980 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb370118620 con 0x7fb370106e80 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb377cba640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb370118920 con 0x7fb37006ab10 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47908/0 (socket says 172.21.15.87:47908) 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 -- 172.21.15.87:0/14447448 learned_addr learned my addr 172.21.15.87:0/14447448 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375a56640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37006ab10 0x7fb3701a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb376257640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 -- 172.21.15.87:0/14447448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37006ab10 msgr2=0x7fb3701a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37006ab10 0x7fb3701a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 -- 172.21.15.87:0/14447448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 msgr2=0x7fb37019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb375255640 1 -- 172.21.15.87:0/14447448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37019f260 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.798+0000 7fb376257640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb375255640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb360005940 tx=0x7fb36002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360043020 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb36002cbe0 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37019f4f0 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360003bf0 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.799+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3701ab7f0 con 0x7fb370104980 2024-04-22T13:45:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.800+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3700691c0 con 0x7fb370104980 2024-04-22T13:45:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.801+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb360046020 con 0x7fb370104980 2024-04-22T13:45:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.801+0000 7fb366ffd640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 0x7fb35c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T13:45:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.801+0000 7fb375a56640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 0x7fb35c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T13:45:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.801+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8173+0+0 (secure 0 0 0) 0x7fb3600f2780 con 0x7fb370104980 2024-04-22T13:45:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.802+0000 7fb375a56640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 0x7fb35c0a61e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00bb60 tx=0x7fb36c00b390 comp rx=0 tx=0).ready entity=mgr.15303 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T13:45:00.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.806+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb36004d020 con 0x7fb370104980 2024-04-22T13:45:00.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:00 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:00.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:00 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:00.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:00 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:00.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:00.992+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb3701ab9c0 con 0x7fb370104980 2024-04-22T13:45:01.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.674+0000 7fb366ffd640 1 -- 172.21.15.87:0/14447448 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7fb3600bf0e0 con 0x7fb370104980 2024-04-22T13:45:01.671 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cephfs_data' removed 2024-04-22T13:45:01.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 msgr2=0x7fb35c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:01.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 0x7fb35c0a61e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00bb60 tx=0x7fb36c00b390 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 msgr2=0x7fb3701a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb360005940 tx=0x7fb36002c7e0 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 shutdown_connections 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:6800/2766661417,v1:172.21.15.87:6801/2766661417] conn(0x7fb35c0a3d20 0x7fb35c0a61e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb370106e80 0x7fb37019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb370104980 0x7fb3701a4e70 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 --2- 172.21.15.87:0/14447448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37006ab10 0x7fb3701a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 >> 172.21.15.87:0/14447448 conn(0x7fb370077eb0 msgr2=0x7fb370103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 shutdown_connections 2024-04-22T13:45:01.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-04-22T13:45:01.677+0000 7fb377cba640 1 -- 172.21.15.87:0/14447448 wait complete. 2024-04-22T13:45:01.680 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:45:01.260+0000 7f070def1700 -1 client.15339: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-22T13:45:01.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:45:01.724 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:45:01.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:45:01.794 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T13:45:01.794 DEBUG:teuthology.orchestra.run.smithi087:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T13:45:01.811 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T13:45:01.827 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T13:45:01.851 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T13:45:01.851 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T13:45:01.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a 2024-04-22T13:45:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:01 smithi132 ceph-mon[31531]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:45:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T13:45:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:01 smithi132 ceph-mon[31531]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T13:45:01.963 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:01 smithi132 ceph-mon[31531]: from='client.? 172.21.15.87:0/14447448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:01 smithi087 ceph-mon[27999]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:45:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T13:45:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:01 smithi087 ceph-mon[27999]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T13:45:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:01 smithi087 ceph-mon[27999]: from='client.? 172.21.15.87:0/14447448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:01.988 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:01 smithi087 systemd[1]: Stopping Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:01 smithi120 ceph-mon[31101]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T13:45:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/4074733955' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T13:45:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:01 smithi120 ceph-mon[31101]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T13:45:02.170 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:01 smithi120 ceph-mon[31101]: from='client.? 172.21.15.87:0/14447448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T13:45:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a[27995]: 2024-04-22T13:45:02.228+0000 7f12ae613700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a[27995]: 2024-04-22T13:45:02.228+0000 7f12ae613700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T13:45:02.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:02 smithi087 podman[99538]: 2024-04-22 13:45:02.311428778 +0000 UTC m=+0.216913471 container died 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:45:02.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:02 smithi087 podman[99538]: 2024-04-22 13:45:02.860097574 +0000 UTC m=+0.765582269 container cleanup 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T13:45:02.987 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:02 smithi087 bash[99538]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a 2024-04-22T13:45:03.516 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T13:45:03.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:03 smithi087 podman[99547]: 2024-04-22 13:45:03.278771835 +0000 UTC m=+0.962977529 container remove 2c60a359183708bf30ee112496dbda53a58568f69bc357d3a12f0d6ed8952b61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T13:45:03.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:03 smithi087 systemd[1]: ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T13:45:03.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:03 smithi087 systemd[1]: Stopped Ceph mon.a for 176003e8-00ab-11ef-bc93-c7b262605968. 2024-04-22T13:45:03.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Apr 22 13:45:03 smithi087 systemd[1]: ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.a.service: Consumed 22.790s CPU time. 2024-04-22T13:45:03.552 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:03.553 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T13:45:03.553 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T13:45:03.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.b 2024-04-22T13:45:03.920 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:03 smithi120 systemd[1]: Stopping Ceph mon.b for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b[31097]: 2024-04-22T13:45:04.122+0000 7f2db8a60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b[31097]: 2024-04-22T13:45:04.122+0000 7f2db8a60700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T13:45:04.420 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:04 smithi120 podman[67114]: 2024-04-22 13:45:04.235257661 +0000 UTC m=+0.492764449 container died 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-22T13:45:04.879 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:04 smithi120 podman[67114]: 2024-04-22 13:45:04.62333664 +0000 UTC m=+0.880843441 container cleanup 86f7ebe0fc37f2700fd3f72a719f901d5cdce7331cea5b7727c923c7e686fe23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:45:04.879 INFO:journalctl@ceph.mon.b.smithi120.stdout:Apr 22 13:45:04 smithi120 bash[67114]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-b 2024-04-22T13:45:05.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T13:45:05.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:05.091 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T13:45:05.091 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T13:45:05.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.c 2024-04-22T13:45:05.405 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 systemd[1]: Stopping Ceph mon.c for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:05.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c[31514]: 2024-04-22T13:45:05.408+0000 7fc568435700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:05.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c[31514]: 2024-04-22T13:45:05.408+0000 7fc568435700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T13:45:05.714 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 podman[68505]: 2024-04-22 13:45:05.572393482 +0000 UTC m=+0.280592921 container died 16324ba06b2f1ce725bce1a8a1e6786c8150cdfad5673f37c5ef347d5bdf200e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T13:45:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 podman[68505]: 2024-04-22 13:45:05.976137556 +0000 UTC m=+0.684336996 container cleanup 16324ba06b2f1ce725bce1a8a1e6786c8150cdfad5673f37c5ef347d5bdf200e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T13:45:06.213 INFO:journalctl@ceph.mon.c.smithi132.stdout:Apr 22 13:45:05 smithi132 bash[68505]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mon-c 2024-04-22T13:45:06.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T13:45:06.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:06.437 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T13:45:06.437 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T13:45:06.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x 2024-04-22T13:45:06.736 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:06 smithi087 systemd[1]: Stopping Ceph mgr.x for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:06.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:45:06.740+0000 7f07a9fd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:06.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x[28235]: 2024-04-22T13:45:06.740+0000 7f07a9fd6700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T13:45:06.987 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:06 smithi087 podman[99653]: 2024-04-22 13:45:06.771528979 +0000 UTC m=+0.144954158 container died e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:45:07.405 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:07 smithi087 podman[99653]: 2024-04-22 13:45:07.158286338 +0000 UTC m=+0.531711508 container cleanup e76228482716d49919376fa3ffeab5f7c3143332a7da0730f6369e1c7da558c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T13:45:07.405 INFO:journalctl@ceph.mgr.x.smithi087.stdout:Apr 22 13:45:07 smithi087 bash[99653]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-x 2024-04-22T13:45:07.575 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T13:45:07.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:07.610 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T13:45:07.610 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T13:45:07.611 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.y 2024-04-22T13:45:07.920 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:45:07 smithi120 systemd[1]: Stopping Ceph mgr.y for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:08.274 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:45:07 smithi120 podman[67236]: 2024-04-22 13:45:07.990223504 +0000 UTC m=+0.186820634 container died d17c117fa7ca56d57e5b2258a4d148e5723265ffd18e6fe6df483a81888ff654 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T13:45:08.625 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:45:08 smithi120 podman[67236]: 2024-04-22 13:45:08.369772354 +0000 UTC m=+0.566369471 container cleanup d17c117fa7ca56d57e5b2258a4d148e5723265ffd18e6fe6df483a81888ff654 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:45:08.625 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Apr 22 13:45:08 smithi120 bash[67236]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-y 2024-04-22T13:45:08.786 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T13:45:08.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:08.822 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T13:45:08.823 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T13:45:08.823 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.z 2024-04-22T13:45:09.123 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:45:08 smithi132 systemd[1]: Stopping Ceph mgr.z for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:09.412 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:45:09 smithi132 podman[68625]: 2024-04-22 13:45:09.127208786 +0000 UTC m=+0.100401155 container died 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T13:45:09.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:45:09 smithi132 podman[68625]: 2024-04-22 13:45:09.515766814 +0000 UTC m=+0.488959183 container cleanup 08944774e250e759df6f86cd301b9d9fca01a4f5e615e945c67e48870feba96e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T13:45:09.713 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Apr 22 13:45:09 smithi132 bash[68625]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mgr-z 2024-04-22T13:45:10.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T13:45:10.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:10.310 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T13:45:10.310 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T13:45:10.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.0 2024-04-22T13:45:10.633 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:10 smithi087 systemd[1]: Stopping Ceph osd.0 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:10.987 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0[40130]: 2024-04-22T13:45:10.637+0000 7f8b5b40c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:10.987 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0[40130]: 2024-04-22T13:45:10.637+0000 7f8b5b40c700 -1 osd.0 106 *** Got signal Terminated *** 2024-04-22T13:45:10.987 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0[40130]: 2024-04-22T13:45:10.637+0000 7f8b5b40c700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:16.487 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:16 smithi087 podman[99777]: 2024-04-22 13:45:16.218030379 +0000 UTC m=+5.707926867 container died 8b32e5ee0acb91e8ac5f01b93bbd627a245c143767c69cf23bad30abfd74186d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T13:45:16.868 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:16 smithi087 podman[99777]: 2024-04-22 13:45:16.629952399 +0000 UTC m=+6.119848895 container cleanup 8b32e5ee0acb91e8ac5f01b93bbd627a245c143767c69cf23bad30abfd74186d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T13:45:16.868 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:16 smithi087 bash[99777]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0 2024-04-22T13:45:17.192 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:16 smithi087 podman[99794]: 2024-04-22 13:45:16.872956993 +0000 UTC m=+0.652180841 container remove 8b32e5ee0acb91e8ac5f01b93bbd627a245c143767c69cf23bad30abfd74186d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:45:17.464 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.098945696 +0000 UTC m=+0.017675108 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:17.464 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.215541891 +0000 UTC m=+0.134271277 container create 021972e9016fce510bf8d09bbfb2a121dc18d5bd973b7e2df10852413a43fc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:45:17.464 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.468466359 +0000 UTC m=+0.387195750 container init 021972e9016fce510bf8d09bbfb2a121dc18d5bd973b7e2df10852413a43fc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T13:45:17.737 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.471562386 +0000 UTC m=+0.390291779 container start 021972e9016fce510bf8d09bbfb2a121dc18d5bd973b7e2df10852413a43fc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T13:45:17.738 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.577003965 +0000 UTC m=+0.495733371 container attach 021972e9016fce510bf8d09bbfb2a121dc18d5bd973b7e2df10852413a43fc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:45:18.109 INFO:journalctl@ceph.osd.0.smithi087.stdout:Apr 22 13:45:17 smithi087 podman[99887]: 2024-04-22 13:45:17.762660016 +0000 UTC m=+0.681389411 container died 021972e9016fce510bf8d09bbfb2a121dc18d5bd973b7e2df10852413a43fc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T13:45:18.416 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T13:45:18.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:18.455 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T13:45:18.455 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T13:45:18.455 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.1 2024-04-22T13:45:18.753 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:18 smithi120 systemd[1]: Stopping Ceph osd.1 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:19.170 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1[35704]: 2024-04-22T13:45:18.757+0000 7fa45e2d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:19.170 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1[35704]: 2024-04-22T13:45:18.757+0000 7fa45e2d8700 -1 osd.1 106 *** Got signal Terminated *** 2024-04-22T13:45:19.171 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1[35704]: 2024-04-22T13:45:18.757+0000 7fa45e2d8700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:24.511 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:24 smithi120 podman[67360]: 2024-04-22 13:45:24.198124901 +0000 UTC m=+5.541849452 container died 77fc0d67188a4529c5027f3c40614b6f2b8a74f821c8d7ea558779357e28316e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:45:24.511 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:24 smithi120 podman[67360]: 2024-04-22 13:45:24.51502019 +0000 UTC m=+5.858744751 container cleanup 77fc0d67188a4529c5027f3c40614b6f2b8a74f821c8d7ea558779357e28316e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-22T13:45:24.920 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:24 smithi120 bash[67360]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1 2024-04-22T13:45:24.921 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:24 smithi120 podman[67377]: 2024-04-22 13:45:24.716688941 +0000 UTC m=+0.523075353 container remove 77fc0d67188a4529c5027f3c40614b6f2b8a74f821c8d7ea558779357e28316e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-22T13:45:25.382 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.025746827 +0000 UTC m=+0.018733150 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:25.383 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.142400422 +0000 UTC m=+0.135386731 container create 33d3586ac18e7414fb1a9ba8f3b4fd0428c83002a4305c8d5bb6f5197c5f666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T13:45:25.671 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.386647232 +0000 UTC m=+0.379633540 container init 33d3586ac18e7414fb1a9ba8f3b4fd0428c83002a4305c8d5bb6f5197c5f666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:45:25.671 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.390104956 +0000 UTC m=+0.383091271 container start 33d3586ac18e7414fb1a9ba8f3b4fd0428c83002a4305c8d5bb6f5197c5f666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T13:45:25.671 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.48673666 +0000 UTC m=+0.479722997 container attach 33d3586ac18e7414fb1a9ba8f3b4fd0428c83002a4305c8d5bb6f5197c5f666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T13:45:26.077 INFO:journalctl@ceph.osd.1.smithi120.stdout:Apr 22 13:45:25 smithi120 podman[67470]: 2024-04-22 13:45:25.690254556 +0000 UTC m=+0.683240868 container died 33d3586ac18e7414fb1a9ba8f3b4fd0428c83002a4305c8d5bb6f5197c5f666f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-22T13:45:26.378 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T13:45:26.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:26.415 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T13:45:26.415 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T13:45:26.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.2 2024-04-22T13:45:26.714 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:26 smithi132 systemd[1]: Stopping Ceph osd.2 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:27.214 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2[36682]: 2024-04-22T13:45:26.777+0000 7effbaa36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:27.214 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2[36682]: 2024-04-22T13:45:26.777+0000 7effbaa36700 -1 osd.2 106 *** Got signal Terminated *** 2024-04-22T13:45:27.214 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2[36682]: 2024-04-22T13:45:26.777+0000 7effbaa36700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:32.667 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:32 smithi132 podman[68751]: 2024-04-22 13:45:32.371494809 +0000 UTC m=+5.730815711 container died 95be258861495be4efc69697c53ce9f0e9dc5cbfc689e50be60292e73ce5a4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T13:45:32.963 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:32 smithi132 podman[68751]: 2024-04-22 13:45:32.764044834 +0000 UTC m=+6.123365752 container cleanup 95be258861495be4efc69697c53ce9f0e9dc5cbfc689e50be60292e73ce5a4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:45:32.964 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:32 smithi132 bash[68751]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2 2024-04-22T13:45:33.338 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68779]: 2024-04-22 13:45:33.02388637 +0000 UTC m=+0.652241977 container remove 95be258861495be4efc69697c53ce9f0e9dc5cbfc689e50be60292e73ce5a4b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T13:45:33.599 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.244572355 +0000 UTC m=+0.017776223 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:33.599 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.358344609 +0000 UTC m=+0.131548470 container create eaa545400d885505cacdf9d2e5d39fc124500dd9c7e8c14fd06612c6e43fd8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T13:45:33.893 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.603322001 +0000 UTC m=+0.376525864 container init eaa545400d885505cacdf9d2e5d39fc124500dd9c7e8c14fd06612c6e43fd8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T13:45:33.893 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.60670116 +0000 UTC m=+0.379905021 container start eaa545400d885505cacdf9d2e5d39fc124500dd9c7e8c14fd06612c6e43fd8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:45:33.893 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.722328549 +0000 UTC m=+0.495532430 container attach eaa545400d885505cacdf9d2e5d39fc124500dd9c7e8c14fd06612c6e43fd8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T13:45:34.212 INFO:journalctl@ceph.osd.2.smithi132.stdout:Apr 22 13:45:33 smithi132 podman[68867]: 2024-04-22 13:45:33.897159985 +0000 UTC m=+0.670363838 container died eaa545400d885505cacdf9d2e5d39fc124500dd9c7e8c14fd06612c6e43fd8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T13:45:34.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T13:45:34.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:34.604 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T13:45:34.604 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T13:45:34.605 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.3 2024-04-22T13:45:34.922 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:34 smithi087 systemd[1]: Stopping Ceph osd.3 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:35.237 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3[46676]: 2024-04-22T13:45:34.925+0000 7f7ed7ed3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:35.238 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3[46676]: 2024-04-22T13:45:34.925+0000 7f7ed7ed3700 -1 osd.3 106 *** Got signal Terminated *** 2024-04-22T13:45:35.238 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:34 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3[46676]: 2024-04-22T13:45:34.925+0000 7f7ed7ed3700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:37.874 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:37 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:37.545+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:38.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:38.245+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:38.737 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:38 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:38.391+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:38.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:38 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:38.537+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:39.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:39.241+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:39.737 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:39 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:39.440+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:39.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:39 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:39.584+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:40.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:40.265+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:40.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:40 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:40.455+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:40.738 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:40 smithi087 podman[100004]: 2024-04-22 13:45:40.488373596 +0000 UTC m=+5.688147446 container died aab30a1aeffe7f685612d905328e4fad4ace7bd8f000c00124e20156e25974af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-22T13:45:40.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:40 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:40.610+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:41.132 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:40 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:40.802+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:41.132 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:40 smithi087 podman[100004]: 2024-04-22 13:45:40.892879283 +0000 UTC m=+6.092653107 container cleanup aab30a1aeffe7f685612d905328e4fad4ace7bd8f000c00124e20156e25974af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T13:45:41.132 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:40 smithi087 bash[100004]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3 2024-04-22T13:45:41.465 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100021]: 2024-04-22 13:45:41.135918532 +0000 UTC m=+0.648984248 container remove aab30a1aeffe7f685612d905328e4fad4ace7bd8f000c00124e20156e25974af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T13:45:41.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:41.260+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:41.721 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:41 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:41.468+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:41.721 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100116]: 2024-04-22 13:45:41.465319855 +0000 UTC m=+0.018402025 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:41.721 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100116]: 2024-04-22 13:45:41.595067648 +0000 UTC m=+0.148149847 container create 74d6360d9ef4b86acbba43824617d9e065a445269bac4a86bbfab2444d734166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T13:45:41.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:41 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:41.567+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:41.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:41 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:41.810+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:41.988 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100116]: 2024-04-22 13:45:41.856150492 +0000 UTC m=+0.409232649 container init 74d6360d9ef4b86acbba43824617d9e065a445269bac4a86bbfab2444d734166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T13:45:41.988 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100116]: 2024-04-22 13:45:41.859220445 +0000 UTC m=+0.412302602 container start 74d6360d9ef4b86acbba43824617d9e065a445269bac4a86bbfab2444d734166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:45:41.988 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:41 smithi087 podman[100116]: 2024-04-22 13:45:41.947826635 +0000 UTC m=+0.500908807 container attach 74d6360d9ef4b86acbba43824617d9e065a445269bac4a86bbfab2444d734166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T13:45:42.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:41.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:42.418 INFO:journalctl@ceph.osd.3.smithi087.stdout:Apr 22 13:45:42 smithi087 podman[100116]: 2024-04-22 13:45:42.150300164 +0000 UTC m=+0.703382313 container died 74d6360d9ef4b86acbba43824617d9e065a445269bac4a86bbfab2444d734166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T13:45:42.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:42.288+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:42.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:42 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:42.483+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:42.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T13:45:42.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:42.830 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T13:45:42.830 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T13:45:42.830 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.4 2024-04-22T13:45:42.855 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:42 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:42.589+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:43.171 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:42 smithi120 systemd[1]: Stopping Ceph osd.4 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:43.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:42.957+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:43.237 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:42 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:42.799+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:43.576 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:43 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:45:43.300+0000 7f1bf2647700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:43.576 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:43 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:45:43.300+0000 7f1bf2647700 -1 osd.4 106 *** Got signal Terminated *** 2024-04-22T13:45:43.576 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:43 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:45:43.300+0000 7f1bf2647700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:43.713 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:43.335+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:43.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:43 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:43.523+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:43.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:43 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:43.579+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:44.237 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:43 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:43.761+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:44.364 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:43.978+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:44.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:44 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:44.367+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:44.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:44 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:44.533+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:44.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:44 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:44.724+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:44.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:44 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:44.508+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:45.315 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:44 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:44.992+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:45.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:45.319+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:45.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:45.319+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:45.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:45.710+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:45.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:45.478+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:45.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:45 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:45.478+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:45.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:45.554+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:45.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:45 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:45.696+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:46.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:45 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:45.957+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:46.711 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:46 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:46.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:46.711 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:46 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:46.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:46.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:46.383+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:46.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:46.278+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:46.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:46.278+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:46.921 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:46 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:45:46.492+0000 7f1beac33700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:20.544270+0000 front 2024-04-22T13:45:20.544299+0000 (oldest deadline 2024-04-22T13:45:45.844381+0000) 2024-04-22T13:45:46.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:46 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:46.688+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:46.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:46 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:46.541+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:46.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:46 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:46.714+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:47.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:46.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:47.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:46 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:46.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:47.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:47.395+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:47.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:47.270+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:47.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:47.270+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:47.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:47.421+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:47.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:47.421+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:47.921 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4[42658]: 2024-04-22T13:45:47.496+0000 7f1beac33700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:20.544270+0000 front 2024-04-22T13:45:20.544299+0000 (oldest deadline 2024-04-22T13:45:45.844381+0000) 2024-04-22T13:45:47.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:47.646+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:47.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:47 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:47.568+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:48.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:47.942+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:48.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:47 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:47.942+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:48.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:47.746+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:48.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:48.242+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:48.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:48.242+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:48.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:48.429+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:48.723 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:48 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:48.448+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:48.723 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:48 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:48.448+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:48.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:48 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:48.661+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:48.921 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:48 smithi120 podman[67589]: 2024-04-22 13:45:48.71656085 +0000 UTC m=+5.683972446 container died 85cd42f6de0b3f15ad1b24e04e8ed767e64f2c6d5845e7848ccfd62254b6d3c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:45:48.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:48 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:48.537+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:48.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:48 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:48.726+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:49.201 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:48.906+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:49.201 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:48 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:48.906+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:49.357 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67589]: 2024-04-22 13:45:49.117512586 +0000 UTC m=+6.084924172 container cleanup 85cd42f6de0b3f15ad1b24e04e8ed767e64f2c6d5845e7848ccfd62254b6d3c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:45:49.357 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 bash[67589]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4 2024-04-22T13:45:49.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:49.204+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:49.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:49.204+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:49.649 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67606]: 2024-04-22 13:45:49.360459483 +0000 UTC m=+0.642873897 container remove 85cd42f6de0b3f15ad1b24e04e8ed767e64f2c6d5845e7848ccfd62254b6d3c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-22T13:45:49.649 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:49.502+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:49.649 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:49.502+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:49.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:49.452+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:49.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:49.452+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:49.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:49 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:49.652+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:49.921 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67697]: 2024-04-22 13:45:49.586896607 +0000 UTC m=+0.017976708 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:49.922 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67697]: 2024-04-22 13:45:49.719695204 +0000 UTC m=+0.150775305 container create 165f4a7bebda451156c4368b130494498729c9f0a86857967541492243786de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:45:49.943 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:49.476+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:50.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:50.162+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:50.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:50.162+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:50.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:49.947+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:50.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:49 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:49.947+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:50.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:49.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:50.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:49 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:49.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:50.240 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67697]: 2024-04-22 13:45:49.947534796 +0000 UTC m=+0.378614897 container init 165f4a7bebda451156c4368b130494498729c9f0a86857967541492243786de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T13:45:50.240 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:49 smithi120 podman[67697]: 2024-04-22 13:45:49.950660201 +0000 UTC m=+0.381740310 container start 165f4a7bebda451156c4368b130494498729c9f0a86857967541492243786de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T13:45:50.240 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:50 smithi120 podman[67697]: 2024-04-22 13:45:50.055815006 +0000 UTC m=+0.486895120 container attach 165f4a7bebda451156c4368b130494498729c9f0a86857967541492243786de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T13:45:50.547 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:50.489+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:50.547 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:50.489+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:50.547 INFO:journalctl@ceph.osd.4.smithi120.stdout:Apr 22 13:45:50 smithi120 podman[67697]: 2024-04-22 13:45:50.244020422 +0000 UTC m=+0.675100517 container died 165f4a7bebda451156c4368b130494498729c9f0a86857967541492243786de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True) 2024-04-22T13:45:50.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:50.464+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:50.735 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:50.475+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:50.735 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:50.475+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:50.839 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:50 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:50.643+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:50.862 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T13:45:50.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:50.938 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T13:45:50.939 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T13:45:50.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.5 2024-04-22T13:45:50.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:50.738+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:50.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:50 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:50.738+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:51.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:50.930+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:51.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:50 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:50.930+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:51.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:50 smithi132 systemd[1]: Stopping Ceph osd.5 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:51.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:51.176+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:51.215 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:51.176+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:51.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:51.496+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:51.714 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:51.268+0000 7f6de29bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:51.715 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:51.268+0000 7f6de29bc700 -1 osd.5 106 *** Got signal Terminated *** 2024-04-22T13:45:51.715 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:51.268+0000 7f6de29bc700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:51.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:51.651+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:51.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:51.474+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:51.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:51 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:51.474+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:51.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:51.523+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:51.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:51.523+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:51.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:51.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:51.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:51.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:52.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:52.195+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:52.214 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:52.195+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:52.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:51.951+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:52.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:51 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:51.951+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:52.907 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:52.545+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:52.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:52.622+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:52.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:52.512+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:52.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:52 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:52.512+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:52.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:52.533+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:52.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:52.533+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:52.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:52.701+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:52.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:52 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:52.701+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:53.177 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:52.909+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:53.177 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:52 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:52.909+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:53.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:53.179+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:53.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:53.179+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:53.897 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:53.546+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:53.897 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:53.546+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:53.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:53.642+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:53.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:53.518+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:53.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:53 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:53.518+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:53.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:53.505+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:53.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:53.505+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:53.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:53.663+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:53.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:53 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:53.663+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:54.198 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:53.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:54.198 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:53 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:53.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:54.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:54.201+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:54.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:54.201+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:54.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:54.201+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.538918+0000 front 2024-04-22T13:45:28.538940+0000 (oldest deadline 2024-04-22T13:45:53.239043+0000) 2024-04-22T13:45:54.854 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:54.592+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:54.855 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:54.592+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:54.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:54.661+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:54.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:54.477+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:54.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:54 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:54.477+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:54.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:54.507+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:54.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:54.507+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:54.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:54.685+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:54.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:54.685+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:54.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:54 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:54.685+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:45:55.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:54.857+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:55.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:54.857+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:55.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:54 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:54.857+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:45:55.628 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:55.226+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:55.628 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:55.227+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:55.628 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:55.227+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.538918+0000 front 2024-04-22T13:45:28.538940+0000 (oldest deadline 2024-04-22T13:45:53.239043+0000) 2024-04-22T13:45:55.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:55.702+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:55.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:55.702+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:55.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:55.702+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:45:55.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:55.474+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:55.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:55.474+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:55.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:55.474+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:45:55.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:55.495+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:55.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:55.495+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:55.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:55.627+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:55.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:55 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:55.627+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:45:55.964 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:55.630+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:55.964 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:55.630+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:55.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:55.834+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:55.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:55.834+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:55.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:55 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:55.834+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:45:56.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:56.177+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.136380+0000 front 2024-04-22T13:45:15.136418+0000 (oldest deadline 2024-04-22T13:45:37.436505+0000) 2024-04-22T13:45:56.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:56.177+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.737778+0000 front 2024-04-22T13:45:22.737803+0000 (oldest deadline 2024-04-22T13:45:44.437876+0000) 2024-04-22T13:45:56.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5[43346]: 2024-04-22T13:45:56.177+0000 7f6ddefb0700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.538918+0000 front 2024-04-22T13:45:28.538940+0000 (oldest deadline 2024-04-22T13:45:53.239043+0000) 2024-04-22T13:45:56.736 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:56.649+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:56.736 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:56.649+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:56.707+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:56.707+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:56.707+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:56.481+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:56.481+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:56.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:56 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:56.481+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:45:56.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:56.662+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:56.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:56.662+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:45:56.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:56.527+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:56.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:56.527+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:56.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:56 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:56.527+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:45:57.174 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:56 smithi132 podman[68975]: 2024-04-22 13:45:56.870190421 +0000 UTC m=+5.733019099 container died bd41d0c7559f0d1bd30d7e07316d02dadd2d38fb3b836c2ff21eb0d71547443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T13:45:57.175 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:56.879+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:57.175 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:56.879+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:57.175 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:56 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:56.879+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:45:57.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:57 smithi132 podman[68975]: 2024-04-22 13:45:57.319338389 +0000 UTC m=+6.182167061 container cleanup bd41d0c7559f0d1bd30d7e07316d02dadd2d38fb3b836c2ff21eb0d71547443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T13:45:57.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:57 smithi132 bash[68975]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5 2024-04-22T13:45:57.713 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:57.436+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:57.713 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:57.436+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:57.713 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:57.436+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:45:57.822 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:57 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:57.704+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:57.823 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:57 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:57.704+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:45:57.823 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:57 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:57.496+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:57.823 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:57 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:57.496+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:57.823 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:57 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:57.496+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:45:57.866 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:57 smithi132 podman[68996]: 2024-04-22 13:45:57.645960575 +0000 UTC m=+0.770895421 container remove bd41d0c7559f0d1bd30d7e07316d02dadd2d38fb3b836c2ff21eb0d71547443d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T13:45:57.866 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:57 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:57.599+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:57.866 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:57 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:57.599+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:57.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:57.716+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:57.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:57.716+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:57.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:57 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:57.716+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:45:58.132 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:57 smithi132 podman[69089]: 2024-04-22 13:45:57.895812977 +0000 UTC m=+0.018219021 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:45:58.132 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:58 smithi132 podman[69089]: 2024-04-22 13:45:58.038818367 +0000 UTC m=+0.161224452 container create c25beecd7932139d341e0d3e8073ebce6e50c9783bb783f790dc844da3a486f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T13:45:58.133 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:57 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:57.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:58.133 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:57 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:57.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:58.133 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:57 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:57.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:45:58.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:58 smithi132 podman[69089]: 2024-04-22 13:45:58.266739439 +0000 UTC m=+0.389145486 container init c25beecd7932139d341e0d3e8073ebce6e50c9783bb783f790dc844da3a486f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T13:45:58.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:58 smithi132 podman[69089]: 2024-04-22 13:45:58.269908934 +0000 UTC m=+0.392314977 container start c25beecd7932139d341e0d3e8073ebce6e50c9783bb783f790dc844da3a486f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T13:45:58.464 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:58 smithi132 podman[69089]: 2024-04-22 13:45:58.36667539 +0000 UTC m=+0.489081434 container attach c25beecd7932139d341e0d3e8073ebce6e50c9783bb783f790dc844da3a486f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T13:45:58.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:58.670+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:58.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:58.670+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:58.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:58.670+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:45:58.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:58.448+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:58.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:58.448+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:58.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:58 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:58.448+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:45:58.852 INFO:journalctl@ceph.osd.5.smithi132.stdout:Apr 22 13:45:58 smithi132 podman[69089]: 2024-04-22 13:45:58.55938158 +0000 UTC m=+0.681787617 container died c25beecd7932139d341e0d3e8073ebce6e50c9783bb783f790dc844da3a486f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-22T13:45:58.852 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:58.576+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:58.852 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:58.576+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:58.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:58.672+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:58.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:58.672+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:45:58.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:58.493+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:58.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:58.493+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:58.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:58 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:58.493+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:45:59.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:58.854+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:45:59.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:58.854+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:45:59.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:58 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:58.854+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:45:59.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T13:45:59.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:45:59.285 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T13:45:59.285 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T13:45:59.285 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.6 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 systemd[1]: Stopping Ceph osd.6 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.456+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.456+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.456+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.596+0000 7fa095508700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.596+0000 7fa095508700 -1 osd.6 106 *** Got signal Terminated *** 2024-04-22T13:45:59.594 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:45:59.596+0000 7fa095508700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:45:59.868 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:59.529+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:45:59.868 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:45:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:45:59.529+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:45:59.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:59 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:59.706+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:45:59.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:45:59 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:45:59.706+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:45:59.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:59 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:59.535+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:45:59.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:59 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:59.535+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:45:59.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:45:59 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:45:59.535+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:45:59.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:59.709+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:45:59.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:59.709+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:45:59.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:45:59 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:45:59.709+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:00.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:59.870+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:00.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:59.870+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:00.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:45:59 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:45:59.870+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:00.719 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:00.468+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:46:00.719 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:00.468+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:46:00.719 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:00.468+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:46:00.848 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:00 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:00.515+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:00.848 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:00 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:00.515+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:00.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:00 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:00.683+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:00.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:00 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:00.683+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:00.921 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:00 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:00.584+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:00.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:00 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:00.584+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:00.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:00 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:00.584+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:00.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:00.722+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:00.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:00.722+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:00.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:00.722+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:01.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:00 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:00.851+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:01.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:00 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:00.851+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:01.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:00 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:00.851+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:01.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:01.428+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:46:01.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:01.428+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:46:01.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:01.428+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:46:01.802 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:01 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:01.536+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:01.802 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:01 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:01.536+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:01.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:01.684+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:01.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:01.684+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:01.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:01.684+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:01.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:01.630+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:01.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:01.630+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:01.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:01 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:01.630+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:02.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:01 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:01.804+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:02.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:01 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:01.804+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:02.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:01 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:01.804+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:02.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:01.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:02.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:01.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:02.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:01.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:02.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:01 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:01.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:02.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:02.696+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:02.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:02.696+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:02.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:02.696+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:02.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:02.696+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:02.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:02.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:46:02.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:02.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:46:02.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:02.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:46:02.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:02 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:02.453+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.769578+0000 front 2024-04-22T13:45:39.769563+0000 (oldest deadline 2024-04-22T13:46:01.469666+0000) 2024-04-22T13:46:02.839 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:02 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:02.540+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:02.840 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:02 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:02.540+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:02.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:02.711+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:02.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:02.711+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:02.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:02.711+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:02.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:02.590+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:02.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:02.590+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:02.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:02.590+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:02.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:02 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:02.590+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:03.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:02 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:02.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:03.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:02 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:02.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:03.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:02 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:02.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:03.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:03.695+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:03.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:03.695+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:03.695+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:03.695+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:03.464+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:03.464+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:03.464+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:46:03.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:03 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:03.464+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.769578+0000 front 2024-04-22T13:45:39.769563+0000 (oldest deadline 2024-04-22T13:46:01.469666+0000) 2024-04-22T13:46:03.811 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:03.497+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:03.811 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:03.497+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:03.811 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:03.497+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:03.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:03.717+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:03.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:03.717+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:03.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:03.717+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:03.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:03.547+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:03.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:03.547+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:03.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:03.547+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:03.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:03 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:03.547+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:04.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:03.814+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:04.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:03.814+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:04.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:03.814+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:04.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:03 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:03.814+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:04.738 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:04.473+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:13.565168+0000 front 2024-04-22T13:45:13.565210+0000 (oldest deadline 2024-04-22T13:45:37.665319+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:04.473+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:19.966498+0000 front 2024-04-22T13:45:19.966648+0000 (oldest deadline 2024-04-22T13:45:44.666290+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:04.473+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.468132+0000 front 2024-04-22T13:45:30.468246+0000 (oldest deadline 2024-04-22T13:45:54.568106+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6[53514]: 2024-04-22T13:46:04.473+0000 7fa091afc700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.769578+0000 front 2024-04-22T13:45:39.769563+0000 (oldest deadline 2024-04-22T13:46:01.469666+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:04.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:04.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:04.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:04.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:04 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:04.711+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:04.812 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:04.544+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:04.812 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:04.544+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:04.812 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:04.544+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:04.921 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:04.737+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:04.737+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:04.737+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:04.503+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:04.503+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:04.503+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:04.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:04.503+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:05.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:04.815+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:05.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:04.815+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:05.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:04.815+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:05.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:04 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:04.815+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:05.238 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:04 smithi087 podman[100233]: 2024-04-22 13:46:04.988594816 +0000 UTC m=+5.506672711 container died 3c5cabe93a64629377a12b5645ea86d0ccb3ab53bf793f354297ff7490c1a8a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T13:46:05.661 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:05 smithi087 podman[100233]: 2024-04-22 13:46:05.354993955 +0000 UTC m=+5.873071830 container cleanup 3c5cabe93a64629377a12b5645ea86d0ccb3ab53bf793f354297ff7490c1a8a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:46:05.661 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:05 smithi087 bash[100233]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6 2024-04-22T13:46:05.807 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:05.559+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:05.808 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:05.559+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:05.808 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:05.559+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:05.741+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:05.741+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:05.741+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:05.516+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:05.516+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:05.516+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:05.922 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:05 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:05.516+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:05.988 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:05 smithi087 podman[100250]: 2024-04-22 13:46:05.665089602 +0000 UTC m=+0.671549534 container remove 3c5cabe93a64629377a12b5645ea86d0ccb3ab53bf793f354297ff7490c1a8a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:46:05.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:05 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:05.717+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:05.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:05 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:05.717+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:05.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:05 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:05.717+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:05.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:05 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:05.717+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:06.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:05.810+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:06.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:05.810+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:06.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:05.810+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:06.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:05 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:05.810+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:06.268 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:05.923629916 +0000 UTC m=+0.017709866 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:06.268 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:06.032755771 +0000 UTC m=+0.126835723 container create b4102891efa5827082ec8a268578bd1aa123015a86ff27dc4636cfe98c215a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T13:46:06.268 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:06.26871789 +0000 UTC m=+0.362797842 container init b4102891efa5827082ec8a268578bd1aa123015a86ff27dc4636cfe98c215a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T13:46:06.268 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:06.271944098 +0000 UTC m=+0.366024051 container start b4102891efa5827082ec8a268578bd1aa123015a86ff27dc4636cfe98c215a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:06.559 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:06.377105243 +0000 UTC m=+0.471185210 container attach b4102891efa5827082ec8a268578bd1aa123015a86ff27dc4636cfe98c215a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T13:46:06.771 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:06.480+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:06.771 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:06.480+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:06.771 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:06.480+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:06.772 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:06.480+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:06.809 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:06.512+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:06.810 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:06.512+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:06.810 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:06.512+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:06.877 INFO:journalctl@ceph.osd.6.smithi087.stdout:Apr 22 13:46:06 smithi087 podman[100337]: 2024-04-22 13:46:06.563974689 +0000 UTC m=+0.658054640 container died b4102891efa5827082ec8a268578bd1aa123015a86ff27dc4636cfe98c215a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T13:46:06.878 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:06.753+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:06.878 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:06.753+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:06.878 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:06.753+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:06.878 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:06 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:06.753+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:07.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:06.774+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:07.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:06.774+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:07.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:06 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:06.774+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:07.182 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:06.812+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:07.182 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:06.812+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:07.182 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:06.812+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:07.182 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:06 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:06.812+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:07.227 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T13:46:07.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:07.268 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T13:46:07.268 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T13:46:07.268 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.7 2024-04-22T13:46:07.550 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 systemd[1]: Stopping Ceph osd.7 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:07.550 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:07.478+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:07.550 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:07.478+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:07.550 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:07.478+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:07.550 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:07.478+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:07.774 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:07.488+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:07.774 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:07.488+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:07.774 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:07.488+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:07.800 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.552+0000 7f886e149700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:07.800 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.552+0000 7f886e149700 -1 osd.7 106 *** Got signal Terminated *** 2024-04-22T13:46:07.801 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.552+0000 7f886e149700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:46:08.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.803+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:08.172 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.803+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:08.172 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:07.803+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:08.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:07.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:08.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:07.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:08.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:07.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:08.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:07 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:07.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:08.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:07.795+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:08.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:07.795+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:08.239 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:07.795+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:08.239 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:07 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:07.795+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:08.790 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:08.478+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:08.790 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:08.478+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:08.790 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:08.478+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:08.845 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:08.446+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:08.846 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:08.446+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:08.846 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:08.446+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:08.846 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:08.446+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:09.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:08.848+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:09.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:08.848+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:09.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:08 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:08.848+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:09.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:08.792+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:09.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:08.792+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:09.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:08.792+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:09.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:08 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:08.792+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:09.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:08.775+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:09.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:08.775+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:09.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:08.775+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:09.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:08 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:08.775+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:09.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:09.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:09.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:09.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:09.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:09.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:09.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:09.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:09.765 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:09.481+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:09.765 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:09.481+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:09.765 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:09.481+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:10.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:09.862+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:10.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:09.862+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:10.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:09 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:09.862+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:10.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:09.768+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:10.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:09.768+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:10.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:09.768+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:10.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:09 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:09.768+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:10.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:09.747+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:10.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:09.747+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:10.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:09.747+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:10.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:09 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:09.747+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:10.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:10.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:10.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:10.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:10.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:10.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:10.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:10.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:10.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:10.399+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:10.793 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:10.491+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:10.793 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:10.491+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:10.793 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:10.491+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:10.793 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:10.491+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:11.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:10.812+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:11.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:10.812+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:11.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:10 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:10.812+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:11.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:10.795+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:11.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:10.795+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:11.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:10.795+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:11.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:10 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:10.795+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:11.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:10.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:11.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:10.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:11.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:10.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:11.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:10.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:11.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:10 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:10.741+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:11.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:11.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:11.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:11.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:11.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:11.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:11.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:11.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:11.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:11.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:11.777 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:11.476+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:11.777 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:11.476+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:11.778 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:11.476+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:11.778 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:11.476+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:11.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:11 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:11.712+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:11.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:11 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:11.712+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:11.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:11 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:11.712+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:11.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:11 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:11.712+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:11.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:11 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:11.712+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:12.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:11.824+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.426298+0000 front 2024-04-22T13:45:21.426275+0000 (oldest deadline 2024-04-22T13:45:44.926396+0000) 2024-04-22T13:46:12.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:11.824+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:30.728110+0000 front 2024-04-22T13:45:30.728087+0000 (oldest deadline 2024-04-22T13:45:54.827881+0000) 2024-04-22T13:46:12.171 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:11.824+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:39.528858+0000 front 2024-04-22T13:45:39.528840+0000 (oldest deadline 2024-04-22T13:46:01.228846+0000) 2024-04-22T13:46:12.172 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7[49180]: 2024-04-22T13:46:11.824+0000 7f8866735700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.230383+0000 front 2024-04-22T13:45:48.230403+0000 (oldest deadline 2024-04-22T13:46:11.130480+0000) 2024-04-22T13:46:12.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:11.780+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:12.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:11.780+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:12.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:11.780+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:12.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:11 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:11.780+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:12.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:12.316+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:12.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:12.316+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:12.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:12.316+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:12.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:12.316+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:12.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:12 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:12.316+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:12.796 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:12.474+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:12.796 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:12.474+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:12.796 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:12.474+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:12.796 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:12.474+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:12.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:12.672+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:12.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:12.672+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:12.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:12.672+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:12.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:12.672+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:12.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:12 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:12.672+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:13.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:12.799+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:13.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:12.799+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:13.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:12.799+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:13.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:12.799+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:13.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:12 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:12.799+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:13.326 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:13 smithi120 podman[67815]: 2024-04-22 13:46:13.058468951 +0000 UTC m=+5.598024873 container died 2115bf0b903a0b8921b3b10ec4180ce0f47126e1226f2fa42993a46fcebcf40b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:13.671 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:13 smithi120 podman[67815]: 2024-04-22 13:46:13.412952687 +0000 UTC m=+5.952508611 container cleanup 2115bf0b903a0b8921b3b10ec4180ce0f47126e1226f2fa42993a46fcebcf40b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T13:46:13.671 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:13 smithi120 bash[67815]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7 2024-04-22T13:46:13.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:13 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:13.354+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:13.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:13 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:13.354+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:13.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:13 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:13.354+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:13.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:13 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:13.354+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:13.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:13 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:13.354+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:13.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:13.443+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:13.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:13.443+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:13.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:13.443+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:13.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:13.443+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:13.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:13 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:13.650+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:13.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:13 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:13.650+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:13.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:13 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:13.650+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:13.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:13 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:13.650+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:13.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:13 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:13.650+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:14.079 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:13 smithi120 podman[67833]: 2024-04-22 13:46:13.730979928 +0000 UTC m=+0.668777964 container remove 2115bf0b903a0b8921b3b10ec4180ce0f47126e1226f2fa42993a46fcebcf40b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T13:46:14.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:13.805+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:14.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:13.805+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:14.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:13.805+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:14.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:13.805+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:14.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:13 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:13.805+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:14.363 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:13.985673727 +0000 UTC m=+0.018260014 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:14.363 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:14.108006721 +0000 UTC m=+0.140592995 container create 682892edf5fc0106079f16a78cc66fe4c703d4e1842d6be1148487b3903a70d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:14.672 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:14.383983528 +0000 UTC m=+0.416569807 container init 682892edf5fc0106079f16a78cc66fe4c703d4e1842d6be1148487b3903a70d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:46:14.672 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:14.386993544 +0000 UTC m=+0.419579824 container start 682892edf5fc0106079f16a78cc66fe4c703d4e1842d6be1148487b3903a70d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T13:46:14.672 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:14.45099981 +0000 UTC m=+0.483586094 container attach 682892edf5fc0106079f16a78cc66fe4c703d4e1842d6be1148487b3903a70d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T13:46:14.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:14 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:14.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:14.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:14 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:14.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:14.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:14 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:14.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:14.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:14 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:14.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:14.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:14 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:14.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:14.783 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:14.473+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:14.783 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:14.473+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:14.784 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:14.473+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:14.784 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:14.473+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:14.959 INFO:journalctl@ceph.osd.7.smithi120.stdout:Apr 22 13:46:14 smithi120 podman[67926]: 2024-04-22 13:46:14.677414053 +0000 UTC m=+0.710000320 container died 682892edf5fc0106079f16a78cc66fe4c703d4e1842d6be1148487b3903a70d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T13:46:14.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:14 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:14.609+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:14.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:14 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:14.609+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:14.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:14 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:14.609+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:14.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:14 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:14.609+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:14.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:14 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:14.609+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:15.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:14.786+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:15.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:14.786+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:15.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:14.786+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:15.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:14.786+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:15.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:14 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:14.786+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:15.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T13:46:15.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:15.275 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T13:46:15.275 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T13:46:15.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.8 2024-04-22T13:46:15.486 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:15.371+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:15.487 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:15.371+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:15.487 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:15.371+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:15.487 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:15.371+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:15.487 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:15.371+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 systemd[1]: Stopping Ceph osd.8 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.492+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.492+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.492+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.492+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.590+0000 7f66d872c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.590+0000 7f66d872c700 -1 osd.8 106 *** Got signal Terminated *** 2024-04-22T13:46:15.588 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:15.590+0000 7f66d872c700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:46:15.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:15.747+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:15.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:15.747+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:15.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:15.747+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:15.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:15.747+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:15.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:15 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:15.747+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:15.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:15.576+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:15.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:15.576+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:15.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:15.576+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:15.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:15.576+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:15.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:15 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:15.576+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:16.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:16 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:16.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:16.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:16 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:16.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:16.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:16 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:16.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:16.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:16 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:16.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:16.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:16 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:16.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:16.964 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:16.479+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:16.479+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:16.479+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:16.479+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:16.713+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:16.713+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:16.713+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:16.713+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:16.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:16 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:16.713+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:16.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:16 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:16.584+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:16.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:16 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:16.584+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:16.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:16 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:16.584+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:16.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:16 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:16.584+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:16.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:16 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:16.584+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:17.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:17 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:17.375+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:17.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:17 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:17.375+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:17.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:17 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:17.375+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:17.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:17 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:17.375+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:17.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:17 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:17.375+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:17.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:17.465+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:17.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:17.465+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:17.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:17.465+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:17.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:17.465+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:17.890 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:17 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:17.557+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:18.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:17.758+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:18.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:17.758+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:18.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:17.758+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:18.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:17.758+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:18.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:17 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:17.758+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:18.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:18.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:18.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:18.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:18.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:18.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:18 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:18.394+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:18.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:18.420+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:18.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:18.420+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:18.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:18.420+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:18.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:18.420+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:18.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:18.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:18.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:18.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:18.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:18.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:18 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:18.547+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:19.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:18.749+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:19.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:18.749+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:19.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:18.749+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:19.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:18.749+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:19.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:18 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:18.749+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:19.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:19 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:19.423+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:19.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:19.401+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:19.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:19.402+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:19.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:19.402+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:19.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:19.402+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:19.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:19 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:19.598+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:20.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:19.735+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:20.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:19.735+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:20.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:19.735+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:20.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:19.735+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:20.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:19 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:19.735+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:20.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:20.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:20.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:20.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:20.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:20.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:20 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:20.396+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:20.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:20.400+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.365009+0000 front 2024-04-22T13:45:21.364815+0000 (oldest deadline 2024-04-22T13:45:45.465008+0000) 2024-04-22T13:46:20.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:20.400+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.366079+0000 front 2024-04-22T13:45:31.366177+0000 (oldest deadline 2024-04-22T13:45:53.066253+0000) 2024-04-22T13:46:20.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:20.400+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:38.867846+0000 front 2024-04-22T13:45:38.867672+0000 (oldest deadline 2024-04-22T13:46:02.967528+0000) 2024-04-22T13:46:20.715 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8[50218]: 2024-04-22T13:46:20.400+0000 7f66d0d18700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.969524+0000 front 2024-04-22T13:45:46.969501+0000 (oldest deadline 2024-04-22T13:46:10.469346+0000) 2024-04-22T13:46:20.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:20.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:20.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:20.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:20.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:20.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:20 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:20.554+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:21.054 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:21.055 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:21.055 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:21.055 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:21.055 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:21.055 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:20.777+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:21.388 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:21 smithi132 podman[69196]: 2024-04-22 13:46:21.057188896 +0000 UTC m=+5.585634759 container died e8e0b54663eebffd8422cd29d6f62025e67be2cf3222573c5927c9d79618f936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T13:46:21.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:21.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:21.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:21.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:21.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:21.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:21 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:21.355+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:21.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:21 smithi132 podman[69196]: 2024-04-22 13:46:21.491211732 +0000 UTC m=+6.019657592 container cleanup e8e0b54663eebffd8422cd29d6f62025e67be2cf3222573c5927c9d79618f936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:46:21.714 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:21 smithi132 bash[69196]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8 2024-04-22T13:46:21.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:21.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:21.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:21.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:21.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:21.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:21 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:21.535+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:22.155 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:21 smithi132 podman[69213]: 2024-04-22 13:46:21.726043429 +0000 UTC m=+0.672356530 container remove e8e0b54663eebffd8422cd29d6f62025e67be2cf3222573c5927c9d79618f936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:22.156 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:21 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:21.783+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:22.465 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.061535947 +0000 UTC m=+0.018081685 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:22.465 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.160138094 +0000 UTC m=+0.116683821 container create 18ab1a7091d55a1442373a79fb76721d48be94d76c73d4672cd597eee3d8e682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T13:46:22.465 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.404947973 +0000 UTC m=+0.361493711 container init 18ab1a7091d55a1442373a79fb76721d48be94d76c73d4672cd597eee3d8e682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T13:46:22.465 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.408272421 +0000 UTC m=+0.364818149 container start 18ab1a7091d55a1442373a79fb76721d48be94d76c73d4672cd597eee3d8e682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T13:46:22.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:22.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:22.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:22.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:22.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:22.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:22 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:22.380+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:22.815 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.512987893 +0000 UTC m=+0.469533646 container attach 18ab1a7091d55a1442373a79fb76721d48be94d76c73d4672cd597eee3d8e682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T13:46:22.816 INFO:journalctl@ceph.osd.8.smithi132.stdout:Apr 22 13:46:22 smithi132 podman[69304]: 2024-04-22 13:46:22.696441804 +0000 UTC m=+0.652987531 container died 18ab1a7091d55a1442373a79fb76721d48be94d76c73d4672cd597eee3d8e682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T13:46:22.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:22.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:22.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:22.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:22.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:22.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:22 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:22.500+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:23.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:23.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:23.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:23.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:23.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:23.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:22 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:22.817+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:23.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T13:46:23.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:23.439 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T13:46:23.439 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T13:46:23.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.9 2024-04-22T13:46:23.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:23.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:23.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:23.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:23.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:23.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:23 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:23.367+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:23.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 systemd[1]: Stopping Ceph osd.9 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.529+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.530+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.530+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.530+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.530+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:23.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.530+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:23.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:23.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:23.964 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:23.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:23.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:23.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:23.847+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:24.238 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.766+0000 7fc3d373b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:24.239 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.766+0000 7fc3d373b700 -1 osd.9 106 *** Got signal Terminated *** 2024-04-22T13:46:24.239 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:23 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:23.766+0000 7fc3d373b700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:46:24.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:24.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:24.673 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:24.674 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:24.674 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:24.674 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:24 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:24.323+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:24.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:24 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:24.553+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:25.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:25.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:25.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:25.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:25.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:25.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:24 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:24.836+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:25.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:25 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:25.341+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:25.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:25.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:25.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:25.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:25.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:25.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:25.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:25 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:25.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:01.438497+0000 front 2024-04-22T13:46:01.438532+0000 (oldest deadline 2024-04-22T13:46:25.538592+0000) 2024-04-22T13:46:26.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:26.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:26.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:26.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:26.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:26.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:25 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:25.882+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:26.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:26.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:26 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:26.308+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:26.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.491+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:26.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:26 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:26.492+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:01.438497+0000 front 2024-04-22T13:46:01.438532+0000 (oldest deadline 2024-04-22T13:46:25.538592+0000) 2024-04-22T13:46:27.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:27.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:27.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:27.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:27.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:27.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:26 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:26.917+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:27.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:27.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:27 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:27.353+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:27.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:27.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:27 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:27.540+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:01.438497+0000 front 2024-04-22T13:46:01.438532+0000 (oldest deadline 2024-04-22T13:46:25.538592+0000) 2024-04-22T13:46:28.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:28.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:28.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:28.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:28.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:28.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:27.902+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:28.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:28.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:28.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:28.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:28.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:28.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:28.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:28 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:28.333+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:28.988 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:14.930756+0000 front 2024-04-22T13:45:14.931101+0000 (oldest deadline 2024-04-22T13:45:40.230976+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:23.032799+0000 front 2024-04-22T13:45:23.032690+0000 (oldest deadline 2024-04-22T13:45:48.932575+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:28.933271+0000 front 2024-04-22T13:45:28.933085+0000 (oldest deadline 2024-04-22T13:45:54.233234+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.534027+0000 front 2024-04-22T13:45:36.534130+0000 (oldest deadline 2024-04-22T13:46:01.234116+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:48.235711+0000 front 2024-04-22T13:45:48.235871+0000 (oldest deadline 2024-04-22T13:46:10.535623+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:51.636638+0000 front 2024-04-22T13:45:51.636614+0000 (oldest deadline 2024-04-22T13:46:17.536560+0000) 2024-04-22T13:46:28.989 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:28 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9[60114]: 2024-04-22T13:46:28.564+0000 7fc3cbd27700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:01.438497+0000 front 2024-04-22T13:46:01.438532+0000 (oldest deadline 2024-04-22T13:46:25.538592+0000) 2024-04-22T13:46:29.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:29.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:29.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:29.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:29.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:29.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:28 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:28.924+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:29.480 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:29 smithi087 podman[100453]: 2024-04-22 13:46:29.172542252 +0000 UTC m=+5.532360487 container died 57316158b18ea4292921e1a0c2935b74131e2c59a2ca869f86df2518a5363ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T13:46:29.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:29.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:29 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:29.338+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:29.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:29 smithi087 podman[100453]: 2024-04-22 13:46:29.574500681 +0000 UTC m=+5.934318912 container cleanup 57316158b18ea4292921e1a0c2935b74131e2c59a2ca869f86df2518a5363ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T13:46:29.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:29 smithi087 bash[100453]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9 2024-04-22T13:46:30.103 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:29 smithi087 podman[100470]: 2024-04-22 13:46:29.792558151 +0000 UTC m=+0.615052258 container remove 57316158b18ea4292921e1a0c2935b74131e2c59a2ca869f86df2518a5363ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T13:46:30.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:30.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:30.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:30.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:30.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:30.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:29 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:29.964+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:30.364 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.009869916 +0000 UTC m=+0.017936502 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:30.364 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.268296434 +0000 UTC m=+0.276363020 container create 6daf22298afaac2f07f7bd82770adfb7cde2a4b22a48974f38091b187cee7b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T13:46:30.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:30.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:30 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:30.373+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:30.738 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.479415295 +0000 UTC m=+0.487481881 container init 6daf22298afaac2f07f7bd82770adfb7cde2a4b22a48974f38091b187cee7b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:46:30.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.482432526 +0000 UTC m=+0.490499113 container start 6daf22298afaac2f07f7bd82770adfb7cde2a4b22a48974f38091b187cee7b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:30.739 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.562645491 +0000 UTC m=+0.570712094 container attach 6daf22298afaac2f07f7bd82770adfb7cde2a4b22a48974f38091b187cee7b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T13:46:31.130 INFO:journalctl@ceph.osd.9.smithi087.stdout:Apr 22 13:46:30 smithi087 podman[100561]: 2024-04-22 13:46:30.774796689 +0000 UTC m=+0.782863276 container died 6daf22298afaac2f07f7bd82770adfb7cde2a4b22a48974f38091b187cee7b96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:31.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:30 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:30.915+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:31.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T13:46:31.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:31.484 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T13:46:31.484 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T13:46:31.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.10 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.365+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:31.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 systemd[1]: Stopping Ceph osd.10 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:32.172 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.797+0000 7fdd5c70e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:32.172 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.797+0000 7fdd5c70e700 -1 osd.10 106 *** Got signal Terminated *** 2024-04-22T13:46:32.172 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:31 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:31.797+0000 7fdd5c70e700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:46:32.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:32.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:32.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:32.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:32.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:32.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:31.926+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:32.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:32 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:32.344+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:33.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:33.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:33.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:33.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:33.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:33.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:32 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:32.900+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:33.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:33 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:33.296+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:34.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:34.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:33 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:33.868+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:34.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:34 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:34.282+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:35.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:34 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:34.895+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:35.671 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:35.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:35 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:35.294+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:36.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:36.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:35 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:35.883+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.289069+0000 front 2024-04-22T13:45:15.289079+0000 (oldest deadline 2024-04-22T13:45:36.988947+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:22.889963+0000 front 2024-04-22T13:45:22.889940+0000 (oldest deadline 2024-04-22T13:45:48.790134+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.691212+0000 front 2024-04-22T13:45:31.691364+0000 (oldest deadline 2024-04-22T13:45:56.391167+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:36.391875+0000 front 2024-04-22T13:45:36.391727+0000 (oldest deadline 2024-04-22T13:46:02.291587+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:46.392626+0000 front 2024-04-22T13:45:46.392673+0000 (oldest deadline 2024-04-22T13:46:09.892803+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:55.094382+0000 front 2024-04-22T13:45:55.094238+0000 (oldest deadline 2024-04-22T13:46:17.994126+0000) 2024-04-22T13:46:36.672 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:36 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10[55930]: 2024-04-22T13:46:36.251+0000 7fdd58d02700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.87:6822 osd.6 since back 2024-04-22T13:46:03.795776+0000 front 2024-04-22T13:46:03.795833+0000 (oldest deadline 2024-04-22T13:46:26.095682+0000) 2024-04-22T13:46:37.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:37.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:36 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:36.845+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:37.545 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:37 smithi120 podman[68058]: 2024-04-22 13:46:37.269118882 +0000 UTC m=+5.585655845 container died 9a71db4965e4ae6eb19b03aacefc266df4a4ebdcc086a189d7ada6698e0c7a28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T13:46:37.871 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:37 smithi120 podman[68058]: 2024-04-22 13:46:37.639779655 +0000 UTC m=+5.956316603 container cleanup 9a71db4965e4ae6eb19b03aacefc266df4a4ebdcc086a189d7ada6698e0c7a28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T13:46:37.871 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:37 smithi120 bash[68058]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10 2024-04-22T13:46:38.172 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:37 smithi120 podman[68075]: 2024-04-22 13:46:37.874596784 +0000 UTC m=+0.603875745 container remove 9a71db4965e4ae6eb19b03aacefc266df4a4ebdcc086a189d7ada6698e0c7a28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-22T13:46:38.214 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:38.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:37 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:37.842+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:38.483 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.109123584 +0000 UTC m=+0.017880785 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:38.483 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.216972744 +0000 UTC m=+0.125729978 container create d9a43c4c295b7f2d5f22aef925028cd38eb5b8554a6ffd60f1c8af84c252bfbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T13:46:38.483 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.419223823 +0000 UTC m=+0.327981017 container init d9a43c4c295b7f2d5f22aef925028cd38eb5b8554a6ffd60f1c8af84c252bfbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:38.483 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.422446505 +0000 UTC m=+0.331203698 container start d9a43c4c295b7f2d5f22aef925028cd38eb5b8554a6ffd60f1c8af84c252bfbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T13:46:38.483 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.486144033 +0000 UTC m=+0.394901239 container attach d9a43c4c295b7f2d5f22aef925028cd38eb5b8554a6ffd60f1c8af84c252bfbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T13:46:38.831 INFO:journalctl@ceph.osd.10.smithi120.stdout:Apr 22 13:46:38 smithi120 podman[68164]: 2024-04-22 13:46:38.715750276 +0000 UTC m=+0.624507473 container died d9a43c4c295b7f2d5f22aef925028cd38eb5b8554a6ffd60f1c8af84c252bfbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:39.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:38 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:38.867+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:39.292 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T13:46:39.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:39.328 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T13:46:39.329 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T13:46:39.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.11 2024-04-22T13:46:39.652 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 systemd[1]: Stopping Ceph osd.11 for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:39.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.653+0000 7ff207034700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:39.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.653+0000 7ff207034700 -1 osd.11 106 *** Got signal Terminated *** 2024-04-22T13:46:39.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.653+0000 7ff207034700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:39.966 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:39.967 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:39 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:39.831+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:41.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:40 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:40.832+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:42.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:41 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:41.813+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6820 osd.8 since back 2024-04-22T13:46:15.635532+0000 front 2024-04-22T13:46:15.635519+0000 (oldest deadline 2024-04-22T13:46:41.535633+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:43.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:42 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:42.846+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6820 osd.8 since back 2024-04-22T13:46:15.635532+0000 front 2024-04-22T13:46:15.635519+0000 (oldest deadline 2024-04-22T13:46:41.535633+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6806 osd.0 since back 2024-04-22T13:45:15.225378+0000 front 2024-04-22T13:45:15.225419+0000 (oldest deadline 2024-04-22T13:45:41.626144+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-04-22T13:45:21.626967+0000 front 2024-04-22T13:45:21.626793+0000 (oldest deadline 2024-04-22T13:45:46.227415+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6804 osd.2 since back 2024-04-22T13:45:31.528402+0000 front 2024-04-22T13:45:31.528383+0000 (oldest deadline 2024-04-22T13:45:54.428564+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.87:6814 osd.3 since back 2024-04-22T13:45:37.830122+0000 front 2024-04-22T13:45:37.830065+0000 (oldest deadline 2024-04-22T13:46:03.129936+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-04-22T13:45:47.231343+0000 front 2024-04-22T13:45:47.231274+0000 (oldest deadline 2024-04-22T13:46:12.531164+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6812 osd.5 since back 2024-04-22T13:45:52.531634+0000 front 2024-04-22T13:45:52.531692+0000 (oldest deadline 2024-04-22T13:46:20.732084+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-04-22T13:46:11.134245+0000 front 2024-04-22T13:46:11.134379+0000 (oldest deadline 2024-04-22T13:46:33.434170+0000) 2024-04-22T13:46:44.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:43 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11[56805]: 2024-04-22T13:46:43.890+0000 7ff203628700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.132:6820 osd.8 since back 2024-04-22T13:46:15.635532+0000 front 2024-04-22T13:46:15.635519+0000 (oldest deadline 2024-04-22T13:46:41.535633+0000) 2024-04-22T13:46:45.543 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:45 smithi132 podman[69424]: 2024-04-22 13:46:45.272684571 +0000 UTC m=+5.741137373 container died e1158dc63aec495938bed57ce13ab624e2d60640fbbdf15d676626d9648543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T13:46:45.910 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:45 smithi132 podman[69424]: 2024-04-22 13:46:45.645222807 +0000 UTC m=+6.113675601 container cleanup e1158dc63aec495938bed57ce13ab624e2d60640fbbdf15d676626d9648543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-22T13:46:45.911 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:45 smithi132 bash[69424]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11 2024-04-22T13:46:46.215 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:45 smithi132 podman[69440]: 2024-04-22 13:46:45.913710049 +0000 UTC m=+0.637078342 container remove e1158dc63aec495938bed57ce13ab624e2d60640fbbdf15d676626d9648543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T13:46:46.523 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.163892253 +0000 UTC m=+0.017875506 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-22T13:46:46.523 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.264895649 +0000 UTC m=+0.118878876 container create 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T13:46:46.524 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.451405492 +0000 UTC m=+0.305388720 container init 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T13:46:46.524 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.45462443 +0000 UTC m=+0.308607656 container start 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T13:46:46.524 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.526304707 +0000 UTC m=+0.380287960 container attach 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:46:46.965 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:46 smithi132 podman[69530]: 2024-04-22 13:46:46.743339517 +0000 UTC m=+0.597322753 container died 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T13:46:47.308 INFO:journalctl@ceph.osd.11.smithi132.stdout:Apr 22 13:46:47 smithi132 podman[69556]: 2024-04-22 13:46:47.289035155 +0000 UTC m=+0.540790765 container remove 1acad4a5db809fdc33d1b67ede76af88976f9ad1e35928a14175d11c49860cec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5) 2024-04-22T13:46:47.313 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T13:46:47.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:47.352 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T13:46:47.352 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T13:46:47.352 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.a 2024-04-22T13:46:47.650 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:47 smithi087 systemd[1]: Stopping Ceph mds.a for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:47.930 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a[67118]: 2024-04-22T13:46:47.651+0000 7fceb25ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:47.930 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:47 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a[67118]: 2024-04-22T13:46:47.651+0000 7fceb25ed700 -1 mds.a *** got signal Terminated *** 2024-04-22T13:46:50.489 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:50 smithi087 podman[100680]: 2024-04-22 13:46:50.15994717 +0000 UTC m=+2.604891751 container died 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T13:46:50.891 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:50 smithi087 podman[100680]: 2024-04-22 13:46:50.610028344 +0000 UTC m=+3.054972915 container cleanup 8ed971f7776c8985e2aff996775cc7b3a45baf7970c9429a50360917e4d0da06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:46:50.891 INFO:journalctl@ceph.mds.a.smithi087.stdout:Apr 22 13:46:50 smithi087 bash[100680]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-a 2024-04-22T13:46:51.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T13:46:51.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:51.056 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T13:46:51.056 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T13:46:51.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.d 2024-04-22T13:46:51.238 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:51 smithi087 systemd[1]: Stopping Ceph mds.d for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:51.738 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d[66155]: 2024-04-22T13:46:51.387+0000 7f349a201700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:51.739 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:51 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d[66155]: 2024-04-22T13:46:51.387+0000 7f349a201700 -1 mds.d *** got signal Terminated *** 2024-04-22T13:46:54.661 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:54 smithi087 podman[100812]: 2024-04-22 13:46:54.392875436 +0000 UTC m=+3.119985306 container died 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-22T13:46:54.988 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:54 smithi087 podman[100812]: 2024-04-22 13:46:54.738845965 +0000 UTC m=+3.465955845 container cleanup 4c0c8e5509235d1f4e35fa98263efe3319e219b1c56f3b2963bc5e26e9f0489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T13:46:54.989 INFO:journalctl@ceph.mds.d.smithi087.stdout:Apr 22 13:46:54 smithi087 bash[100812]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-d 2024-04-22T13:46:55.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T13:46:55.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:46:55.166 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T13:46:55.166 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T13:46:55.166 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.g 2024-04-22T13:46:55.488 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:55 smithi087 systemd[1]: Stopping Ceph mds.g for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:46:55.989 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g[66751]: 2024-04-22T13:46:55.491+0000 7f9c3edf6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:46:55.989 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:55 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g[66751]: 2024-04-22T13:46:55.491+0000 7f9c3edf6700 -1 mds.g *** got signal Terminated *** 2024-04-22T13:46:59.573 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:59 smithi087 podman[100932]: 2024-04-22 13:46:59.313316312 +0000 UTC m=+3.919715728 container died 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:46:59.940 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:59 smithi087 podman[100932]: 2024-04-22 13:46:59.692441075 +0000 UTC m=+4.298840490 container cleanup 68eeec7baf05a0e0500c47531928e56e5f3e07f5c9dd12769e864a504ab25dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-22T13:46:59.941 INFO:journalctl@ceph.mds.g.smithi087.stdout:Apr 22 13:46:59 smithi087 bash[100932]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-g 2024-04-22T13:47:00.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T13:47:00.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:00.144 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T13:47:00.144 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T13:47:00.144 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.j 2024-04-22T13:47:00.489 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:00 smithi087 systemd[1]: Stopping Ceph mds.j for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:00.489 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j[65802]: 2024-04-22T13:47:00.454+0000 7fcd93162700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:00.489 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:00 smithi087 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j[65802]: 2024-04-22T13:47:00.454+0000 7fcd93162700 -1 mds.j *** got signal Terminated *** 2024-04-22T13:47:03.576 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:03 smithi087 podman[101052]: 2024-04-22 13:47:03.281458434 +0000 UTC m=+2.922913203 container died 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:47:03.576 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:03 smithi087 podman[101052]: 2024-04-22 13:47:03.579231033 +0000 UTC m=+3.220685810 container cleanup 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:47:03.913 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:03 smithi087 bash[101052]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j 2024-04-22T13:47:03.913 INFO:journalctl@ceph.mds.j.smithi087.stdout:Apr 22 13:47:03 smithi087 podman[101066]: 2024-04-22 13:47:03.805254764 +0000 UTC m=+0.517053767 container remove 56184f87efdb8bce80180c52352b66956cec23931c78bcb571b8a9380d03e6fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T13:47:03.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T13:47:03.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:03.963 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T13:47:03.963 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T13:47:03.964 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.b 2024-04-22T13:47:04.262 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:04 smithi120 systemd[1]: Stopping Ceph mds.b for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:04.672 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: 2024-04-22T13:47:04.264+0000 7f02a13ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:04.673 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:04 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b[61438]: 2024-04-22T13:47:04.264+0000 7f02a13ef700 -1 mds.b *** got signal Terminated *** 2024-04-22T13:47:06.887 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:06 smithi120 podman[68270]: 2024-04-22 13:47:06.620909424 +0000 UTC m=+2.462088767 container died 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T13:47:07.172 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:06 smithi120 podman[68270]: 2024-04-22 13:47:06.981153646 +0000 UTC m=+2.822332989 container cleanup 3faaa7854721493aab88ee9c5f3a14e8942eb7fe0ac8a9a4f334cff5e8a0a11d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T13:47:07.172 INFO:journalctl@ceph.mds.b.smithi120.stdout:Apr 22 13:47:06 smithi120 bash[68270]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-b 2024-04-22T13:47:07.345 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T13:47:07.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:07.380 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T13:47:07.380 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T13:47:07.380 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.e 2024-04-22T13:47:07.672 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:07 smithi120 systemd[1]: Stopping Ceph mds.e for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:08.172 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: 2024-04-22T13:47:07.702+0000 7f9dc65a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:08.172 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:07 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e[60819]: 2024-04-22T13:47:07.702+0000 7f9dc65a0700 -1 mds.e *** got signal Terminated *** 2024-04-22T13:47:10.865 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:10 smithi120 podman[68404]: 2024-04-22 13:47:10.61109448 +0000 UTC m=+3.015235117 container died 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T13:47:11.172 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:10 smithi120 podman[68404]: 2024-04-22 13:47:10.959152903 +0000 UTC m=+3.363293545 container cleanup 548800f1199633bf4f00edc3b04382d5ee45d121f1f262923c191b73e24739fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T13:47:11.172 INFO:journalctl@ceph.mds.e.smithi120.stdout:Apr 22 13:47:10 smithi120 bash[68404]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-e 2024-04-22T13:47:11.388 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T13:47:11.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:11.420 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T13:47:11.420 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T13:47:11.420 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.h 2024-04-22T13:47:11.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:11 smithi120 systemd[1]: Stopping Ceph mds.h for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:11.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: 2024-04-22T13:47:11.729+0000 7f8819e99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:11.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:11 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h[61128]: 2024-04-22T13:47:11.729+0000 7f8819e99700 -1 mds.h *** got signal Terminated *** 2024-04-22T13:47:14.922 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:14 smithi120 podman[68525]: 2024-04-22 13:47:14.612420746 +0000 UTC m=+2.975237217 container died 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T13:47:15.237 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:15 smithi120 podman[68525]: 2024-04-22 13:47:15.021685306 +0000 UTC m=+3.384501786 container cleanup 858154abee1376323539a6d399941fa67c05f290fcb78f1120f34faec2900a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T13:47:15.237 INFO:journalctl@ceph.mds.h.smithi120.stdout:Apr 22 13:47:15 smithi120 bash[68525]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-h 2024-04-22T13:47:15.358 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T13:47:15.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:15.390 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T13:47:15.390 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T13:47:15.390 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.k 2024-04-22T13:47:15.672 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:15 smithi120 systemd[1]: Stopping Ceph mds.k for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:16.172 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k[60493]: 2024-04-22T13:47:15.711+0000 7fba45bf0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:16.172 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:15 smithi120 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k[60493]: 2024-04-22T13:47:15.711+0000 7fba45bf0700 -1 mds.k *** got signal Terminated *** 2024-04-22T13:47:20.149 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:19 smithi120 podman[68645]: 2024-04-22 13:47:19.726353481 +0000 UTC m=+4.117153661 container died ea2124c2d7946bfeea7baf4a12d09454d9d0bd6c1a2ba66f107f332820d5cb2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T13:47:20.422 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:20 smithi120 podman[68645]: 2024-04-22 13:47:20.266776567 +0000 UTC m=+4.657576786 container cleanup ea2124c2d7946bfeea7baf4a12d09454d9d0bd6c1a2ba66f107f332820d5cb2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T13:47:20.422 INFO:journalctl@ceph.mds.k.smithi120.stdout:Apr 22 13:47:20 smithi120 bash[68645]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-k 2024-04-22T13:47:20.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T13:47:20.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:20.663 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T13:47:20.663 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T13:47:20.663 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.c 2024-04-22T13:47:20.946 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:20 smithi132 systemd[1]: Stopping Ceph mds.c for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:21.215 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: 2024-04-22T13:47:20.947+0000 7fe3a319d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:21.215 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:20 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c[62253]: 2024-04-22T13:47:20.947+0000 7fe3a319d700 -1 mds.c *** got signal Terminated *** 2024-04-22T13:47:22.921 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:22 smithi132 podman[69634]: 2024-04-22 13:47:22.612354622 +0000 UTC m=+1.758954701 container died 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T13:47:23.215 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:23 smithi132 podman[69634]: 2024-04-22 13:47:23.022387602 +0000 UTC m=+2.168987675 container cleanup 04749dec1a2dd1455a288059bdebfea0fa2af17983712eb5d2dddb184260970f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T13:47:23.215 INFO:journalctl@ceph.mds.c.smithi132.stdout:Apr 22 13:47:23 smithi132 bash[69634]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-c 2024-04-22T13:47:23.435 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T13:47:23.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:23.470 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T13:47:23.470 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T13:47:23.470 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.f 2024-04-22T13:47:23.783 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:23 smithi132 systemd[1]: Stopping Ceph mds.f for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:24.215 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: 2024-04-22T13:47:23.784+0000 7f6f11dee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:24.215 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:23 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f[61944]: 2024-04-22T13:47:23.784+0000 7f6f11dee700 -1 mds.f *** got signal Terminated *** 2024-04-22T13:47:26.866 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:26 smithi132 podman[69767]: 2024-04-22 13:47:26.606003798 +0000 UTC m=+2.920909743 container died bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:47:27.215 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:26 smithi132 podman[69767]: 2024-04-22 13:47:26.97639433 +0000 UTC m=+3.291300274 container cleanup bc3319dc9651821e48a96e3f790b398350587a74d6670929818526cd4a18a0b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T13:47:27.215 INFO:journalctl@ceph.mds.f.smithi132.stdout:Apr 22 13:47:26 smithi132 bash[69767]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-f 2024-04-22T13:47:27.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T13:47:27.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:27.430 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T13:47:27.430 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T13:47:27.430 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.i 2024-04-22T13:47:27.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:27 smithi132 systemd[1]: Stopping Ceph mds.i for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:27.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: 2024-04-22T13:47:27.754+0000 7f242dafd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:27.965 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:27 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i[62558]: 2024-04-22T13:47:27.754+0000 7f242dafd700 -1 mds.i *** got signal Terminated *** 2024-04-22T13:47:30.903 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:30 smithi132 podman[69889]: 2024-04-22 13:47:30.612119166 +0000 UTC m=+2.968003099 container died 927e4bf6acebd101b217c4ac4440655d5a0b3dcc1f9cc138719a2608b67248b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-22T13:47:31.215 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:30 smithi132 podman[69889]: 2024-04-22 13:47:30.996608685 +0000 UTC m=+3.352492605 container cleanup 927e4bf6acebd101b217c4ac4440655d5a0b3dcc1f9cc138719a2608b67248b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T13:47:31.215 INFO:journalctl@ceph.mds.i.smithi132.stdout:Apr 22 13:47:30 smithi132 bash[69889]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-i 2024-04-22T13:47:31.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T13:47:31.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:31.405 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T13:47:31.405 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T13:47:31.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.l 2024-04-22T13:47:31.715 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:31 smithi132 systemd[1]: Stopping Ceph mds.l for 176003e8-00ab-11ef-bc93-c7b262605968... 2024-04-22T13:47:32.215 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l[61632]: 2024-04-22T13:47:31.724+0000 7f172b8bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T13:47:32.215 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:31 smithi132 ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l[61632]: 2024-04-22T13:47:31.724+0000 7f172b8bc700 -1 mds.l *** got signal Terminated *** 2024-04-22T13:47:33.898 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:33 smithi132 podman[70011]: 2024-04-22 13:47:33.588733037 +0000 UTC m=+1.967248231 container died 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T13:47:33.898 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:33 smithi132 podman[70011]: 2024-04-22 13:47:33.900355886 +0000 UTC m=+2.278871080 container cleanup 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T13:47:34.215 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:33 smithi132 bash[70011]: ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l 2024-04-22T13:47:34.215 INFO:journalctl@ceph.mds.l.smithi132.stdout:Apr 22 13:47:34 smithi132 podman[70025]: 2024-04-22 13:47:34.134819711 +0000 UTC m=+0.542976893 container remove 501453153bdeca8a924827f0990cde4294d4a189f1fad8b7bf4a36f131c6d180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-176003e8-00ab-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T13:47:34.264 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176003e8-00ab-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T13:47:34.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T13:47:34.295 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T13:47:34.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T13:47:34.483 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:47:36.901 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T13:47:37.088 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:47:39.518 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T13:47:39.710 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:47:42.025 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:47:42.056 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:47:42.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T13:47:42.112 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T13:47:42.115 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi087/crash 2024-04-22T13:47:42.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash -- . 2024-04-22T13:47:42.141 INFO:teuthology.orchestra.run.smithi087.stderr:tar: /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T13:47:42.141 INFO:teuthology.orchestra.run.smithi087.stderr:tar: Error is not recoverable: exiting now 2024-04-22T13:47:42.143 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi120/crash 2024-04-22T13:47:42.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash -- . 2024-04-22T13:47:42.187 INFO:teuthology.orchestra.run.smithi120.stderr:tar: /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T13:47:42.187 INFO:teuthology.orchestra.run.smithi120.stderr:tar: Error is not recoverable: exiting now 2024-04-22T13:47:42.189 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi132/crash 2024-04-22T13:47:42.190 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash -- . 2024-04-22T13:47:42.226 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/176003e8-00ab-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T13:47:42.227 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-04-22T13:47:42.228 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T13:47:42.228 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T13:47:42.274 INFO:tasks.cephadm:Compressing logs... 2024-04-22T13:47:42.274 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T13:47:42.277 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T13:47:42.279 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T13:47:42.299 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T13:47:42.301 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T13:47:42.301 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T13:47:42.301 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T13:47:42.301 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T13:47:42.303 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.a.log: 87.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T13:47:42.303 INFO:teuthology.orchestra.run.smithi087.stderr: 90.2% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T13:47:42.303 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T13:47:42.303 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr: 94.6% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr: 94.6% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-22T13:47:42.309 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T13:47:42.313 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T13:47:42.313 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- 94.4% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T13:47:42.313 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/cephadm.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr: /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log: -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T13:47:42.315 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr: 90.3% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T13:47:42.316 INFO:teuthology.orchestra.run.smithi132.stderr: 92.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:: No such file or directory 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T13:47:42.320 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.3% 87.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T13:47:42.321 INFO:teuthology.orchestra.run.smithi132.stderr: 83.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T13:47:42.322 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T13:47:42.322 INFO:teuthology.orchestra.run.smithi087.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T13:47:42.323 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T13:47:42.329 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T13:47:42.335 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T13:47:42.338 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T13:47:42.338 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T13:47:42.342 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T13:47:42.343 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T13:47:42.345 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T13:47:42.356 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T13:47:42.362 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.6% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T13:47:42.367 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T13:47:42.368 INFO:teuthology.orchestra.run.smithi132.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T13:47:42.383 INFO:teuthology.orchestra.run.smithi120.stderr: 94.6% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-22T13:47:42.384 INFO:teuthology.orchestra.run.smithi132.stderr: 92.7% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T13:47:42.402 INFO:teuthology.orchestra.run.smithi132.stderr: 92.7% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T13:47:42.410 INFO:teuthology.orchestra.run.smithi120.stderr: 94.5% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T13:47:42.696 INFO:teuthology.orchestra.run.smithi132.stderr: 91.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T13:47:42.701 INFO:teuthology.orchestra.run.smithi087.stderr: 90.3% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T13:47:42.858 INFO:teuthology.orchestra.run.smithi120.stderr: 91.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T13:47:43.683 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T13:47:46.948 INFO:teuthology.orchestra.run.smithi132.stderr: 97.4% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T13:48:07.170 INFO:teuthology.orchestra.run.smithi087.stderr: 93.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T13:48:12.179 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T13:48:12.560 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T13:48:16.339 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T13:48:17.590 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T13:48:18.023 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T13:48:18.373 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T13:48:19.817 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T13:48:20.662 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T13:48:20.664 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-04-22T13:48:20.664 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m38.370s 2024-04-22T13:48:20.664 INFO:teuthology.orchestra.run.smithi132.stderr:user 2m17.672s 2024-04-22T13:48:20.664 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m4.776s 2024-04-22T13:48:21.019 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T13:48:21.021 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-04-22T13:48:21.021 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m38.732s 2024-04-22T13:48:21.021 INFO:teuthology.orchestra.run.smithi087.stderr:user 2m8.629s 2024-04-22T13:48:21.021 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m4.338s 2024-04-22T13:48:26.660 INFO:teuthology.orchestra.run.smithi120.stderr: 92.9% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T13:48:31.257 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T13:48:40.225 INFO:teuthology.orchestra.run.smithi120.stderr: 94.0% -- replaced with /var/log/ceph/176003e8-00ab-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T13:48:40.227 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-04-22T13:48:40.227 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m57.933s 2024-04-22T13:48:40.227 INFO:teuthology.orchestra.run.smithi120.stderr:user 3m36.062s 2024-04-22T13:48:40.227 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m7.125s 2024-04-22T13:48:40.227 INFO:tasks.cephadm:Archiving logs... 2024-04-22T13:48:40.228 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi087/log 2024-04-22T13:48:40.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T13:49:00.096 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi120/log 2024-04-22T13:49:00.097 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T13:49:30.227 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi132/log 2024-04-22T13:49:30.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T13:49:51.375 INFO:tasks.cephadm:Removing cluster... 2024-04-22T13:49:51.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force 2024-04-22T13:49:51.567 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:49:52.002 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force 2024-04-22T13:49:52.190 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:49:52.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176003e8-00ab-11ef-bc93-c7b262605968 --force 2024-04-22T13:49:52.885 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 176003e8-00ab-11ef-bc93-c7b262605968 2024-04-22T13:49:53.334 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T13:49:53.334 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T13:49:53.352 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T13:49:53.370 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T13:49:53.385 INFO:tasks.cephadm:Teardown complete 2024-04-22T13:49:53.386 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T13:49:53.398 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T13:49:53.398 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T13:49:53.401 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T13:49:53.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T13:49:53.525 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T13:49:53.536 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T13:49:53.537 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:49:53.540 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:49:53.542 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T13:49:53.553 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:49:53.555 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:49:53.556 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:^- hv01.front.sepia.ceph.com 2 6 377 8 +520us[ +520us] +/- 601ms 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -2679us[-2726us] +/- 53ms 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +3499us[+3499us] +/- 69ms 2024-04-22T13:49:53.558 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:^- hv01.front.sepia.ceph.com 2 6 377 6 +486us[ +433us] +/- 601ms 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -2738us[-2792us] +/- 53ms 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +3434us[+3434us] +/- 69ms 2024-04-22T13:49:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:49:53.561 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T13:49:53.561 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-04-22T13:49:53.561 INFO:teuthology.orchestra.run.smithi120.stdout:^- hv01.front.sepia.ceph.com 2 6 377 6 +521us[ +521us] +/- 601ms 2024-04-22T13:49:53.561 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -2701us[-2749us] +/- 53ms 2024-04-22T13:49:53.561 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +3491us[+3491us] +/- 69ms 2024-04-22T13:49:53.562 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T13:49:53.562 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T13:49:53.572 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T13:49:53.573 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T13:49:53.584 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T13:49:53.684 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T13:49:53.753 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T13:49:53.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:49:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:53.847 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:49:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:53.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T13:49:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:53.902 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T13:49:53.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T13:49:53.926 INFO:teuthology.task.internal:Duration was 2169.141184 seconds 2024-04-22T13:49:53.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T13:49:53.939 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T13:49:53.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T13:49:53.942 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T13:49:53.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T13:49:53.977 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:49:53.979 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:49:53.979 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T13:49:54.468 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T13:49:54.468 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-04-22T13:49:54.469 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T13:49:54.490 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-04-22T13:49:54.490 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T13:49:54.537 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-04-22T13:49:54.538 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T13:49:54.560 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T13:49:54.561 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T13:49:54.563 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T13:49:54.580 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T13:49:54.625 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T13:49:54.625 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T13:49:54.631 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T13:49:54.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T13:49:55.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T13:49:55.017 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T13:49:55.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T13:49:55.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T13:49:55.063 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T13:49:55.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T13:49:55.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.102 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.121 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-04-22T13:49:55.125 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-04-22T13:49:55.149 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-04-22T13:49:55.166 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:55.187 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:55.217 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T13:49:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T13:49:55.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T13:49:55.242 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T13:49:55.244 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi087 2024-04-22T13:49:55.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T13:49:55.285 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi120 2024-04-22T13:49:55.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T13:49:55.320 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_12:36:42-fs-wip-lusov-quiescer-distro-default-smithi/7668274/remote/smithi132 2024-04-22T13:49:55.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T13:49:55.354 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T13:49:55.355 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T13:49:55.357 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T13:49:55.359 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T13:49:55.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T13:49:55.417 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T13:49:55.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T13:49:55.496 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T13:49:55.496 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T13:49:55.498 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T13:49:55.501 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T13:49:55.510 INFO:teuthology.orchestra.run.smithi087.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 13:49 /home/ubuntu/cephtest 2024-04-22T13:49:55.512 INFO:teuthology.orchestra.run.smithi120.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 13:49 /home/ubuntu/cephtest 2024-04-22T13:49:55.515 INFO:teuthology.orchestra.run.smithi132.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 13:49 /home/ubuntu/cephtest 2024-04-22T13:49:55.516 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T13:49:55.567 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T13:49:55.642 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} duration: 2169.1411843299866 flavor: default owner: scheduled_leonidus@teuthology success: true 2024-04-22T13:49:55.642 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T13:49:55.731 INFO:teuthology.run:pass